Posted on
• Artificial Intelligence

Private Artificial Intelligence on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Private Artificial Intelligence on Linux: Run Powerful Models Without Leaking Your Data

What if you could chat with a state‑of‑the‑art AI, summarize sensitive PDFs, and build search over your private notes—without any cloud account, sign‑in, or telemetry? Private AI on Linux gives you that power: no vendor lock‑in, no per‑token bills, and your data never leaves your machine.

This guide explains why private AI is worth your time, then walks you through practical, Linux‑friendly steps to run local language models, wire up private retrieval‑augmented generation (RAG), and keep everything locked down.

Why private AI on Linux?

  • Control and compliance: Keep PHI, PII, code, legal docs, and proprietary data on hardware you trust.

  • Cost and predictability: Pay once with your CPU/GPU; skip per‑request/cloud costs.

  • Latency and availability: Local models respond fast and keep working offline.

  • Reproducibility: Pin model files and repos; your stack won’t change out from under you.

  • Open ecosystem: Linux pairs perfectly with open‑source models and toolchains.

Real‑world examples:

  • A journalist analyzes a leak offline, avoiding any third‑party exposure.

  • A small business builds a secure, on‑prem knowledge assistant for policy docs.

  • A researcher prototyping agents avoids usage limits and unannounced API changes.

Actionable path to Private AI (3–5 steps)

1) Install and run your first private LLM with Ollama (easy mode)

Ollama bundles fast local inference and model management with a simple CLI and local HTTP API.

  • Ensure curl is installed:

    • Debian/Ubuntu (apt):
    sudo apt update
sudo apt install -y curl
    • Fedora/RHEL/CentOS (dnf):
    sudo dnf install -y curl
    • openSUSE (zypper):
    sudo zypper refresh
sudo zypper install -y curl

  • Install Ollama:

    curl -fsSL https://ollama.com/install.sh | sh

  • Start and enable the service (usually automatic after install):

    sudo systemctl enable --now ollama
systemctl status ollama

  • Run a model entirely locally (Ollama will pull it once, then cache):

    ollama run llama3

    Example:

    ollama run llama3 "List three reasons private AI on Linux is compelling."

  • Use the local HTTP API:

    curl http://localhost:11434/api/generate -d '{
"model": "llama3",
"prompt": "Explain private AI in one paragraph.",
"stream": false
}'

Tip: Try different models: mistral, llama3, phi3, qwen. Ollama handles downloads and quantized formats for you.

2) Prefer fine‑grained control? Build llama.cpp from source (power user mode)

llama.cpp is a battle‑tested C/C++ inference engine for GGUF models. Build once; run anywhere—no cloud required.

  • Install build tools:

    • Debian/Ubuntu (apt):
    sudo apt update
sudo apt install -y build-essential cmake git
    • Fedora/RHEL/CentOS (dnf):
    sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y cmake git
    • openSUSE (zypper):
    sudo zypper refresh
sudo zypper install -y -t pattern devel_basis cmake git

  • Build:

    git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j"$(nproc)"

  • Get a small, instruction‑tuned GGUF (example link; choose a Q4_K_M or similar quant):

    mkdir -p models && cd models
wget -O mistral-7b-instruct.Q4_K_M.gguf \
https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4_K_M.gguf
cd ..

  • Run a prompt:

    ./main -m models/mistral-7b-instruct.Q4_K_M.gguf \
     -p "Explain private AI on Linux in 3 bullets." \
     -n 128

  • Optional server mode (bind to localhost for privacy):

    ./server -m models/mistral-7b-instruct.Q4_K_M.gguf --host 127.0.0.1 --port 8000
curl -s http://127.0.0.1:8000/completion -d '{
"prompt":"You are local. List 3 privacy benefits."
}'

Notes:

  • GPU acceleration is available (CUDA/ROCm/Metal); build flags vary (e.g., make LLAMA_CUBLAS=1). Install vendor GPU stacks via their official repos.

3) Build a private RAG in 15 minutes (Chroma + Sentence-Transformers + Ollama)

RAG augments your model with your documents. Index locally; ask questions; get citations.

  • Install Python and pip:

    • Debian/Ubuntu (apt):
    sudo apt update
sudo apt install -y python3 python3-venv python3-pip
    • Fedora/RHEL/CentOS (dnf):
    sudo dnf install -y python3 python3-pip
    • openSUSE (zypper):
    sudo zypper refresh
sudo zypper install -y python3 python3-pip

  • Create a project and virtual environment:

    mkdir -p ~/private-rag && cd ~/private-rag
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install chromadb sentence-transformers pypdf requests

  • Put some docs in a folder (PDFs or TXTs), e.g. ~/private_docs.

  • Indexer script (rag_index.py):

    import os, glob
import chromadb
from chromadb.config import Settings
from sentence_transformers import SentenceTransformer
from pypdf import PdfReader

DOC_DIR = os.path.expanduser("~/private_docs")
persist_dir = "./chroma"

os.makedirs(persist_dir, exist_ok=True)
client = chromadb.Client(Settings(persist_directory=persist_dir, chroma_db_impl="duckdb+parquet"))
coll = client.get_or_create_collection("private_docs")

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def load_texts():
texts, metadatas, ids = [], [], []
for path in glob.glob(os.path.join(DOC_DIR, "**/*"), recursive=True):
  if os.path.isdir(path): continue
  text = ""
  if path.lower().endswith(".pdf"):
    try:
      r = PdfReader(path)
      text = "\n".join([p.extract_text() or "" for p in r.pages])
    except Exception as e:
      print(f"Skip PDF {path}: {e}")
  elif path.lower().endswith((".txt", ".md")):
    try:
      with open(path, "r", encoding="utf-8", errors="ignore") as f:
        text = f.read()
    except Exception as e:
      print(f"Skip text {path}: {e}")
  if text.strip():
    texts.append(text[:8000])  # simple chunk; tune as needed
    metadatas.append({"source": path})
    ids.append(path)
return texts, metadatas, ids

texts, metas, ids = load_texts()
if not texts:
print("No documents found in", DOC_DIR)
exit(0)

embs = embedder.encode(texts, convert_to_numpy=True, normalize_embeddings=True).tolist()
# Upsert to make re-runs idempotent
coll.upsert(documents=texts, embeddings=embs, metadatas=metas, ids=ids)
print(f"Indexed {len(texts)} docs into Chroma at {persist_dir}")

  • Simple query script using Ollama for generation (rag_query.py):

    import sys, requests, chromadb
from chromadb.config import Settings
from sentence_transformers import SentenceTransformer

question = " ".join(sys.argv[1:]) or "What are the key points across my docs?"
persist_dir = "./chroma"
model_name = "llama3"  # Must exist in Ollama (ollama run will auto-pull)

client = chromadb.Client(Settings(persist_directory=persist_dir, chroma_db_impl="duckdb+parquet"))
coll = client.get_collection("private_docs")
embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

q_emb = embedder.encode([question], convert_to_numpy=True, normalize_embeddings=True).tolist()[0]
results = coll.query(query_embeddings=[q_emb], n_results=4)
contexts = []
for doc, meta in zip(results["documents"][0], results["metadatas"][0]):
src = meta.get("source", "unknown")
contexts.append(f"[Source: {src}]\n{doc}")

prompt = f"""You are a privacy-first assistant. Use only the CONTEXT to answer.

QUESTION:
{question}

CONTEXT:
{'\n\n'.join(contexts)}

Answer concisely and cite sources in brackets.
"""

resp = requests.post("http://localhost:11434/api/generate",
                   json={"model": model_name, "prompt": prompt, "stream": False}).json()
print(resp.get("response", "").strip())

  • Run it:

    # Index
python3 rag_index.py

# Ask a private question
python3 rag_query.py "Summarize the differences between Policy A and Policy B."

Everything—documents, embeddings, and generation—stays on your machine.

4) Add private speech recognition (offline) with Vosk

Turn meetings or voice notes into text locally—no uploads.

  • In your existing virtual environment:

    pip install vosk sounddevice

  • Download a small Vosk model (e.g., English):

    mkdir -p models && cd models
# See https://alphacephei.com/vosk/models for options
wget -O vosk-model-small-en-us-0.15.zip https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
unzip vosk-model-small-en-us-0.15.zip
cd ..

  • Minimal microphone demo (stt.py):

    import json, queue, sounddevice as sd
from vosk import Model, KaldiRecognizer

model = Model("models/vosk-model-small-en-us-0.15")
rec = KaldiRecognizer(model, 16000)
rec.SetWords(True)

q = queue.Queue()
def callback(indata, frames, time, status):
  q.put(bytes(indata))

with sd.RawInputStream(samplerate=16000, blocksize=8000, dtype='int16',
                     channels=1, callback=callback):
  print("Speak (Ctrl+C to stop)...")
  try:
      while True:
          data = q.get()
          if rec.AcceptWaveform(data):
              print(json.loads(rec.Result())["text"])
          else:
              partial = json.loads(rec.PartialResult())["partial"]
              if partial: print("...", partial)
  except KeyboardInterrupt:
      print("\nFinal:", json.loads(rec.FinalResult())["text"])

This keeps your audio on-device while giving you fast, usable transcripts you can feed into your RAG.

5) Keep it private, fast, and maintainable

  • Bind to localhost:

    • llama.cpp server: --host 127.0.0.1
    • Ollama: defaults to localhost; don’t reverse‑proxy to the internet unless you know what you’re doing.

  • Principle of least exposure:

    • Only open ports on LAN if necessary; use SSH tunnels for remote access.
    • Avoid uploading documents to third‑party “model hubs” if they’re sensitive.

  • Measure and tune:

    • Try smaller quantizations (Q4_K_M/Q5_K_M) for speed with acceptable quality.
    • Adjust threads based on nproc and avoid oversubscription when RAG + LLM run together.

  • Reproducibility:

    • Pin model files (GGUF) by checksum.
    • Capture your environment in a shell script so teammates can recreate it.

  • Licensing:

    • Check each model’s license (commercial use may be restricted for some weights).

Conclusion and next steps (CTA)

You now have everything you need to run a local LLM, search your private knowledge base, and even transcribe audio—entirely offline on Linux. Pick your path:

  • Want speed and simplicity? Install Ollama and start chatting now.

  • Need fine control? Build llama.cpp and tune for your hardware.

  • Ready for value? Wire up the RAG demo and ask your docs real questions.

Next steps:

  • Add more document types (HTML, DOCX) to your indexer.

  • Try different embedding models and quantizations for quality vs. speed.

  • Wrap your stack in a script or systemd units for one‑command startup.

  • Share what you build (sanitized) and help grow the private‑AI Linux ecosystem.

Your data. Your machine. Your model. Happy hacking!

Related posts