Posted on
• Artificial Intelligence

Private Artificial Intelligence Workflows on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Private Artificial Intelligence Workflows on Linux

Build, run, and automate AI locally—without sending your data to the cloud.

Hook: Why run AI privately?

What if your prompts, documents, and audio never left your laptop or server? Private AI keeps your sensitive data in-house, cuts vendor lock-in, reduces latency, and can be cheaper at scale. Linux is the perfect home for private AI: scriptable, reproducible, and flexible—from laptops to air‑gapped racks.

In this guide, you’ll set up fast, private AI workflows on Linux using battle-tested CLI tools. You’ll get practical steps to:

  • Run a local LLM in minutes

  • Build and launch models without phone‑home daemons

  • Add retrieval-augmented generation (RAG) to query your own documents

  • Transcribe audio offline

  • Harden and automate the whole pipeline

All examples are shell-first, with installation instructions for apt, dnf, and zypper.

Why private AI on Linux is worth it

  • Data sovereignty and compliance: Keep data on systems you control (GDPR/PII-friendly).

  • Predictable cost: No per-token surprises; reuse hardware for multiple workloads.

  • Latency and reliability: Local inference doesn’t wait on an API.

  • Reproducibility and portability: Script it once; run it anywhere—WSL, laptops, workstations, homelabs, or servers.

Prerequisites (install once)

Update your system and install common build and runtime dependencies.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git curl wget build-essential cmake pkg-config \
  python3 python3-venv python3-pip ffmpeg \
  poppler-utils unzip

Fedora/RHEL/CentOS (dnf):

sudo dnf -y update
sudo dnf install -y git curl wget gcc gcc-c++ make cmake pkgconfig \
  python3 python3-virtualenv python3-pip ffmpeg \
  poppler-utils unzip
# Note: On some Fedora/RHEL spins, ffmpeg may require RPM Fusion:
# sudo dnf install -y https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm
# sudo dnf install -y ffmpeg

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y git curl wget gcc gcc-c++ make cmake pkg-config \
  python3 python3-pip python3-virtualenv ffmpeg \
  poppler-tools unzip

Optional performance libraries for CPU inference:

  • apt: sudo apt install -y libopenblas-dev

  • dnf: sudo dnf install -y openblas-devel

  • zypper: sudo zypper install -y openblas-devel

Optional container runtime (choose one):

  • apt (Docker): sudo apt install -y docker.io && sudo systemctl enable --now docker

  • apt (Podman): sudo apt install -y podman

  • dnf (Podman): sudo dnf install -y podman

  • zypper (Podman): sudo zypper install -y podman

Firewall tools (pick what matches your distro policy):

  • apt (UFW): sudo apt install -y ufw

  • dnf/zypper (Firewalld): sudo dnf install -y firewalld or sudo zypper install -y firewalld

1) The 10-minute private LLM: Ollama

Ollama runs and manages local LLMs via a single CLI. Great for quick wins, prototyping, and local dev tools.

Install:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Pull and run a model (examples: Llama 3, Mistral; review licenses for your use):

ollama pull llama3:8b-instruct
ollama run llama3:8b-instruct

Use it from Bash:

printf "You are a privacy-focused Linux assistant. Summarize: %s" \
"How to securely back up /etc with tar and gpg" | \
ollama run llama3:8b-instruct

Tip: By default Ollama binds to 127.0.0.1. To keep it local-only, don’t expose it on external interfaces. If you use a firewall:

  • UFW (Debian/Ubuntu):

    sudo ufw enable
sudo ufw default deny incoming
sudo ufw allow from 127.0.0.1

  • Firewalld (Fedora/openSUSE):

    sudo systemctl enable --now firewalld
sudo firewall-cmd --set-default-zone=block
sudo firewall-cmd --zone=trusted --add-source=127.0.0.1 --permanent
sudo firewall-cmd --reload

Why Ollama: Fast path to local inference, clean model management, simple REST API for dev tools—all with your data staying on-box.

2) Compile-time control: llama.cpp (pure local, no daemon)

If you want maximum control, a tiny footprint, and no background service, build llama.cpp. It runs GGUF models with great CPU and GPU optimizations.

Build llama.cpp:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j

Optional GPU acceleration:

  • NVIDIA CUDA build:

    make clean
LLAMA_CUDA=1 make -j

    You need CUDA installed and a matching driver. Installation varies by distro and GPU; refer to NVIDIA’s packaging for apt/dnf/zypper or use their runfile installer.

  • AMD ROCm build (supported GPUs only):

    make clean
LLAMA_HIPBLAS=1 make -j

Download a GGUF model (example; pick a compatible, licensed model):

mkdir -p ~/models/llama3
cd ~/models/llama3
# Replace with a specific GGUF URL from the model’s page (e.g., TheBloke/Mistral-7B-Instruct-GGUF)
wget -O model.gguf "https://huggingface.co/.../MODEL-FILE.gguf"

Run inference:

cd ~/llama.cpp
./main -m ~/models/llama3/model.gguf -p "Explain how to rotate ssh keys on Linux."

Batch script example:

#!/usr/bin/env bash
set -euo pipefail
MODEL="${HOME}/models/llama3/model.gguf"
PROMPT="${1:-Summarize the pros and cons of running private AI on Linux.}"
exec "${HOME}/llama.cpp/main" -m "$MODEL" -p "$PROMPT" -n 256 --temp 0.2

Why llama.cpp: Minimal, fast, auditable. Excellent for air‑gapped environments and custom pipelines.

3) Your documents, your answers: a private RAG CLI

Combine a local LLM with a tiny vector store to query your own PDFs/notes—no cloud indexing.

We’ll use:

  • Chroma (local vector DB, stores in SQLite)

  • Sentence-transformers (local embeddings)

  • Any local LLM (Ollama or llama.cpp) for the final answer

Create a Python venv and install dependencies:

python3 -m venv ~/.venvs/rag
source ~/.venvs/rag/bin/activate
pip install --upgrade pip
pip install chromadb sentence-transformers pypdf

Index documents:

#!/usr/bin/env bash
# file: index_docs.sh
set -euo pipefail
source ~/.venvs/rag/bin/activate
python - <<'PY'
import glob, os
import chromadb
from chromadb.utils import embedding_functions
from pypdf import PdfReader

docs_dir = os.environ.get("DOCS_DIR", "./docs")
paths = sorted(glob.glob(os.path.join(docs_dir, "**/*.pdf"), recursive=True))

client = chromadb.PersistentClient(path=".chroma")
coll = client.get_or_create_collection(
    name="docs",
    embedding_function=embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")
)

def pdf_text(p):
    r = PdfReader(p)
    return "\n".join(page.extract_text() or "" for page in r.pages)

ids, texts, metas = [], [], []
for p in paths:
    txt = pdf_text(p)
    if not txt.strip():
        continue
    ids.append(p)
    texts.append(txt[:20000])  # keep it small for demo; chunking recommended
    metas.append({"path": p})

if ids:
    coll.upsert(ids=ids, documents=texts, metadatas=metas)
    print(f"Indexed {len(ids)} documents.")
else:
    print("No PDFs found.")
PY

Query with local context and ask your LLM:

#!/usr/bin/env bash
# file: ask.sh
set -euo pipefail
Q="${*:?Usage: ./ask.sh 'your question'}"
source ~/.venvs/rag/bin/activate
CTX=$(python - <<'PY'
import sys, chromadb
client = chromadb.PersistentClient(path=".chroma")
coll = client.get_collection("docs")
q = sys.stdin.read()
res = coll.query(query_texts=[q], n_results=4)
chunks = []
for doc, meta in zip(res["documents"][0], res["metadatas"][0]):
    chunks.append(f"[{meta['path']}]\n{doc}\n")
print("\n---\n".join(chunks))
PY
<<< "$Q")

PROMPT="You are a precise assistant. Use ONLY the context below to answer.\n\nContext:\n${CTX}\n\nQuestion: ${Q}\nAnswer:"
# With Ollama:
ollama run llama3:8b-instruct <<< "$PROMPT"
# Or llama.cpp:
# ~/llama.cpp/main -m ~/models/llama3/model.gguf -p "$PROMPT" -n 256 --temp 0.1

Notes:

  • For better quality, implement chunking and embeddings per chunk.

  • Everything persists locally in .chroma (SQLite). Back it up like any file.

4) Offline transcription: whisper.cpp

Turn meetings or voice notes into text with no cloud calls.

Build whisper.cpp:

git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
make -j

Download a model (small, fast; pick larger for accuracy):

bash ./models/download-ggml-model.sh base.en

Transcribe:

# Convert audio to a common format if needed
ffmpeg -i input.m4a -ar 16000 -ac 1 input.wav
./main -m ./models/ggml-base.en.bin -f input.wav -of transcript
cat transcript.txt

Batch it:

#!/usr/bin/env bash
# file: transcribe.sh
set -euo pipefail
AUDIO="${1:?Usage: ./transcribe.sh file.(wav|mp3|m4a)}"
MODEL="./models/ggml-base.en.bin"
OUT="${AUDIO%.*}.txt"
ffmpeg -y -i "$AUDIO" -ar 16000 -ac 1 /tmp/whisper.wav >/dev/null 2>&1
./main -m "$MODEL" -f /tmp/whisper.wav -of /tmp/whisper
mv /tmp/whisper.txt "$OUT"
echo "Transcript: $OUT"

5) Hardening and automation

  • Lock down networking:

    • Keep AI services bound to 127.0.0.1.
    • Use UFW/Firewalld to block inbound except localhost (see earlier).

  • Reproducibility:

    • Pin versions. Keep a requirements.txt, model checksums, and scripts in git.

  • Offline operation:

    • Pre-download models and wheels; store in an internal artifact mirror for air‑gapped hosts.

  • Resource isolation:

    • Run heavy jobs via systemd units with CPU/memory limits.
    • Example systemd service for a nightly RAG reindex:
    sudo tee /etc/systemd/system/rag-reindex.service >/dev/null <<'UNIT'
[Unit]
Description=Reindex local RAG store

[Service]
Type=oneshot
WorkingDirectory=/opt/private-ai
ExecStart=/bin/bash /opt/private-ai/index_docs.sh
Nice=10
IOSchedulingClass=best-effort
MemoryMax=2G
UNIT

sudo tee /etc/systemd/system/rag-reindex.timer >/dev/null <<'UNIT'
[Unit]
Description=Run RAG reindex nightly

[Timer]
OnCalendar=03:00
Persistent=true

[Install]
WantedBy=timers.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now rag-reindex.timer

  • Model licensing:

    • Verify that the model’s license allows your intended use (commercial, redistribution, etc.).

Real-world use cases

  • Air‑gapped dev laptop: Use llama.cpp for code assistance and doc Q&A with no internet.

  • Internal helpdesk: Ollama serves a local LLM; RAG indexes policy PDFs; service bound to localhost behind an internal reverse proxy.

  • Compliance-safe transcription: whisper.cpp converts board meeting audio to text on a secure workstation; outputs archived with your retention policy.

Conclusion and next steps

You now have multiple paths to private AI on Linux:

  • Fast start: Install Ollama and run a local LLM today.

  • Maximum control: Build llama.cpp and script it end-to-end.

  • Real utility: Add a private RAG and offline transcription.

Pick one workflow and ship it: 1) Install prerequisites (apt/dnf/zypper). 2) Choose Ollama or llama.cpp. 3) Add RAG or Whisper as needed. 4) Harden, pin versions, and automate with systemd.

Your data stays yours—and your Linux box becomes an AI workstation.

Related posts