Posted on
• Artificial Intelligence

Build Offline Artificial Intelligence Tools on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Build Offline Artificial Intelligence Tools on Linux

What if you could chat with an LLM, transcribe audio, and query your own documents with AI—without sending a single byte to the cloud? Running AI offline on Linux keeps your data private, reduces cost, improves latency, and keeps working even when the internet doesn’t. In this guide, you’ll set up a practical offline AI toolbox: a local LLM, speech-to-text, and private document Q&A (RAG), all from your Linux terminal.

We’ll use widely adopted open-source projects and provide distro-specific install commands (apt, dnf, zypper) wherever packages are needed.

Why offline AI is worth it

  • Privacy and control: No data leaves your machine.

  • Predictable costs: No API bills or rate limits.

  • Low latency and reliability: Works on a plane, in a lab, or on an air-gapped server.

  • Developer ergonomics: Full reproducibility and local debugging.

Trade-offs: You download models once (they can be large), and performance depends on your CPU/GPU and RAM. Thanks to quantized formats like GGUF and efficient runtimes, even older hardware can run capable models.

1) Prepare your Linux for offline AI (one-time setup)

Install core build tools, Python, and multimedia libs (needed by some AI projects).

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y git curl wget build-essential cmake pkg-config python3 python3-venv python3-pip ffmpeg libopenblas-dev
  • Fedora/RHEL (dnf):
sudo dnf -y update
sudo dnf install -y git curl wget gcc gcc-c++ make cmake pkgconf-pkg-config python3 python3-pip ffmpeg openblas-devel
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y git curl wget gcc-c++ make cmake pkg-config python3 python3-pip ffmpeg libopenblas-devel

Tip: Keep at least 20–30 GB free for models and indexes, especially if you explore multiple LLMs.

2) Run a private local LLM with Ollama (easiest path)

Ollama manages local models for you (download, quantization, serving). After the first model download, you can go fully offline.

  • Ensure curl is installed (see step 1), then install Ollama:
curl -fsSL https://ollama.com/install.sh | sh
  • Start/enable the service:
sudo systemctl enable --now ollama
systemctl status ollama
  • Pull and run a model (first run downloads it, later runs are offline):
ollama pull llama3
echo "Write a haiku about penguins and snow." | ollama run llama3
  • Use it in Bash scripts:
PROMPT="Summarize grep vs ripgrep in 3 bullet points."
echo "$PROMPT" | ollama run llama3 > summary.txt

Real-world idea: Keep “on-device” snippets, draft commit messages, or generate shell one-liners without leaking repository details.

Optional (advanced): Build from source with llama.cpp

  • Clone and build:
git clone --recursive https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j
  • Download a GGUF model (e.g., a Mistral or Llama GGUF from Hugging Face) into models/, then run:
./main -m models/mistral-7b-instruct.Q4_K_M.gguf -p "Write a limerick about Linux."

Note: Start with 4-bit or 5-bit quantized models (Q4/Q5) for lower RAM use.

3) Offline speech-to-text with whisper.cpp

Whisper.cpp runs OpenAI’s Whisper models on CPU. Great for meeting notes, quick transcripts, and offline dictation.

  • Build whisper.cpp:
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make -j
  • Download a model (base.en is small and fast, larger models are more accurate):
bash ./models/download-ggml-model.sh base.en
  • Transcribe a sample (produces out.txt):
./main -m models/ggml-base.en.bin -f samples/jfk.wav -otxt
cat samples/jfk.wav.txt
  • Transcribe your own audio:
./main -m models/ggml-base.en.bin -f /path/to/your_audio.wav -ofolder ./out

Tip: If your audio isn’t WAV, use ffmpeg to convert:

ffmpeg -i input.mp3 -ar 16000 -ac 1 input.wav

4) Private document Q&A (RAG) with Ollama + Chroma

Build a tiny offline “chat over your files” pipeline:

  • Sentence-transformers creates local embeddings.

  • Chroma stores them in a local SQLite-backed DB.

  • Ollama answers with context (no cloud).

  • Create a virtual environment and install Python deps:

python3 -m venv .venv-rag
source .venv-rag/bin/activate
pip install --upgrade pip
pip install sentence-transformers chromadb pypdf ollama

  • Put a few PDFs or text files into a folder, e.g., docs/.

  • Save this as rag.py:

#!/usr/bin/env python3
import sys, os, glob
import chromadb
from chromadb.utils import embedding_functions
from sentence_transformers import SentenceTransformer
from pypdf import PdfReader
import subprocess
import json
import textwrap

# Config
DOCS_DIR = "docs"
DB_DIR = ".chroma-db"
MODEL = "llama3"  # already pulled via Ollama

# Load / init
os.makedirs(DB_DIR, exist_ok=True)
client = chromadb.PersistentClient(path=DB_DIR)

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
ef = embedding_functions.SentenceTransformerEmbeddingFunction(model_name="sentence-transformers/all-MiniLM-L6-v2")

collection = client.get_or_create_collection(name="docs", embedding_function=ef)

def read_texts_from_docs():
    texts = []
    for p in glob.glob(os.path.join(DOCS_DIR, "**/*"), recursive=True):
        if os.path.isdir(p): 
            continue
        try:
            if p.lower().endswith(".pdf"):
                reader = PdfReader(p)
                content = "\n".join([p.page_content for p in reader.pages if hasattr(p, 'page_content')])  # fallback if needed
                if not content:
                    content = "\n".join([pg.extract_text() or "" for pg in reader.pages])
            else:
                with open(p, "r", errors="ignore") as f:
                    content = f.read()
            if content.strip():
                texts.append((p, content))
        except Exception as e:
            # Skip unreadable files
            pass
    return texts

def chunk_text(text, chunk_size=800, overlap=100):
    words = text.split()
    chunks = []
    i = 0
    while i < len(words):
        chunk = " ".join(words[i:i+chunk_size])
        chunks.append(chunk)
        i += (chunk_size - overlap)
    return chunks

def ensure_ingested():
    existing = collection.count()
    if existing > 0:
        return
    docs = read_texts_from_docs()
    ids, metadatas, texts = [], [], []
    i = 0
    for path, content in docs:
        for c in chunk_text(content):
            ids.append(f"{path}-{i}")
            metadatas.append({"source": path})
            texts.append(c)
            i += 1
    if texts:
        collection.add(documents=texts, metadatas=metadatas, ids=ids)
        print(f"Ingested {len(texts)} chunks from {len(docs)} documents into ChromaDB.")
    else:
        print("No documents found to ingest. Put files under ./docs and rerun.")
        sys.exit(1)

def ollama_complete(model, prompt):
    cmd = ["ollama", "run", model]
    # Stream via stdin
    res = subprocess.run(cmd, input=prompt.encode(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    return res.stdout.decode()

def answer(question, k=4):
    results = collection.query(query_texts=[question], n_results=k)
    contexts = []
    sources = set()
    for doc, meta in zip(results["documents"][0], results["metadatas"][0]):
        contexts.append(doc)
        sources.add(meta.get("source"))
    context_block = "\n\n".join(contexts)
    prompt = textwrap.dedent(f"""
    You are a helpful assistant. Use the context to answer the question concisely.
    If the answer is not in the context, say you don't know.

    Context:
    {context_block}

    Question: {question}

    Answer:
    """)
    out = ollama_complete(MODEL, prompt)
    return out, list(sources)

def main():
    if len(sys.argv) < 2:
        print("Usage: python rag.py \"your question here\"")
        return
    ensure_ingested()
    q = sys.argv[1]
    ans, srcs = answer(q)
    print("\n--- Answer ---\n")
    print(ans.strip())
    if srcs:
        print("\n--- Sources ---")
        for s in srcs:
            print("-", s)

if __name__ == "__main__":
    main()
  • Run it:
source .venv-rag/bin/activate
ollama pull llama3
python rag.py "What are the key points from our design doc?"

After the first run (which downloads the LLM and the embedding model), you can disconnect from the internet. Everything else is local.

5) Bonus: Offline OCR with Tesseract

Turn scans or screenshots into text completely offline.

  • Install Tesseract:

    • Debian/Ubuntu (apt):
    sudo apt install -y tesseract-ocr
    • Fedora/RHEL (dnf):
    sudo dnf install -y tesseract
    • openSUSE (zypper):
    sudo zypper install -y tesseract

  • Minimal Python wrapper:

python3 -m venv .venv-ocr
source .venv-ocr/bin/activate
pip install pillow pytesseract
python - <<'PY'
from PIL import Image
import pytesseract
img = Image.open("scan.png")
print(pytesseract.image_to_string(img))
PY

Tip: Install language packs for better non-English OCR (package names vary by distro, e.g., tesseract-ocr-deu, tesseract-langpack-…).

Performance and troubleshooting tips

  • Start small: Use 3B–8B LLMs with Q4 quantization for laptops. Upgrade if you have more RAM.

  • Use CPU first: Everything above works CPU-only. Add GPU acceleration later for speed.

  • Memory: If you hit OOM, lower context size or pick a smaller model.

  • Verify offline: After first downloads, disable networking and re-run your commands.

  • Keep models organized: Store GGUF models under ~/models, embeddings DBs under project folders.

Conclusion and next steps

You now have a private, offline AI toolkit on Linux:

  • Local LLM with Ollama (or llama.cpp) for chat and generation.

  • Whisper.cpp for fast offline transcription.

  • A minimal RAG stack to query your own documents.

  • Optional OCR with Tesseract.

Next steps:

  • Automate workflows with Bash scripts and cron (e.g., daily transcript processing).

  • Explore different models in Ollama (code, math, multilingual).

  • Add a simple TUI or web front-end, or extend the RAG script with better chunking and citations.

  • Consider GPU acceleration when you’re ready for more speed.

Your data. Your machine. Your AI. Go build something useful—offline.

Related posts