Open Source Artificial Intelligence Tool Comparisons (for Linux Bash Users)

Want powerful AI on your own Linux box—no cloud bills, no API limits, and no data leaving your machine? Good. Open source AI tools have matured to the point where you can run large language models (LLMs), transcribe audio, and build retrieval pipelines locally with a few Bash commands.

This post compares practical, production-ready open source tools and shows you how to install and use them with apt, dnf, and zypper. You’ll get concrete, copy/paste-ready commands and real-world examples.

Why this matters (and why now)

• Privacy and control: Keep your data local and auditable.

• Cost predictability: Avoid surprise usage bills and per-token pricing.

• Reproducibility: Pin exact versions, automate builds, and run offline.

• Performance: Exploit your CPU/GPU without round-trips to a remote API.

Below are four core tasks most teams need, with side‑by‑side tools, installation, and quick-starts.

---

1) Local LLMs: llama.cpp vs. Ollama

Two excellent, complementary options:

• llama.cpp

  • Pro: Fast, lightweight, minimal dependencies, compiled C/C++.
  • Pro: Runs quantized GGUF models efficiently on CPU; optional GPU builds.
  • Con: Manual model management and CLI plumbing.

• Ollama

  • Pro: One-line installs and model pulls, built-in server and REST API.
  • Pro: Simple “ollama run model” workflow.
  • Con: Slightly more opinionated than raw llama.cpp.

Install prerequisites (build tools, Git)

• apt (Debian/Ubuntu):

  sudo apt update
  sudo apt install -y build-essential cmake git curl ca-certificates
  

• dnf (Fedora/RHEL):

  sudo dnf groupinstall -y "Development Tools"
  sudo dnf install -y cmake git curl ca-certificates
  

• zypper (openSUSE):

  sudo zypper install -y -t pattern devel_C_C++ 
  sudo zypper install -y cmake git curl ca-certificates
  

Option A: Build llama.cpp from source

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

• Optional: NVIDIA GPU build (requires CUDA installed and in PATH) make clean LLAMA_CUBLAS=1 make -j

Download a GGUF model (e.g., Llama 3 8B in GGUF). If you use the Hugging Face CLI, see the “Model management” section below. Or fetch with curl/wget if the model is publicly downloadable.

Example run (binary name may be main or llama-cli depending on version; check your build output):

./llama-cli -m ./models/llama3-8b.Q4_K_M.gguf -p "Explain how to tail a log and filter for errors in Bash."

Option B: Install Ollama (server + CLI)

• Install:

  curl -fsSL https://ollama.com/install.sh | sh
  

  If curl is missing, see prerequisites above.

• Run first model (auto-downloads the quantized file):

  ollama run llama3:8b
  

• Ask a question:

  echo "Write a one-liner to count unique IPs in nginx logs" | ollama run llama3:8b
  

• Start a local API server:

  ollama serve
  

  Then POST prompts to http://localhost:11434 using curl.

When to choose which:

• Prefer llama.cpp if you want maximum control and minimal stack.

• Prefer Ollama if you want turnkey model pulls, a stable local API, and fast iteration.

---

2) Speech-to-Text: whisper.cpp (offline transcription)

whisper.cpp is a high-quality, efficient port of OpenAI’s Whisper for CPUs/GPUs.

Install prerequisites

• apt:

  sudo apt update
  sudo apt install -y build-essential cmake git ffmpeg
  

• dnf:

  sudo dnf groupinstall -y "Development Tools"
  sudo dnf install -y cmake git ffmpeg
  

• zypper:

  sudo zypper install -y -t pattern devel_C_C++
  sudo zypper install -y cmake git ffmpeg
  

Build and run

git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make -j

Download a model and transcribe:

./models/download-ggml-model.sh base.en
./main -m ./models/ggml-base.en.bin -f samples/jfk.wav

Tip: Batch a directory of audio files with Bash:

for f in audio/*.wav; do
  ./main -m models/ggml-base.en.bin -f "$f" -of "${f%.wav}.txt"
done

---

3) Lightweight RAG: Chroma + Sentence-Transformers + Your LLM

You don’t need a massive stack to do Retrieval Augmented Generation (RAG). A simple local recipe:

• Chroma (embedded vector store, runs in your process)

• Sentence-Transformers (to compute embeddings)

• A local LLM (llama.cpp or Ollama) to answer with context

Install Python + pip

• apt:

  sudo apt update
  sudo apt install -y python3 python3-pip
  

• dnf:

  sudo dnf install -y python3 python3-pip
  

• zypper:

  sudo zypper install -y python3 python3-pip
  

Install Python packages locally (no root)

python3 -m pip install --user "chromadb[sqlite]" sentence-transformers

Minimal RAG script (index a docs/ directory and ask a question)

python3 <<'PY'
import os
import chromadb
from sentence_transformers import SentenceTransformer

docs_dir = "docs"
question = "What are the main steps to deploy with systemd on Linux?"

# 1) Load an embedding model
emb = SentenceTransformer("all-MiniLM-L6-v2")

# 2) Start Chroma (persists to .chroma/)
client = chromadb.Client(chromadb.config.Settings(persist_directory=".chroma"))
collection = client.get_or_create_collection(name="kb")

# 3) Index documents
docs, ids = [], []
for i, name in enumerate(sorted(os.listdir(docs_dir))):
    p = os.path.join(docs_dir, name)
    if os.path.isfile(p) and name.lower().endswith((".md", ".txt")):
        with open(p, "r", encoding="utf-8", errors="ignore") as fh:
            text = fh.read()
        docs.append(text)
        ids.append(f"doc-{i}")

if docs:
    embs = emb.encode(docs, normalize_embeddings=True).tolist()
    collection.upsert(documents=docs, embeddings=embs, ids=ids)

# 4) Retrieve context
q_emb = emb.encode([question], normalize_embeddings=True).tolist()[0]
res = collection.query(query_embeddings=[q_emb], n_results=3)

# 5) Print retrieved chunks; copy/paste into your local LLM prompt
print("Top context:")
for d in res["documents"][0]:
    print("-" * 60)
    print(d[:1200] + ("..." if len(d) > 1200 else ""))

print("\nSuggestion: paste the above into your llama.cpp or Ollama prompt.")
PY

Now paste the “Top context” into your local LLM prompt (llama.cpp or ollama run ...) to generate a grounded answer.

---

4) Model Management: Hugging Face CLI + Git LFS

For reproducible downloads and version pinning, the Hugging Face Hub plus Git LFS is a great fit.

Install Git and Git LFS

• apt:

  sudo apt update
  sudo apt install -y git git-lfs
  git lfs install
  

• dnf:

  sudo dnf install -y git git-lfs
  git lfs install
  

• zypper:

  sudo zypper install -y git git-lfs
  git lfs install
  

Install the Hugging Face CLI

Option A: pip (simple, works everywhere)

• apt:

  sudo apt install -y python3-pip
  python3 -m pip install --user "huggingface_hub[cli]"
  

• dnf:

  sudo dnf install -y python3-pip
  python3 -m pip install --user "huggingface_hub[cli]"
  

• zypper:

  sudo zypper install -y python3-pip
  python3 -m pip install --user "huggingface_hub[cli]"
  

Option B: pipx (isolated, recommended if available)

• apt:

  sudo apt install -y pipx
  pipx ensurepath
  pipx install "huggingface_hub[cli]"
  

• dnf:

  sudo dnf install -y pipx
  pipx ensurepath
  pipx install "huggingface_hub[cli]"
  

• zypper:

  sudo zypper install -y python3-pipx
  pipx ensurepath
  pipx install "huggingface_hub[cli]"
  

Login (if needed for gated models):

huggingface-cli login

Download a model or file (example GGUF file):

huggingface-cli download TheBloke/Llama-2-7B-GGUF llama-2-7b.Q4_K_M.gguf --local-dir ./models

Pin exact revisions:

huggingface-cli download some/model --revision 8f3ac1 --local-dir ./models

---

Real-world comparisons and tips

1) Choose the right “runner”

• Need quick experiments and a REST API? Use Ollama.

• Need surgical control, custom flags, or embedded library usage? Use llama.cpp.

2) Keep everything scriptable

• Wrap runs in Bash scripts so you can repeat results: # run-llm.sh ./llama-cli -m ./models/llama3-8b.Q4_K_M.gguf -n 256 -p "$*" Then: ./run-llm.sh "Summarize Nginx access log patterns."

3) Cache and pin models

• Use Hugging Face CLI with explicit revisions and store in a models/ dir checked by your CI/CD.

• Document the exact quantization (e.g., Q4_K_M) you used.

4) Start small, scale later

• Begin with CPU quantized models (Q4_K_M) and base Whisper models (base.en) for speed.

• If you need more accuracy or context length, move to bigger models or enable GPU builds.

5) Build a minimal RAG before introducing complexity

• A simple Chroma + sentence-transformers setup often outperforms “bigger stacks” for narrow domains.

• Only add vector DB servers, orchestration layers, or GPUs when you have a measured need.

---

Conclusion and next steps (CTA)

You can run modern AI entirely on Linux with first-class, open source tools:

• LLMs: llama.cpp (control) or Ollama (convenience)

• Speech-to-text: whisper.cpp (fast, offline)

• RAG: Chroma + sentence-transformers + your local LLM

• Model mgmt: Git LFS + Hugging Face CLI (reproducible)

Pick a starter path now:

• I want chat offline in 5 minutes:

  curl -fsSL https://ollama.com/install.sh | sh
  ollama run llama3:8b
  

• I want tight control and scripting:

  sudo apt/dnf/zypper install build tools (see above)
  git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make -j
  ./llama-cli -m ./models/your.gguf -p "Hello!"
  

• I want local transcription:

  git clone https://github.com/ggerganov/whisper.cpp && cd whisper.cpp && make -j
  ./models/download-ggml-model.sh base.en
  ./main -m models/ggml-base.en.bin -f samples/jfk.wav
  

Then, add reproducible model downloads with:

python3 -m pip install --user "huggingface_hub[cli]"
huggingface-cli download <repo> <file> --local-dir ./models

Your shell is now an AI workstation. Script it, version it, and ship it.