Posted on
• Artificial Intelligence

Artificial Intelligence Linux Home Lab Projects

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Linux Home Lab Projects: 4 Hands-On Builds You Can Do This Weekend

If you’ve been curious about AI but don’t want cloud bills, vendor lock-in, or privacy headaches, your Linux box can be a powerful AI lab. With a few packages and some Bash, you can run local language models, offline speech-to-text, searchable embeddings, and even expose an AI API—all on your own hardware.

This article shows you why a Linux AI home lab is worth it, and walks you through 4 practical, reproducible projects. Each section includes distro-specific install commands for apt, dnf, and zypper.

Why a Linux AI Home Lab?

  • Ownership and privacy: Keep your data local and under your control.

  • Cost and performance: No per-token or per-GPU-minute charges; tune for your hardware.

  • Learning and reliability: Understand the stack, automate with Bash/systemd, and document reproducibility.

  • Skills that transfer: Git, containers, Python, build tooling, and performance profiling.

What you’ll need:

  • A Linux machine (8–32 GB RAM recommended; more RAM = larger models).

  • Optional GPU (NVIDIA/AMD/Intel) if you want acceleration.

  • A user with sudo, shell access, and a few GBs of disk.

Tip: If you’re low on RAM, start with tiny models (0.5B–3B parameters). You can still do impressive things while learning the fundamentals.

Project 1 — Run a Local LLM with llama.cpp

llama.cpp is a fast, lightweight C/C++ inference engine for GGUF models. It runs well on CPUs and can use GPU acceleration if available.

1) Install build dependencies

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y build-essential cmake git pkg-config libopenblas-dev
  • Fedora/RHEL/CentOS (dnf):
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y cmake git openblas-devel
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y gcc gcc-c++ make cmake git openblas-devel

2) Build llama.cpp

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j$(nproc)

Optional: Build the REST server binary too:

make -j$(nproc) server

3) Download a small public model (0.5–1.5B)

Use the Hugging Face CLI to fetch a compact, public GGUF model (e.g., Qwen 0.5B Instruct).

First, ensure Python/pip is available:

  • Debian/Ubuntu (apt):
sudo apt install -y python3 python3-venv python3-pip
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y python3 python3-pip
  • openSUSE (zypper):
sudo zypper install -y python311 python311-pip

Install the CLI and grab the model:

python3 -m pip install --user -U "huggingface_hub[cli]"
mkdir -p ~/models/qwen0_5b
huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct-GGUF qwen2_5b-instruct-q4_k_m.gguf \
  --local-dir ~/models/qwen0_5b --local-dir-use-symlinks False

4) Chat locally

cd ~/llama.cpp
./main -m ~/models/qwen0_5b/qwen2_5b-instruct-q4_k_m.gguf -c 2048 -n 256 -i --color \
  -p "You are a helpful Linux assistant. In one paragraph, explain what a Bash function is."

Pro tip: For REST access, run the server:

./server -m ~/models/qwen0_5b/qwen2_5b-instruct-q4_k_m.gguf --host 0.0.0.0 --port 8080 -c 2048

Then:

curl -s http://127.0.0.1:8080/completion -H "Content-Type: application/json" \
  -d '{"prompt":"List 3 common Bash pitfalls:\n1)","n_predict":128}'

Notes:

  • Start small; once stable, try larger models (e.g., 7B) if you have RAM/GPU.

  • GPU acceleration (cuBLAS/CLBlast/Metal) requires vendor toolkits—see llama.cpp docs.

Project 2 — Offline Speech-to-Text with whisper.cpp

Turn audio into text with zero cloud calls. Great for meeting notes, voice memos, or transcribing lectures.

1) Install dependencies

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y build-essential cmake git ffmpeg alsa-utils
  • Fedora/RHEL/CentOS (dnf):
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y cmake git ffmpeg alsa-utils
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y gcc gcc-c++ make cmake git ffmpeg alsa-utils

2) Build whisper.cpp

git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
make -j$(nproc)

3) Download a model and transcribe

Tiny/Base models are fast and good for testing:

bash ./models/download-ggml-model.sh base.en

Record 10 seconds from your mic (mono, 16 kHz) and transcribe:

arecord -f S16_LE -c 1 -r 16000 -d 10 sample.wav
./main -m models/ggml-base.en.bin -f sample.wav

Tip:

  • For other languages, download small, medium, or large variants (higher accuracy, more RAM/CPU).

  • Use -t 4 (or your core count) to speed up on multi-core CPUs:

./main -t 4 -m models/ggml-base.en.bin -f sample.wav

Project 3 — Vector Search with Qdrant (Podman) + Python Embeddings

Build a local semantic search engine using a vector database. You’ll store embeddings for documents and query them semantically.

1) Install Podman (daemonless container runtime)

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y podman
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y podman
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y podman

2) Run Qdrant

mkdir -p ~/qdrant_data
podman run -d --name qdrant -p 6333:6333 \
  -v ~/qdrant_data:/qdrant/storage \
  docker.io/qdrant/qdrant:v1.11.0

Health check:

curl -s http://127.0.0.1:6333/ | jq .

3) Set up Python env and install clients

  • Debian/Ubuntu (apt):
sudo apt install -y python3 python3-venv python3-pip
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y python3 python3-pip
  • openSUSE (zypper):
sudo zypper install -y python311 python311-pip python311-venv

Create venv and install CPU-friendly packages:

python3 -m venv ~/venvs/vec
source ~/venvs/vec/bin/activate
pip install --upgrade pip
pip install qdrant-client "sentence-transformers<3" torch --index-url https://download.pytorch.org/whl/cpu

4) Ingest and query

Save as semantic_search.py:

#!/usr/bin/env python3
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
from sentence_transformers import SentenceTransformer

# Connect to local Qdrant
qc = QdrantClient(url="http://127.0.0.1:6333")

collection = "docs"
dim = 384  # e.g., all-MiniLM-L6-v2 outputs 384-dim vectors
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Create collection if not exists
if collection not in [c.name for c in qc.get_collections().collections]:
    qc.recreate_collection(
        collection_name=collection,
        vectors_config=VectorParams(size=dim, distance=Distance.COSINE),
    )

docs = [
    "Bash functions let you group reusable commands with local scope.",
    "Podman is a daemonless container engine compatible with Docker CLI.",
    "llama.cpp runs quantized LLMs efficiently on CPUs.",
    "Whisper models transcribe speech to text fully offline."
]
# Embed
vectors = model.encode(docs, normalize_embeddings=True)

# Upsert
points = [PointStruct(id=i, vector=v.tolist(), payload={"text": docs[i]}) for i, v in enumerate(vectors)]
qc.upsert(collection_name=collection, points=points)

# Query
query = "How do I reuse code in shell scripts?"
qvec = model.encode([query], normalize_embeddings=True)[0].tolist()
res = qc.search(collection_name=collection, query_vector=qvec, limit=3)

print("Query:", query)
for hit in res:
    print(f"score={hit.score:.3f} text={hit.payload['text']}")

Run it:

source ~/venvs/vec/bin/activate
python3 semantic_search.py

You now have a local semantic search you can expand with your docs, notes, or logs.

Project 4 — Expose a Local AI API with llama.cpp Server + systemd

Turn your local model into a service that other apps (or curl) can use.

1) Ensure the llama.cpp server exists

If you didn’t already:

cd ~/llama.cpp
make -j$(nproc) server

2) Test the server

./server -m ~/models/qwen0_5b/qwen2_5b-instruct-q4_k_m.gguf --host 0.0.0.0 --port 8080 -c 2048

In another terminal:

curl -s http://127.0.0.1:8080/completion -H "Content-Type: application/json" \
  -d '{"prompt":"Write a one-line Bash tip:","n_predict":64}'

3) Create a systemd service

Create ~/.config/systemd/user/llama.service:

[Unit]
Description=Local LLM API (llama.cpp)
After=network-online.target

[Service]
Type=simple
ExecStart=%h/llama.cpp/server -m %h/models/qwen0_5b/qwen2_5b-instruct-q4_k_m.gguf --host 0.0.0.0 --port 8080 -c 2048
Restart=on-failure
Environment=OMP_NUM_THREADS=4

[Install]
WantedBy=default.target

Enable and start (user service):

systemctl --user daemon-reload
systemctl --user enable --now llama.service
systemctl --user status llama.service

Optional: Open the port on your LAN firewall (adjust to your setup). For ufw:

sudo ufw allow 8080/tcp

For firewalld:

sudo firewall-cmd --add-port=8080/tcp --permanent
sudo firewall-cmd --reload

Now any machine on your LAN can call your private AI endpoint.

Practical Tips for a Smooth Home Lab

  • Use swap if RAM is tight:
sudo fallocate -l 8G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo swapon -a
  • Monitor resources:
sudo apt install -y htop || sudo dnf install -y htop || sudo zypper install -y htop
htop

  • Version everything: Keep a lab-notes.md with commit SHAs, model names, and package versions for reproducibility.

  • Security basics: Bind services to 127.0.0.1 unless you need LAN access; use firewalls and non-default ports.

Conclusion and Next Steps (CTA)

You just saw four concrete AI projects you can run entirely on Linux: 1) Local LLM chat with llama.cpp 2) Offline speech-to-text with whisper.cpp 3) Vector search with Qdrant and Python 4) A local AI API with systemd

Pick one to complete this weekend. Start small, document your steps, and iterate. When you’re ready, try:

  • Swapping in larger or domain-specific models

  • Adding GPU acceleration

  • Wiring projects together (e.g., Whisper → LLM → Qdrant)

  • Exposing a simple CLI or TUI wrapper in Bash

Have questions or want deeper dives (GPU builds, model selection, benchmarking)? Tell me which project you’re starting with and what hardware you have—I’ll tailor the next guide to your setup.

Related posts