Posted on
• Artificial Intelligence

Artificial Intelligence Homelab Guide

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Homelab Guide: Run Private LLMs On Your Linux Box

Want ChatGPT‑style power without the cloud’s cost, quotas, or data exposure? An AI homelab lets you run large language models (LLMs) on your own hardware—fast, private, and hackable. This guide shows you practical, Bash-first steps to get from zero to local inference on Debian/Ubuntu, Fedora/RHEL, or openSUSE systems, with CPU‑only and GPU‑accelerated options.

What you’ll get:

  • Why building an AI homelab is worth it

  • A minimal, distro-agnostic setup flow

  • 3–5 actionable paths: fastest local LLMs via Ollama, build-from-source via llama.cpp, optional GPU acceleration, and serving an API

  • Install commands for apt, dnf, and zypper where needed

Why an AI Homelab?

  • Privacy and control: Your prompts, embeddings, and documents never leave your network.

  • Predictable performance: No rate limits or “model busy” errors.

  • Cost efficiency: One-time hardware investment beats monthly API fees for steady workloads.

  • Skills that transfer: Containers, drivers, inference frameworks, and Linux automation you’ll reuse everywhere.

What you need:

  • CPU-only works for smaller models (1–7B params) and testing.

  • NVIDIA/AMD GPUs bring big speedups and let you use larger models.

  • RAM: 8–32 GB for small models; more is better. Disk: 20–60 GB free is comfortable.

Step 1: Prep your Linux box (tools, Python, containers)

Install common build tools, Python, and a container runtime (Docker or Podman). Use your distro’s package manager.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git curl wget build-essential cmake pkg-config \
  python3 python3-venv python3-pip podman docker.io docker-compose-plugin
# Start Docker if you plan to use it:
sudo systemctl enable --now docker

Fedora/RHEL/CentOS (dnf):

sudo dnf -y update
sudo dnf -y groupinstall "Development Tools"
sudo dnf -y install git curl wget cmake pkgconfig \
  python3 python3-pip python3-virtualenv podman moby-engine docker-compose-plugin
# Start Docker if you plan to use it:
sudo systemctl enable --now docker
sudo systemctl start docker

openSUSE Leap/Tumbleweed (zypper):

sudo zypper refresh
# Base tools
sudo zypper install -y git curl wget cmake pkg-config python3 python3-pip python3-virtualenv podman docker docker-compose
# Full build toolchain (pattern)
sudo zypper install -t pattern devel_basis
# Start Docker if you plan to use it:
sudo systemctl enable --now docker

Tip: Prefer Podman if Docker isn’t readily available on your distro; both work for this guide.

Step 2: Fastest path to a local LLM (Ollama)

Ollama packages model downloads, quantization, and a local REST API into a single, simple tool. It runs on CPU and will automatically use your GPU if configured.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Pull and run a model interactively (try a small one first):

ollama pull llama3:8b
ollama run llama3:8b

Call the local API from Bash:

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3:8b",
  "prompt": "Explain bash pipefail in one paragraph."
}' | jq -r '.response'

Run Ollama in a container:

  • Docker:
sudo docker run -d --name ollama -p 11434:11434 \
  -v ollama:/root/.ollama --restart=unless-stopped ollama/ollama
# With NVIDIA GPU (if configured with NVIDIA Container Toolkit):
# sudo docker run -d --gpus all --name ollama -p 11434:11434 -v ollama:/root/.ollama --restart=unless-stopped ollama/ollama
  • Podman (CPU example):
podman run -d --name ollama -p 11434:11434 \
  -v ollama:/root/.ollama --restart=always docker.io/ollama/ollama

Optional: open the firewall for remote clients on your LAN (choose one based on your distro):

# Debian/Ubuntu (ufw)
sudo ufw allow 11434/tcp

# Fedora/RHEL/openSUSE (firewalld)
sudo firewall-cmd --add-port=11434/tcp --permanent
sudo firewall-cmd --reload

Real-world example: Point your code editor (via an extension like Continue or Codeium’s local mode) at http://localhost:11434 to get on-device code completion and chat.

Step 3: Build-from-source CPU inference (llama.cpp)

If you like fine-grained control, llama.cpp gives you a small, fast C/C++ inference engine for GGUF models.

Build llama.cpp:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

Download a small GGUF model locally (using Hugging Face CLI) and run:

python3 -m pip install --user --upgrade huggingface_hub
# Example: TinyLlama 1.1B chat model (fits easily on CPU)
python3 -m huggingface_hub download TinyLlama/TinyLlama-1.1B-Chat-v1.0-GGUF \
  TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf --local-dir .

./main -m TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf -p "Write a one-liner about Bash aliases."

Serve a simple local endpoint with llama.cpp’s built-in server:

./server -m TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf -c 2048 --port 8080
# Query it:
curl -s -X POST http://127.0.0.1:8080/completion -H "Content-Type: application/json" \
  -d '{"prompt":"Explain grep -r in two sentences.","n_predict":128}' | jq -r '.content'

Why this path: zero external dependencies, portable, and you can enable GPU acceleration later with a simple rebuild.

Step 4: Optional GPU acceleration (NVIDIA or AMD)

GPU acceleration dramatically speeds up inference and unlocks larger models. The exact driver/toolkit steps vary by distro and GPU generation—follow the official docs to install and verify first.

NVIDIA quick notes:

  • Install the proprietary driver via your distro’s recommended method.

  • Verify with:

nvidia-smi
  • If you already have CUDA installed, rebuild llama.cpp with cuBLAS:
cd llama.cpp
make clean
make -j LLAMA_CUBLAS=1

AMD ROCm quick notes:

  • Install ROCm from AMD’s official instructions for your distro and GPU.

  • Rebuild llama.cpp with HIP BLAS:

cd llama.cpp
make clean
make -j LLAMA_HIPBLAS=1

PyTorch for GPU (in a virtual environment):

python3 -m venv ~/venvs/ai
source ~/venvs/ai/bin/activate
# Visit https://pytorch.org/get-started/locally/ to get the exact command for your CUDA/ROCm version.
# Example for CUDA 12.1 wheels (adjust as per PyTorch site):
pip install --upgrade pip
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

Sanity-check GPU in Python:

python - <<'PY'
import torch
print("CUDA available:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("CUDA device:", torch.cuda.get_device_name(0))
PY

Tip: Ollama will also use your GPU automatically if supported drivers/toolkits are present.

Step 5: Make it a service and share it (safely)

  • Keep it running:
# Ollama (native install)
sudo systemctl enable ollama
sudo systemctl start ollama

  • Reverse proxy and TLS (optional): Place Nginx/Caddy in front if you need HTTPS and auth before exposing it beyond your LAN.

  • Secure by default: Bind to localhost unless you need remote access; if you do, use a VPN or mTLS. Always restrict firewall ports.

Troubleshooting and tips

  • RAM/VRAM matters: Prefer quantized models (e.g., Q4_K_M) to fit your hardware.

  • Start small, scale up: TinyLlama (1.1B) and Phi‑2/Phi‑3 mini are great for testing; move to 7–8B models when ready.

  • Containers reduce “works on my machine” issues: Podman/Docker images encapsulate drivers and dependencies.

  • Monitor usage: Use tools like htop, nvidia-smi, and iostat to spot bottlenecks.

  • Keep models on fast storage (NVMe > SATA) for faster loads.

Call to Action

Pick your path and ship something today:

  • Fastest win: Install Ollama and run llama3:8b locally.

  • Tinker-friendly: Build llama.cpp, run TinyLlama, and script around its HTTP server.

  • Performance jump: Add GPU acceleration once your workflow is stable.

When you’re done, automate it. Write a Bash script that updates models, restarts services, and tails logs. Your homelab, your rules—private, fast, reliable.

If you want a follow-up, ask for a one-script installer that detects your distro, installs prerequisites via apt/dnf/zypper, and launches a local LLM API in minutes.

Related posts