Posted on
• Artificial Intelligence

Build an Artificial Intelligence Homelab

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Build an Artificial Intelligence Homelab on Linux (with Bash)

What if you could run your own ChatGPT-like assistant, generate images, and experiment with cutting-edge AI — all privately, on your own hardware, with nothing but Linux and the shell? That’s the promise of an AI homelab: low-latency responses, predictable costs, and total control.

In this guide you’ll see why an AI homelab is worth building, the core pieces to get right, and a practical, step-by-step path to run your first private LLM and image-generation workloads. Everything is command-line friendly, and where we install packages we include apt, dnf, and zypper instructions.

Why an AI homelab?

  • Privacy and control: Keep your prompts and data on your own drives.

  • Predictable cost: No surprise cloud bills; reuse hardware you already own.

  • Speed: On a decent GPU, local models feel snappy and are available offline.

  • Learning value: A hands-on lab strengthens your Linux, Bash, containers, and AI fundamentals.

What you’ll build

  • A minimal, reliable Linux base for AI work

  • Optional GPU acceleration (NVIDIA shown; AMD/ROCm notes included)

  • Containers or Python virtual environments for easy experimentation

  • Real services: a local LLM (Ollama + Open WebUI) and an image generator (ComfyUI)

1) Prepare the OS and shell essentials

Create a clean working directory for AI artifacts and install a baseline toolset.

Create folders:

sudo mkdir -p /opt/ai/{models,data,containers}
sudo chown -R "$USER":"$USER" /opt/ai

Install common CLI/build tools:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y git curl wget tmux htop build-essential cmake pkg-config \
  python3 python3-venv python3-pip ca-certificates gnupg lsb-release unzip
  • Fedora/RHEL (dnf):
sudo dnf -y install git curl wget tmux htop @development-tools cmake \
  python3 python3-venv python3-pip ca-certificates gnupg2 unzip
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y git curl wget tmux htop gcc gcc-c++ make cmake \
  python3 python3-venv python3-pip ca-certificates gpg2 unzip

Pro tip: Keep a shell session manager handy (tmux or screen) for long-running downloads and builds.

2) Containers you can trust: Docker or Podman

Containers make AI apps one-liners. Choose Docker (common in docs) or Podman (rootless-friendly).

Install Podman (simple on all distros):

  • apt:
sudo apt update
sudo apt install -y podman
  • dnf:
sudo dnf install -y podman
  • zypper:
sudo zypper install -y podman

Install Docker CE:

  • apt:
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
$(. /etc/os-release; echo "$VERSION_CODENAME") stable" \
| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker
  • dnf (Fedora):
sudo dnf -y install dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker
  • zypper (openSUSE):
sudo zypper addrepo https://download.docker.com/linux/opensuse/docker-ce.repo
sudo zypper refresh
sudo zypper install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

Note: For GPU inside containers, you’ll need the NVIDIA Container Toolkit (NVIDIA GPUs) or compatible hooks. We’ll keep containers CPU-friendly below and offer native GPU paths where simpler.

3) (Optional) GPU acceleration

A single recent NVIDIA GPU (e.g., RTX 3060/4060 12GB+, 3090/4090 for larger models) can accelerate LLMs and diffusion models dramatically.

Install NVIDIA drivers:

  • Debian/Ubuntu:
sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot
  • Fedora (via RPM Fusion):
sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
  https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo reboot
  • openSUSE:
    • Tumbleweed: sudo zypper ar -f https://download.nvidia.com/opensuse/tumbleweed/ NVIDIA sudo zypper refresh sudo zypper install -y nvidia-driver-G06 sudo reboot
    • Leap (example: 15.5): sudo zypper ar -f https://download.nvidia.com/opensuse/leap/15.5/ NVIDIA sudo zypper refresh sudo zypper install -y nvidia-driver-G06 sudo reboot

Verify:

nvidia-smi

AMD/ROCm note: ROCm support varies by distro and GPU model. AMD’s official docs provide the current, tested driver/runtime matrix and install steps. Many PyTorch containers and wheels work with ROCm; for containers, pass /dev/kfd and /dev/dri appropriately.

4) Deploy your first workloads

You can run both LLMs and image generation locally within minutes. Below are practical options that are friendly to Bash users.

A) Local LLM with Ollama + Open WebUI

Ollama runs open models (e.g., Llama 3, Mistral) locally, CPU or GPU. Easiest native install:

curl -fsSL https://ollama.com/install.sh | sh

Start the service (if not auto-started), then pull and run a model:

ollama serve &
ollama pull llama3:8b
ollama run llama3:8b

Quick Bash one-liner to query Ollama’s API:

curl -s http://localhost:11434/api/generate -d '{"model":"llama3:8b","prompt":"Explain bash pipes in one paragraph."}' \
 | jq -r '.response' | tr -d '\n'; echo

Prefer containers? CPU-only is straightforward (GPU inside Docker requires NVIDIA’s container runtime):

Create an isolated network and run Ollama + Open WebUI:

docker network create ai

docker run -d --name ollama --network ai -p 11434:11434 \
  -v /opt/ai/models:/root/.ollama \
  ollama/ollama:latest

docker run -d --name openwebui --network ai -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v /opt/ai/containers/openwebui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Test with curl:

curl http://localhost:3000/health

And a simple chat function for your shell:

ai() {
  prompt="$*"
  curl -s http://localhost:11434/api/generate \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"llama3:8b\",\"prompt\":\"$prompt\"}" \
    | jq -r '.response' | tr -d '\n'; echo
}
# Usage: ai Write a bash script to tail a log and colorize errors.

B) Image generation with ComfyUI (Stable Diffusion)

Run ComfyUI in a container (port 8188). CPU works; for GPU in containers, see NVIDIA Container Toolkit. If you installed native NVIDIA drivers and prefer Python venv, see section C.

docker run -d --name comfyui -p 8188:8188 \
  -v /opt/ai/containers/comfyui:/root/ComfyUI \
  ghcr.io/comfyanonymous/comfyui:latest

Open http://localhost:8188 and follow on-screen steps to download models/checkpoints.

C) Optional: Python venv for hands-on experimentation

Create a Python environment and install PyTorch + Transformers. If you have NVIDIA drivers, you can use GPU wheels that bundle CUDA.

Create venv:

python3 -m venv ~/venvs/ai
source ~/venvs/ai/bin/activate
pip install --upgrade pip

Install CPU-only (portable):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install transformers accelerate sentencepiece

Install NVIDIA GPU build (example: CUDA 12.1 wheels):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers accelerate sentencepiece
python - <<'PY'
import torch
print("CUDA available:", torch.cuda.is_available())
print("GPU:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "CPU")
PY

Tiny text-generation sample:

python - <<'PY'
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tok = AutoTokenizer.from_pretrained(model_id)
pipe = pipeline("text-generation", model=model_id, tokenizer=tok, device_map="auto")
out = pipe("Explain the difference between grep and ripgrep in one paragraph.", max_new_tokens=120)
print(out[0]["generated_text"])
PY

Tip: Store models under /opt/ai/models and use environment variables like HF_HOME to control cache locations:

export HF_HOME=/opt/ai/models/huggingface

5) Hardening and ergonomics

  • Enable services on boot:
# Docker
sudo systemctl enable --now docker

# Podman socket (rootless)
systemctl --user enable --now podman.socket
loginctl enable-linger "$USER"

  • Open firewall ports as needed:

    • Ubuntu UFW:
    sudo ufw allow 22/tcp
sudo ufw allow 11434/tcp   # Ollama
sudo ufw allow 3000/tcp    # Open WebUI
sudo ufw allow 8188/tcp    # ComfyUI
    • Fedora/openSUSE firewalld:
    sudo firewall-cmd --add-port=22/tcp --permanent
sudo firewall-cmd --add-port=11434/tcp --permanent
sudo firewall-cmd --add-port=3000/tcp --permanent
sudo firewall-cmd --add-port=8188/tcp --permanent
sudo firewall-cmd --reload

  • Backups and storage:

    • Keep large model files on a dedicated volume (/opt/ai/models).
    • Use version control for configs and scripts.
    • Consider btrfs/ZFS snapshots for quick rollback.

  • Monitoring:

watch -n1 nvidia-smi   # GPU usage (NVIDIA)
htop                   # CPU/memory
docker stats           # Container metrics

Real-world sizing tips

  • General-purpose LLM chat: 8–16 CPU cores, 32–64 GB RAM, 12–24 GB VRAM if you want mid-sized models (7–13B) at decent speed.

  • Image generation: More VRAM helps (12 GB+). A 24 GB GPU is comfortable for SDXL pipelines.

  • Disk: NVMe SSDs dramatically reduce model load times. Budget 50–200 GB for models and checkpoints as you explore.

Conclusion and next steps (CTA)

You now have the essentials to run private, local AI:

  • Base Linux toolset and directories

  • Containers or Python venvs for clean, reproducible experiments

  • Optional GPU acceleration with NVIDIA drivers

  • Real workloads: Ollama + Open WebUI and ComfyUI

Next steps:

  • Add a vector database (Chroma, Qdrant) and build a simple RAG pipeline for your documents.

  • Script repeatable setups with Bash and Compose files.

  • Explore scheduling/queuing (systemd timers, cron) and monitoring (Prometheus + Grafana).

  • Benchmark models on your hardware to pick the right size/quantization.

If this helped, try one small project today: install Ollama, pull a 7B model, and wire it to Open WebUI. In under an hour you’ll have your own private assistant — powered by Linux and Bash.

Related posts