Posted on
• Artificial Intelligence

Artificial Intelligence Homelab with Ollama

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Homelab with Ollama: Private, Fast, and All-Linux

What if you could run capable AI models at home—no cloud bill, no data leaving your network, and full control from your Bash shell? That’s exactly what an AI homelab with Ollama gives you. In this guide you’ll go from zero to a private, always-on LLM box you can script from Linux, with practical steps, copy-paste commands, and real-world examples.

Why this matters:

  • Privacy and compliance: keep sensitive prompts and data local.

  • Low latency: responses travel inches, not continents.

  • Cost control: run on hardware you already own.

  • Hackability: Ollama’s simple CLI and HTTP API make it perfect for Bash-first automation.

What you’ll build

  • A Linux host running the Ollama daemon (systemd service)

  • One or more local language models pulled on demand

  • A scriptable HTTP API at http://localhost:11434 for automation

  • Optional GPU acceleration if you have an NVIDIA card

Prerequisites

  • A Linux machine (Debian/Ubuntu, Fedora/RHEL, openSUSE all fine)

  • CPU: modern x86_64 recommended; GPU optional (NVIDIA for acceleration)

  • RAM: 8–32 GB depending on model size

  • Disk: 10–40+ GB free for models you plan to cache

  • Shell tools: curl, jq (for JSON), and optionally ripgrep/tmux

Install those shell tools with your package manager:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl jq ripgrep tmux
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf -y install curl jq ripgrep tmux
  • openSUSE Leap/Tumbleweed (zypper):
sudo zypper refresh
sudo zypper install -y curl jq ripgrep tmux

1) Install Ollama (daemon + CLI)

Ollama provides an official install script for Linux that sets up the binary and systemd service.

  • All distros (using curl):
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
ollama --version

Verify the service:

systemctl status ollama

By default, models will be stored under:

~/.ollama/models

2) Pull and run your first local model

Pick a model that fits your hardware. Rough rule of thumb:

  • 3–8B parameter models: good for CPUs and entry GPUs (4–8 GB VRAM)

  • 13B+ models: benefit from more RAM/VRAM and patience

Examples to try:

# Pull a model (only downloaded once)
ollama pull llama3

# Chat interactively
ollama run llama3

You can also send a one-off prompt non-interactively:

ollama run llama3 "Write a 2-sentence summary of the benefits of local LLMs."

Test the HTTP API from Bash (great for scripting):

curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3","prompt":"List three CLI tools every Linux user should know."}' \
| jq -r '.response'

List what you’ve installed:

ollama list

Remove a model if you need space:

ollama rm MODEL_NAME

3) Optional: GPU acceleration (NVIDIA)

If you have an NVIDIA GPU, Ollama will use it automatically when the proprietary driver is installed and visible. Check your driver:

nvidia-smi

If that command prints GPU info, you’re set. If not, install the driver using your distro’s method. Commands vary by hardware and distro; here are starting points:

  • Debian/Ubuntu (apt):
# Ubuntu (recommended):
sudo ubuntu-drivers autoinstall
sudo reboot

# Debian (generic package name; versions vary):
sudo apt update
sudo apt install -y nvidia-driver
sudo reboot
  • Fedora/RHEL/CentOS (dnf via RPM Fusion for Fedora):
# Enable RPM Fusion (Fedora example)
sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm
sudo dnf install -y https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

# Install NVIDIA driver
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo reboot
  • openSUSE (zypper; use NVIDIA’s official repo for your version):
# Add NVIDIA repo (example for openSUSE Leap; adjust URL for your version)
sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/leap/$releasever NVIDIA
sudo zypper refresh
sudo zypper install -y nvidia-glG06 nvidia-computeG06
sudo reboot

Notes:

  • Exact package names and methods can change; when in doubt, consult your distro’s NVIDIA documentation.

  • After reboot, confirm with nvidia-smi. Ollama will prefer the GPU automatically for supported models.

4) Wire Ollama into your Bash workflows

Here are practical, ready-to-use snippets that turn your homelab into a daily driver.

  • Summarize a log file:
LOG=/var/log/syslog
PAYLOAD=$(jq -n --arg p "Summarize the key events in this log:\n\n$(tail -n 200 "$LOG")" \
  '{model:"llama3", prompt:$p}')

curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d "$PAYLOAD" \
| jq -r '.response'
  • Explain a man page section:
TOPIC=rsync
MAN_TEXT=$(man $TOPIC | col -bx | head -n 200)
curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg p "Explain the following $TOPIC usage for a beginner:\n\n$MAN_TEXT" \
        '{model:"llama3", prompt:$p}')" \
| jq -r '.response'
  • Ask with context from your code using ripgrep:
QUERY="How does authentication work?"
CONTEXT=$(rg -n --no-heading -S "" -g '!node_modules' -g '!dist' -m 200 | head -n 200)
curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg p "Using the following repository context, answer: $QUERY\n\n$CONTEXT" \
        '{model:"llama3", prompt:$p}')" \
| jq -r '.response'
  • Batch-run prompts from a file:
while IFS= read -r prompt; do
  curl -s http://localhost:11434/api/generate \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg p "$prompt" '{model:"llama3", prompt:$p}')" \
  | jq -r '.response'
  echo "----"
done < prompts.txt

5) Share it (safely) on your LAN

By default, Ollama binds to localhost. To allow other machines on your LAN to use it, run the daemon on 0.0.0.0 and secure the port.

  • Bind Ollama to all interfaces with systemd:
sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Then:

sudo systemctl daemon-reload
sudo systemctl restart ollama
  • Open the firewall for TCP/11434 on your distro:

Debian/Ubuntu (UFW):

sudo apt install -y ufw
sudo ufw allow 11434/tcp
sudo ufw enable
sudo ufw status

Fedora/RHEL/CentOS (firewalld):

sudo dnf -y install firewalld
sudo systemctl enable --now firewalld
sudo firewall-cmd --add-port=11434/tcp --permanent
sudo firewall-cmd --reload

openSUSE (firewalld):

sudo zypper install -y firewalld
sudo systemctl enable --now firewalld
sudo firewall-cmd --add-port=11434/tcp --permanent
sudo firewall-cmd --reload

From another LAN machine:

curl -s http://YOUR_HOMELAB_IP:11434/api/tags | jq

Security tip: Don’t expose this port to the public internet without authentication and TLS (use a reverse proxy like Caddy/Traefik/Nginx plus network ACLs).

Maintenance tips

  • Update Ollama:
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl restart ollama
  • Update or add models:
ollama pull llama3
ollama pull mistral
  • Inspect disk usage:
du -h ~/.ollama/models | sort -h | tail
  • Run at boot in tmux with a helper script (optional):
tmux new -s ollama -d "journalctl -fu ollama"

Real-world homelab ideas

  • Private documentation assistant: Point it at internal logs and wikis for Q&A.

  • Dev helper: Generate tests, comments, or code reviews from git diffs.

  • Ops buddy: Summarize alerts and propose remediation steps from recent metrics.

  • Learning hub: Ask “why” and “how” against local man pages and textbooks offline.

Wrap-up and next steps

You now have a private AI box you can script with Bash and reach over your LAN—all powered by Ollama. Start small with a lightweight model, wire it into one real task you do daily, and iterate.

Your next steps:

  • Pull two models and compare answers for your use case.

  • Add a Makefile or shell wrapper to standardize prompts for your team.

  • If you have an NVIDIA GPU, enable it and measure the speedup.

Have a favorite command-line pattern you’d like to automate with Ollama? Try it now:

read -p "Prompt: " P; curl -s http://localhost:11434/api/generate -H "Content-Type: application/json" -d "$(jq -n --arg p "$P" '{model:"llama3", prompt:$p}')" | jq -r '.response'

Happy homelabbing!

Related posts