Posted on
• Artificial Intelligence

Artificial Intelligence Self-Hosted Artificial Intelligence Services

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Self-Hosted AI on Linux: Build Your Private AI Services with Bash

If you’re a Linux power user, you don’t need a cloud subscription to run serious AI. You can run chat assistants, voice transcription, and even retrieval-augmented generation (RAG) right on your own hardware—fast, private, and scriptable with Bash.

Problem/value:

  • Cloud AI risks vendor lock-in, unpredictable costs, and privacy concerns.

  • Self-hosted AI gives you control: keep data on-prem, tune the stack, and automate everything with the CLI you already love.

In this guide you’ll:

  • Spin up a local LLM API (Ollama)

  • Add a friendly web UI

  • Transcribe audio privately

  • Build a lightweight RAG pipeline with a vector database

  • Optionally front it all with Nginx

Notes:

  • Hardware: A modern CPU with 16+ GB RAM is comfortable for 7–8B parameter models. A GPU (CUDA/ROCm) helps a lot but isn’t required.

  • All commands are copy/paste friendly. Package manager installs are shown for apt, dnf, and zypper where used.

1) Prerequisites: CLI and Containers

Choose Docker or Podman (both are fine). Also install common tools.

APT (Debian/Ubuntu):

sudo apt update
# Option A: Docker
sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Option B: Podman
sudo apt install -y podman

# Common tools
sudo apt install -y curl git jq python3 python3-pip ffmpeg

DNF (Fedora/RHEL-like):

sudo dnf -y install curl git jq python3 python3-pip ffmpeg

# Option A: Podman
sudo dnf -y install podman

# Option B: Docker (Fedora often provides moby-engine)
sudo dnf -y install moby-engine
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

ZYPPER (openSUSE/SLE):

sudo zypper refresh
sudo zypper install -y curl git jq python3-pip ffmpeg

# Option A: Docker
sudo zypper install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Option B: Podman
sudo zypper install -y podman

Log out/in (or run newgrp docker) if you just added yourself to the docker group.

2) Serve an LLM Locally with Ollama

Ollama is a lightweight LLM server that runs models locally and exposes a simple HTTP API.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Run the server and pull a model:

# Start the Ollama API for this session
ollama serve >/tmp/ollama.log 2>&1 &

# Pull a capable small model (adjust as you like)
ollama pull llama3.1:8b

# Quick interactive chat
ollama run llama3.1:8b

Containerized alternative (Docker):

docker network create ai || true
docker run -d --name ollama --network ai --restart unless-stopped \
  -p 11434:11434 -v ollama:/root/.ollama \
  ollama/ollama:latest

# Pull a model inside the container
docker exec -it ollama ollama pull llama3.1:8b

Containerized alternative (Podman):

podman network create ai || true
podman run -d --name ollama --network ai --replace \
  -p 11434:11434 -v ollama:/root/.ollama \
  docker.io/ollama/ollama:latest
podman exec -it ollama ollama pull llama3.1:8b

Test the API with curl:

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "List three Linux commands to monitor CPU and memory.",
  "stream": false
}' | jq -r .response

3) Add a Web UI (Open WebUI)

Open WebUI is a clean, self-hosted front end that talks to Ollama.

Docker:

docker run -d --name open-webui --network ai --restart unless-stopped \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Podman:

podman run -d --name open-webui --network ai --replace \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Open your browser to:

http://localhost:3000

4) Private Voice Transcription with Whisper (CPU/GPU)

Install dependencies (ffmpeg + pip already covered above), then install Whisper:

python3 -m pip install --user -U openai-whisper

Transcribe an audio file (CPU-friendly example):

whisper meeting.mp3 --model small --language en --fp16 False --output_format txt

You’ll get meeting.txt with the transcript—no cloud involved.

5) Build a Minimal RAG Stack (Qdrant + Ollama Embeddings)

We’ll use:

  • Ollama for embeddings (e.g., nomic-embed-text)

  • Qdrant for a vector database

Pull an embedding model:

ollama pull nomic-embed-text

Start Qdrant:

Docker:

docker run -d --name qdrant --network ai --restart unless-stopped \
  -p 6333:6333 -v qdrant:/qdrant/storage \
  qdrant/qdrant:latest

Podman:

podman run -d --name qdrant --network ai --replace \
  -p 6333:6333 -v qdrant:/qdrant/storage \
  docker.io/qdrant/qdrant:latest

Create a collection (nomic-embed-text produces 768-dim vectors):

curl -s -X PUT 'http://localhost:6333/collections/docs' \
  -H 'Content-Type: application/json' \
  -d '{
    "vectors": { "size": 768, "distance": "Cosine" }
  }' | jq

Index a directory of text files (docs/*.txt) into Qdrant:

#!/usr/bin/env bash
set -euo pipefail
MODEL="nomic-embed-text"
COLL="docs"
QDRANT="http://localhost:6333"
i=0
for f in docs/*.txt; do
  [ -f "$f" ] || continue
  content=$(tr -s '[:space:]' ' ' <"$f" | head -c 4000)
  vec=$(curl -s http://localhost:11434/api/embeddings \
    -d "{\"model\":\"$MODEL\",\"prompt\":$(jq -R -s . <<<"$content")}" | jq -c .embedding)
  curl -s -X PUT "$QDRANT/collections/$COLL/points?wait=true" \
    -H 'Content-Type: application/json' \
    -d "{\"points\":[{\"id\":$i,\"vector\":$vec,\"payload\":{\"file\":\"$f\"}}]}" >/dev/null
  echo "Indexed $f -> id $i"
  ((i++))
done

Search and answer with context:

query="Reset a forgotten user password on Ubuntu"

qvec=$(curl -s http://localhost:11434/api/embeddings \
  -d "{\"model\":\"nomic-embed-text\",\"prompt\":$(jq -R -s . <<<"$query")}" | jq -c .embedding)

files=$(curl -s -X POST "http://localhost:6333/collections/docs/points/search" \
  -H 'Content-Type: application/json' \
  -d "{\"vector\": $qvec, \"limit\": 3}" | jq -r '.result[].payload.file')

context=""
for f in $files; do
  context="$context
---
$(echo "$f")
$(head -c 2000 "$f")"
done

curl -s http://localhost:11434/api/generate -d "{
  \"model\":\"llama3.1:8b\",
  \"prompt\":\"Using the context below, answer the question concisely.\n\nQuestion: $query\n\nContext:\n$context\",
  \"stream\": false
}" | jq -r .response

That’s a simple, end-to-end private RAG flow.

Optional: Reverse Proxy with Nginx

Serve everything under friendly paths and prepare for TLS.

APT:

sudo apt install -y nginx

DNF:

sudo dnf install -y nginx

ZYPPER:

sudo zypper install -y nginx

Minimal proxy (http://your-host/ollama/ and /chat/):

sudo tee /etc/nginx/conf.d/ai.conf >/dev/null <<'EOF'
server {
  listen 80;
  server_name _;

  client_max_body_size 32m;

  location /ollama/ {
    proxy_pass http://127.0.0.1:11434/;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
  }

  location /chat/ {
    proxy_pass http://127.0.0.1:3000/;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
  }
}
EOF

sudo nginx -t && sudo systemctl enable --now nginx

Add TLS with your preferred method (e.g., Caddy or certbot) when exposing to the internet. For private LAN use, consider firewall rules or a VPN.

Real-World Uses You Can Ship Today

  • Private DevOps runbook assistant: Index your team’s docs and answer on-call questions with RAG.

  • Meeting transcription and summaries: Whisper + an LLM prompt for action items and timelines.

  • Air‑gapped knowledge base: All components run locally; no outbound calls required.

Conclusion and Next Steps

You now have the building blocks for a full, private AI stack:

  • Ollama for LLM serving (CLI + API)

  • Open WebUI for a clean chat interface

  • Whisper for local transcription

  • Qdrant + embeddings for search and RAG

Pick one component and ship it today:

  • If you want immediate value, start with Ollama and Open WebUI.

  • If you handle sensitive audio, add Whisper.

  • If you need “docs-aware” answers, wire up Qdrant.

When you’re ready, automate with Bash scripts, add systemd units, and proxy with Nginx. Your infrastructure, your data, your rules—on Linux.

Related posts