Posted on
• Artificial Intelligence

Self-Hosted Artificial Intelligence for Linux Administrators

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Self-Hosted Artificial Intelligence for Linux Administrators

Tired of shipping logs, configs, and know-how to a cloud AI you don’t control? With modern open models and lightweight tooling, you can run powerful AI locally—on your own metal—keeping data private, latency low, and costs predictable. This guide shows Linux admins how to stand up a self-hosted AI stack, wire it into real ops tasks, and manage it like any other internal service.

What you’ll get:

  • A local LLM API you can cURL like any internal service

  • A web UI for your team

  • A simple RAG (Retrieval-Augmented Generation) assistant that answers questions from your internal docs/runbooks

  • Practical security and operations tips for production use

Why this matters now:

  • Open models are good—really good—especially for infra Q&A, code, and runbook guidance.

  • Quantization lets you run capable models (e.g., Llama 3, Mistral) on CPUs or modest GPUs.

  • The ecosystem (Ollama, OpenWebUI, Qdrant) works well on standard Linux without vendor lock-in.

0) Prerequisites and hardware notes

Approximate sizing (guidance, not rules):

  • CPU-only: 8+ cores and 16–32 GB RAM will run 7B quantized models comfortably for CLI and web chat.

  • GPU (optional): 8–24 GB VRAM can accelerate larger or faster models. CPU fallback always works.

  • Disk: Models are big. Budget 8–20+ GB per model.

Install common tools (choose your package manager):

  • apt (Debian/Ubuntu)
sudo apt update
sudo apt install -y curl git jq podman python3 python3-pip build-essential
  • dnf (Fedora/RHEL/CentOS Stream/Rocky/Alma)
sudo dnf install -y curl git jq podman python3 python3-pip gcc gcc-c++ make
  • zypper (openSUSE/SLES)
sudo zypper refresh
sudo zypper install -y curl git jq podman python3 python3-pip gcc gcc-c++ make

Tip: If you prefer Docker, install it instead of Podman. Podman is in base repos and works rootless; Docker may require enabling upstream repos on some distros.

GPU (optional): Install your vendor driver. For NVIDIA, also enable the NVIDIA Container Toolkit if you plan to run models in containers. CPU-only works fine—use GPU later if/when you need speed.

1) Run a local LLM service with Ollama

Ollama is a simple way to run local models and expose an HTTP API. It’s lightweight, fast, and easy to manage.

Install Ollama (one-line script):

curl -fsSL https://ollama.com/install.sh | sh

Start/enable the service (if not started automatically):

sudo systemctl enable --now ollama
sudo systemctl status ollama

Pull a model and chat:

ollama pull llama3
ollama run llama3

Use the HTTP API (great for scripts and tools):

curl -s http://localhost:11434/api/generate \
  -d '{"model":"llama3","prompt":"Summarize what Ollama does in one sentence.","stream":false}' \
  | jq -r '.response'

Note: Ollama stores model files under /usr/share/ollama or ~/.ollama (varies by install). Ensure you have disk space.

2) Add a web UI for the team (OpenWebUI)

OpenWebUI provides a multi-user, browser-based chat interface that connects to Ollama. You can run it with Podman or Docker.

  • Using Podman (Linux host networking for simplicity):
podman run -d --name openwebui --restart=always \
  --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  ghcr.io/open-webui/open-webui:main
  • Using Docker (Linux host networking works on Linux):
docker run -d --name openwebui --restart=always \
  --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  ghcr.io/open-webui/open-webui:main

Open your browser to:

http://<your-server>:8080

If you didn’t use host networking, map a port:

  • Podman:
podman run -d --name openwebui --restart=always \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://host.containers.internal:11434 \
  ghcr.io/open-webui/open-webui:main

Then browse to http://:3000

3) Build a “Runbook Copilot” with RAG (Qdrant + Python)

We’ll index your internal docs (Markdown, text) into a vector database (Qdrant) and query them with a local model. This gives grounded, doc-aware answers—no hallucinated nonsense about your environment.

Run Qdrant (vector DB) in a container:

  • Podman:
podman run -d --name qdrant --restart=always \
  -p 6333:6333 \
  -v qdrant-data:/qdrant/storage \
  qdrant/qdrant:latest
  • Docker:
docker run -d --name qdrant --restart=always \
  -p 6333:6333 \
  -v qdrant-data:/qdrant/storage \
  qdrant/qdrant:latest

Create a Python venv and install libraries:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install sentence-transformers qdrant-client requests

Prepare a folder with your internal docs, e.g. ./docs containing .md and .txt files (runbooks, SOPs, on-call notes).

Ingest script (save as ingest.py):

#!/usr/bin/env python3
import os, glob, uuid
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

COLLECTION = "runbooks"
DOC_DIR = "./docs"

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")  # 384-dim
client = QdrantClient("http://localhost:6333")

# Create collection if not exists
if COLLECTION not in [c.name for c in client.get_collections().collections]:
    client.recreate_collection(
        collection_name=COLLECTION,
        vectors_config=VectorParams(size=384, distance=Distance.COSINE),
    )

def read_docs(path):
    files = glob.glob(os.path.join(path, "**", "*.*"), recursive=True)
    for f in files:
        if f.lower().endswith((".md", ".txt", ".rst", ".log", ".conf")):
            try:
                with open(f, "r", encoding="utf-8", errors="ignore") as fh:
                    yield f, fh.read()
            except Exception:
                pass

points = []
for path, content in read_docs(DOC_DIR):
    # Chunk naive: per file; for larger docs, split into paragraphs
    emb = model.encode([content])[0]
    points.append(PointStruct(
        id=str(uuid.uuid4()),
        vector=emb.tolist(),
        payload={"path": path, "content": content[:5000]}  # store sample
    ))

if points:
    client.upsert(collection_name=COLLECTION, points=points)
    print(f"Ingested {len(points)} docs into '{COLLECTION}'")
else:
    print("No docs found to ingest.")

Query script (save as ask.py) that retrieves relevant chunks and calls the local LLM (Ollama HTTP API):

#!/usr/bin/env python3
import sys, requests, textwrap
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient

QUESTION = " ".join(sys.argv[1:]) or "How do I safely restart PostgreSQL in production?"
COLLECTION = "runbooks"

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
qdrant = QdrantClient("http://localhost:6333")

qvec = embedder.encode([QUESTION])[0].tolist()
hits = qdrant.search(collection_name=COLLECTION, query_vector=qvec, limit=5)

context = "\n\n".join([f"[{h.payload.get('path')}] {h.payload.get('content','')}" for h in hits])

prompt = f"""
You are a Linux SRE assistant. Use the context to answer the question concisely and safely.
If steps are risky, warn first. If uncertain, say so.

Context:
{context}

Question: {QUESTION}
Answer:
"""

resp = requests.post(
    "http://localhost:11434/api/generate",
    json={"model": "llama3", "prompt": prompt, "stream": False},
    timeout=120
).json()

print(textwrap.fill(resp.get("response","<no response>"), width=100))

Run it:

python3 ingest.py
python3 ask.py "How do I rotate Nginx logs without restarting?"

Result: You’ll get a grounded answer using your runbooks. Adjust chunking and models as you grow.

4) Secure and operate it like production infra

Basic auth and TLS via NGINX in front of OpenWebUI and/or Ollama.

Install NGINX and htpasswd tool:

  • apt:
sudo apt update
sudo apt install -y nginx apache2-utils
  • dnf:
sudo dnf install -y nginx httpd-tools
  • zypper:
sudo zypper install -y nginx apache2-utils

Create a password:

sudo htpasswd -c /etc/nginx/.htpasswd admin

Minimal NGINX site (e.g., /etc/nginx/conf.d/ai.conf):

server {
    listen 443 ssl;
    server_name ai.example.com;

    ssl_certificate     /etc/ssl/certs/ai.crt;
    ssl_certificate_key /etc/ssl/private/ai.key;

    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location /ui/ {
        proxy_pass http://127.0.0.1:8080/;  # OpenWebUI
        proxy_set_header Host $host;
        proxy_http_version 1.1;
    }

    location /api/ {
        proxy_pass http://127.0.0.1:11434/; # Ollama
        proxy_set_header Host $host;
        proxy_http_version 1.1;
    }
}

Reload NGINX:

sudo nginx -t && sudo systemctl reload nginx

Systemd resource controls (example for a custom service):

# /etc/systemd/system/ollama.service.d/limits.conf
[Service]
MemoryMax=8G
CPUQuota=300%

Apply:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Logging/metrics:

  • Journal: journalctl -u ollama -f

  • Container logs: podman logs -f openwebui or docker logs -f openwebui

  • Export basic host metrics with node_exporter; watch GPU with nvidia-smi if applicable.

Access control:

  • Keep the AI services on an internal VLAN.

  • Require auth for web/UI and API.

  • Consider separate model hosts for higher-trust data (e.g., security logs).

Backups:

  • Backup Qdrant volume and your docs repo.

  • Pin model versions you rely on.

Real-world example: “Patch Window Assistant”

  • Ingest your OS hardening guides, maintenance SOPs, and rollback runbooks.

  • Ask: “Generate a safe, step-by-step Linux kernel patch plan for fleet X with pre/post checks and rollback.”

  • The assistant replies with your org’s exact steps, including change window prep and verification commands.

Troubleshooting tips

  • Model too slow? Try a smaller/quantized model (e.g., 7B Q4_K_M). Pull with ollama pull mistral or use ollama run <model>:q4_0.

  • API errors? Check sudo systemctl status ollama and journalctl -u ollama.

  • Container networking issues? Use --network host on Linux to simplify.

  • RAM pressure? Set MemoryMax in systemd and reduce concurrent chats.

Conclusion and next steps (CTA)

You now have:

  • A local LLM API (Ollama) you can script against

  • A team-friendly UI (OpenWebUI)

  • A doc-aware RAG assistant using your runbooks (Qdrant + Python)

  • A path to production with auth, TLS, and resource controls

Next steps: 1) Point the assistant at your real docs and iterate on chunking/collections. 2) Wire it into ChatOps (e.g., Slack or Matrix bot) for on-call workflows. 3) Automate the stack with Ansible/Terraform and add CI for model and prompt changes.

Own your AI. Keep your data in your rack. And make your team faster and safer—without sending your secrets to the cloud.

Related posts