Posted on
• Artificial Intelligence

Artificial Intelligence Consulting on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Consulting on Linux: A Practical, Bash-First Playbook

Modern AI consulting runs best where reliability, automation, and cost control are non-negotiable—Linux. Whether you’re delivering a quick proof of concept or a hardened internal service, Linux gives you reproducibility, scriptability, and first-class tooling for GPUs and containers. In this guide, you’ll get a clear, Bash-friendly blueprint for setting up, prototyping, and shipping AI solutions on Linux—plus concrete install commands for apt, dnf, and zypper.

Why this matters (and why Linux)

  • Cost and control: Linux lets you scale on-prem or in the cloud with minimal overhead and no vendor lock-in for core tooling.

  • Reproducibility: Package managers, containers, and systemd make it easy to move from laptop to server without surprises.

  • Security and compliance: Linux’s mature controls (SELinux/AppArmor, firewalld/ufw, SSH) simplify audits and enterprise reviews.

  • Ecosystem velocity: From Python ML stacks to local LLMs like Ollama or llama.cpp, Linux gets first-class support early and often.

What follows are 5 actionable steps you can reuse across engagements—with real commands and minimal fluff.

1) Bootstrap a reproducible AI workstation/server

Install core developer tooling, Python, and helpers you’ll need in almost every engagement.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y build-essential git curl wget jq python3 python3-venv python3-pip cmake
  • Fedora/RHEL family (dnf):
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git curl wget jq python3 python3-pip cmake
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y -t pattern devel_basis
sudo zypper install -y git curl wget jq python3 python3-pip cmake

Note:

  • On Debian/Ubuntu you need python3-venv for python -m venv. On Fedora/openSUSE, it’s usually bundled with Python; if you hit issues, install python3-virtualenv and use virtualenv.

Containers (Docker or Podman)

  • Debian/Ubuntu (apt):
sudo apt install -y docker.io podman
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
  • Fedora/RHEL (dnf):
sudo dnf install -y podman
# Optional: Docker Engine (packaged as moby)
sudo dnf install -y moby-engine docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
  • openSUSE (zypper):
sudo zypper install -y docker podman
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and back in to apply group membership changes.

Tip: Prefer Podman on Fedora/RHEL for rootless containers and tighter security by default. Use Docker if your client infra/tools already standardize on it.

2) Create an isolated Python environment and baseline stack

Most AI consulting deliverables end up in Python, even if you wrap them with Bash or systemd. Keep dependencies isolated.

mkdir -p ~/ai-demo && cd ~/ai-demo
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip wheel setuptools
pip install numpy pandas scikit-learn fastapi "uvicorn[standard]" requests sentence-transformers faiss-cpu

Optional: PyTorch (CPU-only quick start)

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

GPU note: For NVIDIA/AMD GPUs, follow the official CUDA/ROCm install docs for your distro and match CUDA toolkit versions to your framework. In consulting, consider containerizing GPU workloads to avoid host-driver drift.

3) Run a local LLM for demos with Ollama

When internet access is constrained or data can’t leave the network, a local LLM can make or break a pilot. Ollama is a great “it just works” choice for Linux.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Start and test:

# Service usually starts automatically; verify:
systemctl status ollama

# Pull and chat with a small model (e.g., Llama 3)
ollama run llama3

Wrap it with a simple Bash helper to get JSON-free output:

#!/usr/bin/env bash
# ask.sh — Ask a local LLM via Ollama HTTP API
prompt="$*"
resp=$(curl -s http://localhost:11434/api/generate \
  -d "$(jq -n --arg p "$prompt" '{model:"llama3", prompt:$p, stream:false}')" )
echo "$resp" | jq -r '.response'

Make it executable:

chmod +x ask.sh
./ask.sh "Summarize the latest release notes in 3 bullets."

This gives you an offline-friendly demo path for ideation sessions and stakeholder reviews.

4) Package a lightweight service deliverable (FastAPI + systemd)

Turn a prototype into a reusable service that operations teams can deploy, monitor, and restart automatically.

Create a tiny FastAPI that proxies to your local LLM:

cat > app.py <<'PY'
from fastapi import FastAPI
from pydantic import BaseModel
import requests

app = FastAPI()

class Q(BaseModel):
    prompt: str

@app.post("/ask")
def ask(q: Q):
    r = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": "llama3", "prompt": q.prompt, "stream": False},
        timeout=120,
    )
    r.raise_for_status()
    return {"response": r.json().get("response", "")}
PY

Run locally:

source .venv/bin/activate
uvicorn app:app --host 0.0.0.0 --port 8000

Open a new shell to test:

curl -s -X POST http://localhost:8000/ask \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Write a 2-sentence executive summary about on-prem LLMs."}' | jq

Make it a systemd service (server-safe, auto-restart). Adjust paths and user:

sudo tee /etc/systemd/system/ai-demo.service >/dev/null <<'UNIT'
[Unit]
Description=AI Demo FastAPI Service
After=network-online.target
Wants=network-online.target

[Service]
User=ai
WorkingDirectory=/home/ai/ai-demo
Environment=PYTHONUNBUFFERED=1
ExecStart=/home/ai/ai-demo/.venv/bin/uvicorn app:app --host 0.0.0.0 --port 8000
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now ai-demo.service
systemctl status ai-demo.service

Open the firewall for the API:

  • Debian/Ubuntu (ufw):
sudo apt install -y ufw
sudo ufw allow 8000/tcp
sudo ufw enable
sudo ufw status
  • Fedora/RHEL/openSUSE (firewalld):
# Fedora/RHEL
sudo dnf install -y firewalld || true
# openSUSE
# sudo zypper install -y firewalld

sudo systemctl enable --now firewalld
sudo firewall-cmd --add-port=8000/tcp --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --list-ports

Now your API is a first-class Linux service—discoverable, restartable, and observable via journalctl -u ai-demo.service.

5) Security and ops checklist for client deployments

  • Lock down SSH:
# Create a key-only user
adduser ai
usermod -aG wheel ai  # use 'sudo' group on Debian/Ubuntu
# Configure /etc/ssh/sshd_config: PasswordAuthentication no; PermitRootLogin no
sudo systemctl restart sshd
  • Protect secrets:
# Example: store an API key as a file with strict perms
sudo tee /etc/ai-demo.env >/dev/null <<'ENV'
MY_API_KEY=replace_me
ENV
sudo chmod 600 /etc/ai-demo.env
# Reference it from systemd with EnvironmentFile=
  • Audit and updates (sample):
# Debian/Ubuntu
sudo apt update && sudo apt upgrade -y

# Fedora/RHEL
sudo dnf upgrade -y

# openSUSE
sudo zypper refresh && sudo zypper update -y
  • Observability:
journalctl -u ai-demo.service -f
  • Containers in production: Prefer Podman/systemd user services for rootless deployments where possible. For GPU workloads, use official NVIDIA/AMD container runtimes and pin image/toolkit versions to avoid mismatches.

Real-world example: 2-week internal Q&A assistant

  • Context: A mid-size enterprise wanted an internal Q&A assistant over a private Confluence export. No data could leave the network.

  • Build:

    • Day 1–2: Bootstrap host; Ollama + FastAPI microservice; basic firewall and SSH hardening.
    • Day 3–5: Chunk and embed documents with sentence-transformers + faiss-cpu.
    • Day 6–8: Simple RAG endpoint (/ask) with retrieval + prompt templating (kept local).
    • Day 9–10: Containerize with Podman; systemd user service; logs to journald; runbook + handoff.

  • Outcome: < 2 weeks delivery, no cloud dependencies, monthly infra cost almost nil on existing hardware, green-lit for further departmental rollouts.

Conclusion and next steps

Linux is the consulting multiplier for AI: it’s scriptable, portable, and production-ready from day one. With the steps above, you can go from blank server to demo to durable service—entirely in Bash.

Your next move:

  • Try the quickstart: install the base stack, run Ollama, and stand up the FastAPI service.

  • Containerize it with Podman/Docker and add systemd.

  • Add retrieval (FAISS + sentence-transformers) to turn it into a real internal assistant.

If you want a battle-tested template repo or a tailored on-prem AI plan, reach out—happy to share a minimal reference stack you can drop into client environments.

Related posts