Artificial Intelligence Bash Deployment Guide: From Zero to Local LLM

Want to ship AI without wrestling a full MLOps stack? With nothing but Bash, you can stand up a local language model (LLM) API in minutes—portable, scriptable, and repeatable across distros. This guide shows you how to deploy AI workloads on Linux using only the terminal, with actionable steps you can copy-paste today.

Why this matters:

• Control and privacy: keep models and data on your own machines.

• Cost and reliability: no external service dependencies.

• Automation: Bash makes deployments reproducible in CI, cron, and systemd.

Below you’ll find a clean path to: prepare your system, choose a runtime (venv vs containers), deploy with Ollama (easy) or llama.cpp (DIY), and harden for production. All commands include apt, dnf, and zypper variants where relevant.

---

1) System Prep: Update and Install Build Essentials

Update your system and install tools you’ll use for cloning, building, running, and monitoring.

Ubuntu/Debian (apt):

sudo apt update && sudo apt -y upgrade
sudo apt -y install git curl wget ca-certificates build-essential cmake python3 python3-venv python3-pip tmux

Fedora/RHEL (dnf):

sudo dnf -y upgrade
sudo dnf -y install git curl wget ca-certificates gcc gcc-c++ make cmake python3 python3-virtualenv python3-pip tmux

openSUSE (zypper):

sudo zypper refresh
sudo zypper update -y
sudo zypper install -y git curl wget ca-certificates gcc gcc-c++ make cmake python3 python3-pip python3-virtualenv tmux

Tip:

• tmux keeps long-running sessions alive over SSH.

• If you hit SSL or CA issues pulling containers or models, ensure ca-certificates is present (installed above).

---

2) Pick Your Runtime: Python venv or Containers

There are two common paths. Both are fully scriptable in Bash.

Option A—Python virtual environments:

• Best for lightweight scripts, small APIs, cron jobs, or when you need fine-grained Python control.

• Example: FastAPI serving a tiny CPU model.

Create and activate a venv:

python3 -m venv ~/ai-venv
source ~/ai-venv/bin/activate
pip install --upgrade pip

Install CPU-only PyTorch and Transformers (example):

pip install --extra-index-url https://download.pytorch.org/whl/cpu torch torchvision torchaudio
pip install transformers fastapi uvicorn[standard]

Run a minimal API (toy example with a tiny model):

cat > mini_api.py << 'EOF'
from fastapi import FastAPI
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

app = FastAPI()
tok = AutoTokenizer.from_pretrained("sshleifer/tiny-gpt2")
mod = AutoModelForCausalLM.from_pretrained("sshleifer/tiny-gpt2")

@app.get("/generate")
def generate(q: str = "Hello"):
    ids = tok.encode(q, return_tensors="pt")
    out = mod.generate(ids, max_length=50)
    return {"text": tok.decode(out[0], skip_special_tokens=True)}
EOF

uvicorn mini_api:app --host 0.0.0.0 --port 8000

Option B—Rootless containers with Podman:

• Clean isolation, easy upgrades/rollbacks, great for servers and CI.

Install Podman:

• apt:

sudo apt -y install podman

• dnf:

sudo dnf -y install podman

• zypper:

sudo zypper install -y podman

Test:

podman run --rm quay.io/podman/hello

---

3) The Fast Path: Deploy a Local LLM with Ollama

Ollama is a simple CLI/daemon that pulls and serves many open models out of the box (Llama, Phi, Mistral, etc.). It’s the quickest route to a local LLM API.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Start the service (system-wide):

sudo systemctl enable --now ollama

Pull and chat with a small model:

ollama pull llama3.2:3b
ollama run llama3.2:3b

Expose the HTTP API on 0.0.0.0:11434 (for remote access):

export OLLAMA_HOST=0.0.0.0:11434
ollama serve

Or run it in a container (Podman), with persistent storage:

podman volume create ollama
podman run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama docker.io/ollama/ollama

Test the API:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Summarize the value of Bash-based AI deployments in one sentence."
}'

Tip:

• Use smaller models for tight RAM/CPU budgets. Quantized variants load faster and consume less memory.

---

4) The DIY Path: Build and Serve with llama.cpp

llama.cpp is a lean C/C++ inference engine for GGUF models. It’s great for minimal overhead, easy CLI control, and high performance on CPUs. You can also enable GPU later if needed.

Clone and build:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j"$(nproc)"

Fetch a small open GGUF model (TinyLlama as an example):

pip install --user --upgrade huggingface_hub hf_transfer
export HF_HUB_ENABLE_HF_TRANSFER=1
mkdir -p ~/models/tinyllama-1.1b-chat
huggingface-cli download TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF \
  --include "*Q5_K_M.gguf" \
  --local-dir ~/models/tinyllama-1.1b-chat

Run a single prompt:

./build/bin/llama-cli -m ~/models/tinyllama-1.1b-chat/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.gguf \
  -p "You are a helpful assistant. In one paragraph, explain why Bash is great for reproducible AI deployments."

Expose an HTTP server:

./build/bin/llama-server \
  -m ~/models/tinyllama-1.1b-chat/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.gguf \
  --host 0.0.0.0 --port 8080 -c 2048 -ngl 0 -t "$(nproc)"

Test the server:

curl -s http://localhost:8080/v1/models | jq
curl -s http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
  "model": "local-gguf",
  "prompt": "List 3 reasons to use local inference."
}'

Notes:

• -t sets threads; -ngl 0 means CPU-only. For GPU acceleration, consult llama.cpp’s README for CUDA/ROCm/Metal builds and extra dependencies.

• Swap in any other GGUF model directory you prefer.

---

5) Production Basics: Persist, Secure, Automate

• Persist model/data:

  • Ollama container example already mounts a volume. For llama.cpp, keep models in a dedicated path like /opt/models or ~/models and back them up.

• Environment and secrets:

  • Store API tokens in a .env not checked into git. Load via Bash: set -a; source .env; set +a.

• Systemd services:

  • Ollama (installed via script) already provides a service. To customize bind address or memory/CPU, create a drop-in.

Example: systemd unit for llama.cpp server:

sudo tee /etc/systemd/system/llama-server.service >/dev/null << 'EOF'
[Unit]
Description=llama.cpp HTTP server
After=network-online.target

[Service]
User=YOURUSER
WorkingDirectory=/home/YOURUSER/llama.cpp
ExecStart=/home/YOURUSER/llama.cpp/build/bin/llama-server -m /home/YOURUSER/models/tinyllama-1.1b-chat/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.gguf --host 0.0.0.0 --port 8080 -t 4 -c 2048 -ngl 0
Restart=on-failure
Environment=OMP_NUM_THREADS=4

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now llama-server

• Firewall:

  • Expose only what you need (e.g., 8080 or 11434). Use ufw, firewalld, or nftables to restrict access.

• Monitoring and logs:

  • Use journalctl -u <service> -f to tail logs.
  • Add health checks with curl in cron: */5 * * * * curl -fsS http://127.0.0.1:8080/v1/models || systemctl restart llama-server.

---

Real-World Example: Chatbot CLI that Talks to Ollama

#!/usr/bin/env bash
set -euo pipefail

MODEL="${1:-llama3.2:3b}"
read -rp "You: " Q

curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$MODEL\",\"prompt\":\"$Q\"}" \
| jq -r '.response'

Save as chat.sh, chmod +x chat.sh, start Ollama server, and run:

./chat.sh "Give me 3 Bash deployment tips."

---

Troubleshooting Quick Hits

• RAM pressure or OOM:

  • Use smaller or more heavily quantized models (Q4_K_M or lower).
  • Reduce context size (-c) and threads (-t).

• Slow downloads:

  • Keep HF_HUB_ENABLE_HF_TRANSFER=1 for parallel downloads.

• Permission issues:

  • Ensure your user owns model directories; fix with chown -R.

• CPU only vs GPU:

  • Start CPU-only for simplicity; add GPU later after confirming drivers and toolkits are correct.

---

Conclusion and Next Steps

You now have two clean, Bash-first paths to local AI:

• Fast track: Ollama for instant pull-and-serve.

• DIY: llama.cpp for ultra-lean, high-control inference.

Pick one:

• If you need an API online today, run Ollama and wire it to your app.

• If you want a minimal, embeddable stack, build llama.cpp and serve a GGUF model.

Your call to action:

• Stand up a small model now using the commands above.

• Wrap it in a systemd service.

• Expose a single endpoint through your reverse proxy.

• Iterate: benchmark, quantize, and right-size threads/contexts for your hardware.

When you’re ready, script it all into one deploy.sh and make AI another reproducible step in your Bash toolbelt.