Posted on
• Artificial Intelligence

Local LLMs with Python and AI

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Run Local LLMs on Linux With Python (and Bash): Private, Fast, and Hackable

What if you could run ChatGPT-like models right on your Linux box—no cloud, no API keys, no data leaving your machine? Local LLMs make that possible. Whether you’re automating sysadmin tasks, prototyping AI-powered CLI tools, or keeping sensitive data in-house, running models locally gives you privacy, control, and predictable cost.

This guide shows two practical paths:

  • The fast path: Ollama (a local model server) + a drop-in Python client.

  • The pure-Python path: llama-cpp-python with a self-hosted OpenAI-compatible API.

We’ll use Linux-friendly commands, include apt/dnf/zypper install steps, and give real examples you can run today.

Why Local LLMs?

  • Privacy and control: Keep logs, code, and customer data off third-party servers.

  • Speed and reliability: No rate limits, no outages—your hardware, your rules.

  • Cost and scale: Small to medium workloads can be cheaper on your own metal.

  • Hackability: Integrate with Bash, Python, cron, tmux, and everything you already use.

Prerequisites: Install System Packages

Use your distro’s package manager to install Python, build tools, and common CLI utilities.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git build-essential cmake curl wget jq

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y python3 python3-pip git cmake curl wget jq

openSUSE Leap/Tumbleweed (zypper):

sudo zypper install -y python3 python3-pip git gcc-c++ make cmake curl wget jq

Tip:

  • CPU-only works fine for lightweight models and prototyping.

  • For NVIDIA GPU acceleration with llama.cpp, you’ll need the CUDA toolkit and to build with CUDA flags (covered below).

Path 1: The Fastest Way—Ollama + Python

Ollama is a small, friendly daemon that downloads and runs local models and exposes an OpenAI-compatible API. Great for quick wins.

1) Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

If the service isn’t started automatically:

ollama serve
# or
sudo systemctl enable --now ollama

2) Pull and test a model:

ollama pull llama3
ollama run llama3

Type a question, Ctrl+C to exit.

3) Call your local model from Python (OpenAI-compatible client):

python3 -m venv .venv
. .venv/bin/activate
pip install -U openai

from openai import OpenAI

# Point the client to Ollama
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resp = client.chat.completions.create(
    model="llama3",
    messages=[{"role":"user", "content":"Write a bash one-liner to count files recursively."}],
    temperature=0.2,
)
print(resp.choices[0].message.content)

4) Use it from pure Bash with curl:

curl -s http://localhost:11434/api/generate \
  -d '{"model":"llama3","prompt":"Give a 2-line summary of the Linux boot process."}' \
  | jq -r '.response'

Why this path?

  • Zero-fuss setup

  • Works with many models (Llama 3, Mistral, Phi, etc.)

  • Easy to swap models and integrate with existing tools

Path 2: Pure-Python With llama-cpp-python (Open Source, Minimal)

Prefer staying in Python with no extra daemons? llama-cpp-python runs GGUF models locally and can also serve an OpenAI-compatible API.

1) Create and activate a virtual environment:

python3 -m venv .venv
. .venv/bin/activate
pip install -U pip

2) Install llama-cpp-python (CPU):

pip install llama-cpp-python

GPU (NVIDIA) optional:

  • Ensure CUDA toolkit is installed.

  • Build with CUDA enabled:

CMAKE_ARGS="-DGGML_CUDA=on" pip install --no-cache-dir --force-reinstall llama-cpp-python

3) Download a small, permissively-licensed model (TinyLlama 1.1B Chat, GGUF):

mkdir -p models
wget -O models/tinyllama.Q4_K_M.gguf \
  https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf

You can swap in a larger model later (requires more RAM/VRAM).

4) Minimal Python usage (single-file inference):

from llama_cpp import Llama

llm = Llama(
    model_path="models/tinyllama.Q4_K_M.gguf",
    n_threads=8,       # tune for your CPU
    n_ctx=4096         # context window tokens
)

prompt = "You are a helpful assistant.\nQ: Show a bash command to find large files over 200MB in /var.\nA:"
out = llm.create_completion(
    prompt=prompt,
    max_tokens=128,
    temperature=0.2,
)
print(out["choices"][0]["text"])

5) Serve an OpenAI-compatible API from Python:

python -m llama_cpp.server \
  --model models/tinyllama.Q4_K_M.gguf \
  --host 0.0.0.0 --port 8000 --n-threads 8 --n_ctx 4096

Test the API:

curl http://localhost:8000/v1/models

Call it from Python:

pip install -U openai

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="tinyllama",
    messages=[{"role": "user", "content": "Write a bash script that rotates logs in /var/log/myapp"}],
)
print(resp.choices[0].message.content)

Why this path?

  • Fewer moving parts, fully in Python

  • Easy to embed in existing apps or scripts

  • Fine-grained control over performance settings

Real-World Things You Can Do Today

1) Summarize recent logs from Bash (Ollama example):

journalctl -n 100 --no-pager \
| jq -Rs '{model:"llama3", prompt: ("Summarize these logs in 3 bullets:\n" + .)}' \
| curl -s http://localhost:11434/api/generate -d @- \
| jq -r '.response'

2) Generate shell snippets from Python for a CLI helper:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

task = "Write a robust bash one-liner to recursively count lines of code in .py files, excluding venv and .git."
resp = client.chat.completions.create(
    model="llama3", messages=[{"role":"user","content":task}], temperature=0.1
)
print(resp.choices[0].message.content)

3) Run as a local API service (systemd unit for llama-cpp server):

sudo tee /etc/systemd/system/llama.service >/dev/null <<'UNIT'
[Unit]
Description=Local LLM (llama-cpp-python)
After=network.target

[Service]
User=%i
WorkingDirectory=/home/%i/llm
ExecStart=/home/%i/llm/.venv/bin/python -m llama_cpp.server --model /home/%i/llm/models/tinyllama.Q4_K_M.gguf --host 0.0.0.0 --port 8000 --n-threads 8 --n_ctx 4096
Restart=on-failure

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now llama@$(whoami)

4) Swap models as your hardware allows:

  • Small CPUs: TinyLlama, Phi-2 Mini, Gemma 2B

  • Mid-range: Phi-3 Mini, Mistral 7B Instruct (quantized)

  • Larger RAM/VRAM: Llama 3 8B/70B (quantized), Mixtral

Note: Model licenses differ. Check terms before use.

Performance Tips

  • CPU threads: Set n_threads to your CPU’s core/threads sweet spot.

  • Quantization: Q4_K_M or Q5_K_M often balance speed and quality on CPUs.

  • Context length: Larger n_ctx uses more RAM; keep it reasonable for your tasks.

  • GPU: If you have NVIDIA, enabling CUDA for llama.cpp can drastically speed up inference on bigger models.

Conclusion and Next Steps (CTA)

You now have everything you need to run a local LLM on Linux—either through Ollama for speed and simplicity, or llama-cpp-python for pure-Python control. Pick a path, run a tiny model, and wire it into a script you already use:

  • Try Path 1: Install Ollama, run llama3, and hit it from Python and curl.

  • Try Path 2: Install llama-cpp-python, download a GGUF model, and start your own OpenAI-compatible API.

  • Wrap it in systemd, point your tools at http://localhost, and start automating real work.

When you’re ready, experiment with bigger models, tune performance, and build the CLI tools you wish existed. Your Linux box just learned a new trick.

Related posts