Posted on
• Artificial Intelligence

Private Artificial Intelligence with Local LLMs

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Private Artificial Intelligence with Local LLMs: A Bash-First Guide

What if you could ask an AI to summarize your server logs, review a patch, or redact PII—without sending a single byte to the cloud? Local Large Language Models (LLMs) make that possible. In this guide, you’ll get a practical, Linux-and-Bash-centric path to stand up a private AI stack on your workstation or server, integrate it into shell workflows, and keep your data where it belongs: on your machine.

The problem (and the value)

  • Cloud AI is convenient—but it can be a compliance, privacy, and cost landmine.

  • Latency and availability depend on someone else’s API.

  • Many tasks don’t require a hyperscale GPU cluster; a quantized 7B–8B model can do a lot locally.

Local LLMs let you:

  • Keep data on-prem or on-device (PII, source code, logs).

  • Control cost (one-time hardware + open models).

  • Gain reliability (works offline, even air-gapped).

  • Integrate directly with Linux tooling and automation.

Why this is viable now

  • Efficient inference engines (ollama, llama.cpp) and quantized models run on CPUs/GPUs you likely have.

  • Modern 7B–8B instruction-tuned models handle summarization, Q&A, drafting, basic coding, and classification surprisingly well.

  • Tooling has matured, with simple CLIs and local REST APIs designed for automation.

Quick start: Private LLMs with Ollama

Ollama is the easiest path: one command to install, one to pull a model, one to run it. It exposes a local REST API on 127.0.0.1:11434 by default—great for scripting.

Prerequisites: curl and jq for examples.

Install prerequisites:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl jq
  • Fedora/RHEL (dnf):
sudo dnf install -y curl jq
  • openSUSE (zypper):
sudo zypper install -y curl jq

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Pull and run a model (example: Llama 3 8B Instruct):

ollama pull llama3
ollama run llama3

Non-interactive prompt (good for scripts):

ollama run llama3 -p "Explain what cgroups are in Linux in 3 bullet points."

Call the local REST API:

curl -s http://localhost:11434/api/generate \
  -d '{"model":"llama3","prompt":"One-sentence summary of POSIX signals."}'

Tip: Explore available models:

ollama list
# and visit https://ollama.com/library for names and sizes

3–5 actionable steps to put local LLMs to work

1) Integrate with Bash: Summarize logs, triage, and classify

Slurp any text into a prompt using jq -Rs (read raw, quote safely) and hit the API.

  • Summarize recent syslog:
journalctl -n 300 -xb | \
jq -Rs --arg model llama3 '{model: $model, prompt: "Summarize these logs in 5 bullet points, highlighting errors and anomalies:\n\n" + .}' | \
curl -s http://localhost:11434/api/generate -d @- | \
jq -r '.response'
  • Classify lines (e.g., error vs. info) and output JSONL:
tail -n 200 /var/log/syslog | \
jq -Rs --arg model llama3 '{model: $model, prompt: "Label each line as ERROR, WARNING, or INFO. Output JSON with fields: line, label. Text:\n" + .}' | \
curl -s http://localhost:11434/api/generate -d @- | \
jq -r '.response'
  • Redact PII from a file (emails, phone numbers, IPs):
input="data.txt"
jq -n --arg model llama3 --rawfile t "$input" \
'{
  model: $model,
  prompt: "Redact all PII (emails, phone numbers, IPv4/IPv6, names if confident) using [REDACTED] and preserve formatting:\n\n" + $t
}' | curl -s http://localhost:11434/api/generate -d @- | jq -r '.response'
  • Review a patch:
git diff HEAD~1..HEAD | \
jq -Rs --arg model llama3 '{model: $model, prompt: "Review this diff for correctness, edge cases, security, and style. Output issues as bullet points with file:line references where possible:\n\n" + .}' | \
curl -s http://localhost:11434/api/generate -d @- | \
jq -r '.response'

2) Choose the right model and size it

  • Start with instruction-tuned 7B–8B models for general tasks (e.g., llama3, mistral, qwen2.5 variants).

  • Quantized models (e.g., Q4_K_M or Q5) fit in 4–8 GB VRAM or 8–12 GB RAM; they’re fast enough on modern CPUs.

  • If you have a GPU, Ollama will use CUDA/ROCm/MPS automatically when supported. For pure CPU: it still works, just slower.

  • Check model cards and licenses before use in production.

3) Keep it private by default

  • Ollama binds to 127.0.0.1:11434 by default. Verify:
ss -ltnp | grep 11434
  • Force local binding and set resource limits via a systemd drop-in:
sudo systemctl edit ollama
# Paste the following, then save/exit:
[Service]
Environment=OLLAMA_HOST=127.0.0.1
# Optional: move models to a dedicated path with correct permissions
# Environment=OLLAMA_MODELS=/srv/ollama/models
MemoryMax=8G
TasksMax=2048

Apply:

sudo systemctl daemon-reload
sudo systemctl restart ollama
  • Use your firewall to block unintended access and keep the service local-only.

4) Automate safely and reproducibly

  • Pin model versions:
ollama pull llama3:latest
# or use a specific tag if the model provides one
  • Script with clear prompts and size limits. Example timeout and token limits via API:
jq -n \
  --arg model llama3 \
  --arg prompt "Summarize in <=120 words." \
'{
  model: $model,
  prompt: $prompt,
  options: { num_predict: 256 }
}' | curl -m 30 -s http://localhost:11434/api/generate -d @- | jq -r '.response'
  • Log prompts and responses to help audit and tune in regulated environments.

5) Optional: Build llama.cpp for maximum control

If you prefer a lightweight, zero-daemon binary with GGUF models, build llama.cpp.

Install build dependencies:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y build-essential cmake git python3 python3-pip libopenblas-dev
  • Fedora/RHEL (dnf):
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y cmake git python3 python3-pip openblas-devel
  • openSUSE (zypper):
sudo zypper install -y gcc gcc-c++ make cmake git python3 python3-pip openblas-devel
# or: sudo zypper install -t pattern devel_C_C++

Build:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -S .
cmake --build build -j

Run with a GGUF model you’ve legally obtained:

./build/bin/llama-cli -m /path/to/model.gguf -p "Explain cgroups v2 in 5 sentences."

Notes:

  • CPU-only works everywhere; add GPU backends by enabling CUDA/ROCm/MPS at compile time (see llama.cpp README).

  • Quantized GGUF files are widely available; always check the model’s license and terms.

Real-world use cases you can run today

  • Offline incident triage:
journalctl -u nginx -n 500 | \
jq -Rs --arg model llama3 '{model: $model, prompt: "From these logs, what caused the 5xx spike? Provide likely root causes and next steps:\n\n" + .}' | \
curl -s http://localhost:11434/api/generate -d @- | jq -r '.response'
  • Summarize a PDF report for your team:
sudo apt install -y poppler-utils   # dnf: sudo dnf install -y poppler-utils ; zypper: sudo zypper install -y poppler-tools
pdftotext report.pdf - | \
jq -Rs --arg model llama3 '{model: $model, prompt: "Create an executive summary with 5 bullets and an action list:\n\n" + .}' | \
curl -s http://localhost:11434/api/generate -d @- | jq -r '.response'
  • Draft shell-safe commands from a question:
question="Find the 10 largest files under /var, excluding docker directories."
jq -n --arg model llama3 --arg q "$question" \
'{
  model: $model,
  prompt: "Write a single bash one-liner (no explanations) to: " + $q
}' | curl -s http://localhost:11434/api/generate -d @- | jq -r '.response'

Always review generated commands before running them.

Troubleshooting tips

  • Model too slow? Try a smaller model or a more aggressive quantization (Q4 vs. Q5/Q8). Close other RAM-heavy apps.

  • OOM or crashes? Lower num_predict, use a smaller model, or add swap.

  • GPU not used? Update drivers/CUDA/ROCm; check logs. CPU still works as fallback.

  • Prompt quality matters: be explicit about format, length, and constraints.

Conclusion and next steps (CTA)

Private AI doesn’t require a cloud contract—it needs a terminal and a model. You now have:

  • A local LLM runtime (Ollama) with a REST API.

  • Bash-friendly patterns to summarize, classify, redact, and review.

  • Hardening steps to keep everything on localhost.

  • An optional path (llama.cpp) for ultimate control.

Your next steps: 1) Install Ollama and jq. 2) Pull a model like llama3. 3) Drop one of the pipelines above into a script that solves a real task today (log triage, PII redaction, or patch review). 4) Iterate on prompts and model choice for your hardware.

If you found this useful, turn one of the examples into a cron job or a Git hook—and enjoy private, programmable AI that lives where your data does: on Linux.

Related posts