Posted on
• Artificial Intelligence

Creating Offline Artificial Intelligence Automation on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Creating Offline Artificial Intelligence Automation on Linux

Want private, fast, and reliable AI that doesn’t phone home? You can run modern language and speech models entirely on your Linux box and wire them into Bash for real, hands-free automation—no cloud, no API bills, and no internet required.

In this guide, you’ll set up an offline LLM and offline speech-to-text, then glue them together with Bash to build practical automations. You’ll also see how to schedule and harden them for daily use.

Why offline AI on Linux?

  • Privacy and control: Keep your code, logs, and voice data local.

  • Reliability and speed: No API latency or outage risk; inference runs at LAN speeds.

  • Cost: One-time setup beats usage-based billing.

  • Maturity: Tools like Ollama, llama.cpp, and whisper.cpp are battle-tested and easy to script.

What you’ll build:

  • A local LLM that can analyze text and suggest shell commands.

  • An offline speech-to-text pipeline to talk to your machine.

  • A Bash automation script that transcribes a request, asks the LLM for a safe command, and optionally runs it.

  • A systemd timer or cron job to schedule AI-powered maintenance checks.

1) Prepare your system

We’ll install common build tools and CLI helpers used throughout.

Apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y git build-essential cmake curl wget python3 python3-venv sox alsa-utils jq make

DNF (Fedora/RHEL/CentOS Stream):

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git cmake curl wget python3 python3-venv sox alsa-utils jq make

Zypper (openSUSE Leap/Tumbleweed):

sudo zypper refresh
sudo zypper install -y git gcc gcc-c++ cmake curl wget python3 python3-venv sox alsa-utils jq make

Notes:

  • alsa-utils provides arecord for simple microphone capture.

  • jq is used to parse and validate JSON from the LLM.

  • If you plan to use GPU acceleration later, also install your vendor’s CUDA/ROCm/OpenCL stack.

2) Install a fully local LLM

You have two popular options. Ollama is easiest; llama.cpp offers fine-grained control and tiny footprints.

Option A: Ollama (recommended for simplicity)

Ollama manages models and runs a local HTTP API at port 11434.

Install:

curl -fsSL https://ollama.com/install.sh | sh

Start the service:

sudo systemctl enable --now ollama

Pull a small, capable model (fits on most laptops):

ollama pull llama3.2:3b

Test it:

ollama run llama3.2:3b "Explain bash here-strings in two sentences."

If you see a coherent response, your local LLM is ready.

Option B: llama.cpp (build from source, minimal footprint)

Build:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

Get a GGUF model (ensure you respect licenses; place file locally):

mkdir -p ~/models
# Put your *.gguf model in ~/models, e.g. llama-3.2-3b-instruct.Q4_K_M.gguf

Run the HTTP server:

./server -m ~/models/llama-3.2-3b-instruct.Q4_K_M.gguf --port 8080

The server exposes endpoints like /completion and /chat. You can curl them similarly to Ollama (payload shape differs; see repo docs).

3) Add offline speech-to-text with whisper.cpp

We’ll compile whisper.cpp and use arecord to capture 16 kHz mono WAV (no external libraries required).

Build whisper.cpp:

git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make -j

Download an English model (base.en is small and accurate enough for commands):

cd whisper.cpp
bash ./models/download-ggml-model.sh base.en

Record and transcribe a test:

# Record 5 seconds of audio from default ALSA device
arecord -f S16_LE -r 16000 -c 1 /tmp/clip.wav -d 5

# Transcribe
./main -m models/ggml-base.en.bin -f /tmp/clip.wav -of /tmp/transcript.txt

# Show result
cat /tmp/transcript.txt

If you get readable text, you’re ready to wire voice into your offline assistant.

4) Wire it together with Bash: voice → LLM → safe command execution

This example:

  • Records your voice,

  • Transcribes it locally with whisper.cpp,

  • Asks your local LLM (Ollama) to propose a safe shell command in strict JSON,

  • Asks for confirmation, then executes it.

Create ~/bin/offline_voice_agent.sh:

#!/usr/bin/env bash
set -euo pipefail

# Config
DURATION="${DURATION:-6}"                   # seconds to record
AUDIO="/tmp/voice_cmd.wav"
TRANSCRIPT="/tmp/voice_cmd.txt"
WHISPER_DIR="${WHISPER_DIR:-$HOME/whisper.cpp}"
WHISPER_MODEL="${WHISPER_MODEL:-$WHISPER_DIR/models/ggml-base.en.bin}"
OLLAMA_MODEL="${OLLAMA_MODEL:-llama3.2:3b}"
DRY_RUN="${DRY_RUN:-true}"                 # set to "false" to allow execution
ALLOWLIST="${ALLOWLIST:-apt|dnf|zypper|ls|cat|tail|head|du|df|free|journalctl|systemctl|nmcli|ip|ping}"

# Ensure deps
command -v arecord >/dev/null || { echo "arecord (alsa-utils) is required"; exit 1; }
test -x "$WHISPER_DIR/main" || { echo "Build whisper.cpp and set WHISPER_DIR"; exit 1; }
command -v jq >/dev/null || { echo "jq is required"; exit 1; }
command -v ollama >/dev/null || { echo "ollama is required"; exit 1; }

echo "[1/4] Recording ${DURATION}s of audio..."
arecord -f S16_LE -r 16000 -c 1 "$AUDIO" -d "$DURATION"

echo "[2/4] Transcribing speech (whisper.cpp)..."
"$WHISPER_DIR/main" -m "$WHISPER_MODEL" -f "$AUDIO" -of "$TRANSCRIPT" >/dev/null
QUERY="$(tr -d '\0' < "$TRANSCRIPT" | tr '\n' ' ' | sed 's/  */ /g')"
echo "You said: $QUERY"

# System prompt to enforce STRICT JSON
read -r -d '' SYS <<'EOF'
You are an offline Linux command assistant. Return ONLY JSON:
{"command": "...", "reason": "..."}

- command: a single safe shell command that addresses the user's request using common tools (bash coreutils, package managers, systemctl, journalctl, ip, df, du, free).

- Prefer read-only commands unless explicitly asked to make changes.

- NEVER include explanations outside JSON. No backticks. No multiple commands joined with && or ;.

- Use the user's distro package manager only if asked to install.

- Assume no internet.
EOF

echo "[3/4] Asking local LLM ($OLLAMA_MODEL) for a command..."
RESP="$(curl -s http://localhost:11434/api/generate -d @- <<JSON
{
  "model": "$OLLAMA_MODEL",
  "prompt": "$SYS\nUser: $QUERY\nAssistant:",
  "options": { "temperature": 0.1 },
  "stream": false
}
JSON
)"

# Extract the model's text field
TEXT="$(echo "$RESP" | jq -r '.response')"

# Validate strict JSON and parse command
if ! echo "$TEXT" | jq -e . >/dev/null 2>&1; then
  echo "Model did not return valid JSON. Raw output:"
  echo "$TEXT"
  exit 1
fi

CMD="$(echo "$TEXT" | jq -r '.command')"
REASON="$(echo "$TEXT" | jq -r '.reason')"

if [[ -z "$CMD" || "$CMD" == "null" ]]; then
  echo "No command proposed. Full payload:"
  echo "$TEXT"
  exit 1
fi

# Basic allowlist
if ! [[ "$CMD" =~ ^($ALLOWLIST)\b ]]; then
  echo "Proposed command not in allowlist: $CMD"
  echo "Reason: $REASON"
  exit 1
fi

echo "[4/4] Proposed command:"
echo "  $CMD"
echo "Reason: $REASON"

read -rp "Execute? [y/N] " ans
if [[ "$ans" =~ ^[Yy]$ && "$DRY_RUN" != "true" ]]; then
  bash -lc "$CMD"
else
  echo "Dry-run or not confirmed. Skipping execution."
fi

Make it executable:

chmod +x ~/bin/offline_voice_agent.sh

Run it:

~/bin/offline_voice_agent.sh

Try prompts like:

  • “How full is my disk?” → df -h

  • “Show the last 50 kernel log lines” → journalctl -k -n 50

  • “Check current memory usage” → free -h

Hygiene and safety tips:

  • Keep DRY_RUN=true while testing.

  • Tighten ALLOWLIST to the smallest set you actually need.

  • Consider a review step that prints the command, diffs expected changes, or runs inside a container/namespace.

5) Schedule and operationalize

You can run AI automations on a schedule (health checks, log summaries) entirely offline.

Example: a daily log summary using your local LLM.

Create ~/bin/offline_log_summary.sh:

#!/usr/bin/env bash
set -euo pipefail

MODEL="${MODEL:-llama3.2:3b}"

# Collect recent logs (adjust scope and amount)
journalctl -p 4 -n 500 --no-pager > /tmp/journal_excerpt.txt

read -r -d '' PROMPT <<'EOF'
Summarize the key issues, repeating errors, and actionable steps from the following Linux logs.
Return a concise markdown list with:

- Top problems (short title)

- Probable causes

- Suggested next commands to investigate (read-only)
EOF

DATA="$(printf "%s\n\nLogs:\n%s" "$PROMPT" "$(sed 's/"/\\"/g' /tmp/journal_excerpt.txt)")"

RESP="$(curl -s http://localhost:11434/api/generate -d @- <<JSON
{
  "model": "$MODEL",
  "prompt": "$DATA",
  "options": { "temperature": 0.2 },
  "stream": false
}
JSON
)"

SUMMARY="$(echo "$RESP" | jq -r '.response')"
echo "=== $(date) ===" >> $HOME/offline_log_summaries.md
echo "$SUMMARY" >> $HOME/offline_log_summaries.md
echo >> $HOME/offline_log_summaries.md

Make executable:

chmod +x ~/bin/offline_log_summary.sh

Option A: systemd user timer (no root needed):

mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/offline-log-summary.service <<'UNIT'
[Unit]
Description=Offline log summary via local LLM

[Service]
Type=oneshot
ExecStart=%h/bin/offline_log_summary.sh
UNIT

cat > ~/.config/systemd/user/offline-log-summary.timer <<'UNIT'
[Unit]
Description=Run offline log summary daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
UNIT

systemctl --user daemon-reload
systemctl --user enable --now offline-log-summary.timer

Option B: cron (system-wide):

crontab -e
# Add:
@daily $HOME/bin/offline_log_summary.sh >> $HOME/offline_log_summary.log 2>&1

Real-world examples you can build next

1) Voice-driven maintenance:

  • “Update package index” (read-only check first):

    • Debian/Ubuntu: apt update --print-uris
    • Fedora/RHEL: dnf check-update
    • openSUSE: zypper refresh --force

  • You can allow write operations only after a second explicit confirmation.

2) Local knowledge base Q&A:

  • Point the LLM at /var/log, config directories, or project docs. Use retrieval augmented generation (RAG) with a local vector DB (e.g., sqlite + embeddings generated offline).

3) Air-gapped CI helper:

  • Summarize test logs, suggest failing areas, propose targeted grep or pytest -k invocations—no outbound requests.

Performance tips

  • Choose the right size: 3B–7B instruction-tuned models are great for command generation and summaries on CPU-only machines.

  • Quantization: Use Q4_K_M or similar for speed on modest hardware.

  • Threading: Set OMP_NUM_THREADS=$(nproc) or equivalent for llama.cpp/whisper.cpp if needed.

  • GPU acceleration: llama.cpp and whisper.cpp support CUDA/Metal/ROCm builds; enable if you have the hardware.

Troubleshooting

  • Microphone device:

    • List devices: arecord -l
    • Choose device: arecord -D plughw:0,0 -f S16_LE -r 16000 -c 1 /tmp/clip.wav -d 5

  • Ollama not running:

    • systemctl status ollama (or journalctl -u ollama)

  • Model too slow:

    • Try a smaller or more aggressively quantized model.

  • Whisper accuracy:

    • Use a larger model (e.g., small.en) or speak closer to the mic.

Conclusion and next steps

You now have an end-to-end, completely offline AI stack on Linux:

  • Local LLM via Ollama or llama.cpp

  • Local speech-to-text via whisper.cpp

  • Bash glue that turns voice or text into safe, auditable automation

  • Schedulable jobs for continuous insights—no cloud needed

Call to action:

  • Start with the voice agent and keep DRY_RUN enabled.

  • Tighten allowlists and add unit tests for your automations.

  • Expand to log triage, config audits, or RAG over your docs—entirely offline.

Your Linux box just became an AI workstation that respects your privacy. Build something useful today.

Related posts