AI Agents with Local LLMs: Build Useful Bash Agents on Your Linux Box

If you’ve ever wished your shell could think, plan, and automate tasks without sending your data to the cloud, this is your moment. Local LLMs (large language models you run on your own machine) have become fast, affordable, and surprisingly capable. Pair them with a few Unix tools and you can build AI agents that search logs, summarize web pages, inspect system health, and more—entirely offline, with privacy by default.

This post shows you why local agents are worth your time and walks you through a minimal, hardenable Bash agent you can run today.

Why local AI agents are worth it

• Privacy and control: Keep code, logs, and credentials on your machine. No API keys, no egress.

• Cost and predictability: No per-token fees. Your hardware, your schedule.

• Latency and reliability: Works offline. Snappy responses for day-to-day ops work.

• Linux synergy: LLMs shine when they can call shell tools—grep, curl, awk, journalctl—exactly what Linux does best.

What you’ll build

• A tiny Bash agent that talks to a local LLM

• A safe, allowlisted tool-calling loop (search files, fetch a URL, run a few system commands)

• Real-world examples, plus simple hardening tips

---

1) Install a local LLM runtime

The easiest path is Ollama, which runs models locally and exposes a simple HTTP API. Optionally, you can build llama.cpp from source.

Option A: Ollama (recommended)

Install prerequisites if needed.

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl jq ripgrep

• Fedora/RHEL (dnf):

sudo dnf install -y curl jq ripgrep

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl jq ripgrep

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Pull a capable, compact model (you can change later):

ollama pull llama3.1:8b

Confirm it works:

curl -s http://localhost:11434/api/version

Option B: llama.cpp (advanced, from source)

Install build tools and git.

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git build-essential cmake

• Fedora/RHEL (dnf):

sudo dnf install -y git @development-tools cmake

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y git gcc gcc-c++ make cmake

Build:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j

Download a GGUF model (example uses TheBloke’s mirrors or official model sources) and start the HTTP server:

# Example: place your *.gguf model in ./models and start the server
./server -m ./models/YourModel.gguf -c 4096 --port 8080

If you use llama.cpp server, just swap the API URL in the scripts below.

---

2) Give your agent some tools

We’ll use jq for JSON, ripgrep for fast search, and curl for HTTP. If you haven’t already:

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl jq ripgrep

• Fedora/RHEL (dnf):

sudo dnf install -y curl jq ripgrep

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl jq ripgrep

Optional niceties: bat, fd, html2text, or lynx to pretty-print and de-HTML content.

---

3) A minimal, safe Bash agent

This script implements a small “tool calling” loop. The model chooses an action by emitting JSON. We validate and run only allowlisted commands, feed the result back, and let the model finish with a final answer.

Save as agent.sh and make executable.

#!/usr/bin/env bash
set -euo pipefail

# Config
MODEL="${MODEL:-llama3.1:8b}"
API="${API:-http://localhost:11434/api/chat}"   # Ollama chat endpoint
MAX_STEPS="${MAX_STEPS:-3}"
TMP_DIR="${TMP_DIR:-/tmp/local-agent}"
mkdir -p "$TMP_DIR"

# Allowlisted commands for 'run_cmd'
ALLOWED_CMDS=("uptime" "df" "free" "uname")
# Keep outputs to a sane size
MAX_TOOL_BYTES=20000

json_escape() {
  # Escape for JSON string context
  jq -Rs .
}

allow_run_cmd() {
  local name="$1"; shift || true
  for c in "${ALLOWED_CMDS[@]}"; do
    if [[ "$name" == "$c" ]]; then return 0; fi
  done
  return 1
}

tool_search_files() {
  local pattern="$1"
  local path="${2:-.}"
  local context="${3:-1}"
  rg --no-messages -n -C "$context" -S "$pattern" "$path" | head -c "$MAX_TOOL_BYTES" || true
}

tool_web_get() {
  local url="$1"
  curl -fsSL --max-time 10 "$url" | head -c "$MAX_TOOL_BYTES" || true
}

tool_run_cmd() {
  local name="$1"; shift || true
  if ! allow_run_cmd "$name"; then
    echo "Command not allowed: $name"; return 1
  fi
  # Ensure args are treated as separate words; disallow dangerous characters
  local safe_args=()
  for a in "$@"; do
    if [[ "$a" =~ [\;\|\&\>\\`] ]]; then
      echo "Blocked unsafe arg: $a"; return 1
    fi
    safe_args+=("$a")
  done
  "$name" "${safe_args[@]}" 2>&1 | head -c "$MAX_TOOL_BYTES"
}

system_prompt='You are a careful Linux command-line AI agent.
You can choose one of these actions:

- "search_files": args {"pattern": "...", "path": "/path/optional", "context": 1-5}

- "web_get": args {"url": "https://..."}

- "run_cmd": args {"name": "uptime|df|free|uname", "args": ["...optional..."]}

- "final": args {"answer": "final user-facing answer"}
Rules:

- Always respond with a single compact JSON object: {"action":"...","args":{...},"explanation":"short reason"}.

- If you need more info from a tool, choose a tool action. When enough, choose "final".

- Keep outputs small and focused.'

call_llm() {
  local messages_json="$1"
  curl -s "$API" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg model "$MODEL" --argjson messages "$messages_json" \
          '{model:$model, messages:$messages, stream:false}')" \
    | jq -r '.message.content'
}

run_agent() {
  local user_input="$1"

  # Build message history
  local messages
  messages=$(jq -n --arg sys "$system_prompt" --arg user "$user_input" \
    '[{"role":"system","content":$sys},{"role":"user","content":$user}]')

  for (( step=1; step<=MAX_STEPS; step++ )); do
    local content
    content="$(call_llm "$messages")"

    # Try to parse assistant content as JSON
    if ! echo "$content" | jq -e . >/dev/null 2>&1; then
      # If the model didn't return JSON, wrap it as final
      echo "$content"
      return 0
    fi

    local action
    action="$(echo "$content" | jq -r '.action // empty')"

    case "$action" in
      search_files)
        local pattern path context
        pattern="$(echo "$content" | jq -r '.args.pattern // ""')"
        path="$(echo "$content" | jq -r '.args.path // "."')"
        context="$(echo "$content" | jq -r '.args.context // 1')"
        out="$(tool_search_files "$pattern" "$path" "$context" | head -c "$MAX_TOOL_BYTES")"
        ;;

      web_get)
        local url
        url="$(echo "$content" | jq -r '.args.url // ""')"
        out="$(tool_web_get "$url" | head -c "$MAX_TOOL_BYTES")"
        ;;

      run_cmd)
        local name args_json
        name="$(echo "$content" | jq -r '.args.name // ""')"
        mapfile -t args < <(echo "$content" | jq -r '.args.args // [] | .[]')
        if out="$(tool_run_cmd "$name" "${args[@]}" | head -c "$MAX_TOOL_BYTES")"; then
          :
        else
          out="Tool error or blocked."
        fi
        ;;

      final)
        echo "$(echo "$content" | jq -r '.args.answer // empty')"
        return 0
        ;;

      *)
        echo "Unknown or missing action. Model said:"
        echo "$content"
        return 1
        ;;
    esac

    # Append tool result back to the conversation
    # We label it as a user message with a clear prefix; the system prompt already explains this pattern.
    local tool_msg="TOOL_RESULT:
$(echo "$out")"
    messages="$(echo "$messages" | jq --arg c "$content" --arg t "$tool_msg" \
      '. + [{"role":"assistant","content":$c},{"role":"user","content":$t}]')"
  done

  echo "Reached step limit without final answer."
  return 1
}

if [[ "${1-}" == "--demo" ]]; then
  run_agent "Find recent errors in /var/log and summarize top issues."
else
  if [[ $# -eq 0 ]]; then
    echo "Usage: $0 \"your question or task\""
    echo "Example: $0 \"Check disk space and tell me if any partition is >80% full\""
    exit 1
  fi
  run_agent "$*"
fi

Usage:

chmod +x agent.sh
./agent.sh "Search /var/log for 'error' and summarize the top 3 recurring messages"
./agent.sh "Fetch https://www.kernel.org and summarize the latest stable release"
./agent.sh "Run df -h and tell me which partitions exceed 80% usage"

Notes:

• This is a minimal loop for clarity. You can extend it with more tools, e.g., journalctl, ps, ss, kubectl, etc.

• Never run as root. Keep a tight allowlist. Consider running in a dedicated, unprivileged user or container/VM if you expand capabilities.

---

4) Real-world examples you can try

• Log triage (offline):

  • Prompt: “Scan /var/log for ‘failed’ and summarize probable causes and remediation steps.”
  • The agent uses search_files, returns context snips, then finalizes with a concise summary.

• Quick system health:

  • Prompt: “Run ‘uptime’, ‘free -h’, and ‘df -h’. Are we constrained on CPU, RAM, or disk? Give recommendations.”
  • The agent uses run_cmd on allowlisted commands and provides an actionable conclusion.

• Web note-taking:

  • Prompt: “Fetch https://www.openssh.com/releasenotes.html and summarize the key security fixes in 10 bullets.”
  • The agent uses web_get, then condenses to a readable summary. Add html-to-text if you want richer parsing.

• Focused code search:

  • Prompt: “In ./src, find references to TLS version settings and list files + context that touch TLS 1.0.”
  • The agent calls search_files with a pattern and path, returns context blocks for quick refactors.

---

5) Hardening and performance tips

• Least privilege:

  • Run as a non-root user.
  • Keep a strict allowlist for commands and validate args.
  • Add timeouts: prepend timeout 10s to risky tools.
  • Limit output: the script already truncates tool outputs.

• Sandbox:

  • Consider Firejail, containers, or a VM if you add file-write or network-heavy tools.

• Model choice:

  • Start with 7–8B models for speed on laptops; move to 13B for better reasoning if you have RAM.
  • Quantized models (e.g., Q4_K_M) run faster with smaller footprints.

• Caching and context:

  • Keep prompts short and focused.
  • For repeated tasks, store context snippets in files and feed them explicitly.

• Swap runtimes easily:

  • If using llama.cpp server on port 8080, set:
  • API=http://localhost:8080/v1/chat/completions if using its OpenAI-compatible endpoint
  • Adjust the request body accordingly or add a small adapter function.

---

Troubleshooting

• Ollama not responding:

systemctl status ollama
journalctl -u ollama -e

• Large outputs clipped:

  • Increase MAX_TOOL_BYTES at the top of the script (be cautious).

• Model replies not JSON:

  • Models sometimes drift. Tighten the system prompt or add a lightweight “repair” step that extracts JSON via jq -R and jq -r.

---

Conclusion and next steps

You now have a working, private AI agent running entirely on Linux with a few dozen lines of Bash. From here:

• Add new safe tools for your environment: journalctl, ps, ss, kubectl, terraform, aws s3 ls, etc.

• Create task-specific wrappers: logs-agent.sh, cluster-agent.sh, sec-audit-agent.sh.

• Experiment with different models in Ollama: mistral, llama3.1:8b-instruct, or domain-specific variants.

Call to action:

• Install Ollama, drop the agent.sh script into your bin, and ask it to investigate your logs right now. Then iterate—one tool at a time—until it becomes the teammate you always wanted, running locally under your control.