Building AI Agents with Ollama on Linux (with Bash)

Want an AI agent that runs locally, respects your privacy, doesn’t need API keys, and can actually use your terminal tools? That’s exactly what you get by pairing Ollama with a bit of Bash. In this post, you’ll go from zero to a working, tool-using AI agent on Linux—no cloud required.

We’ll cover:

• Why local AI agents are worth your time

• Installing Ollama on Debian/Ubuntu, Fedora/RHEL, and openSUSE

• Pulling a model and running the API

• Building a real Bash agent that can safely call system tools

• Practical tips for security and performance

Why build agents with Ollama?

• Privacy and control: Everything runs on your machine. Perfect for code, logs, or customer data you can’t send to third-party APIs.

• Latency and cost: No per-token charges. Responses are fast and offline-capable once models are downloaded.

• Unix composability: Ollama exposes a simple HTTP API. That means you can script it with curl and jq—classic Linux workflows.

What we’ll build

A CLI agent that can:

• List directories and read files on request

• Run a safe, read-only subset of shell commands

• Use AI to decide when to call tools, then explain the results in plain English

We’ll do it in Bash using the Ollama HTTP API and jq.

---

1) Install prerequisites and Ollama

First, install basic CLI tools (curl, jq) and then install Ollama via the official script. There aren’t official apt/dnf/zypper packages for Ollama itself at the time of writing—use the installer below or Docker.

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl ca-certificates jq
curl -fsSL https://ollama.com/install.sh | sh

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y curl ca-certificates jq
curl -fsSL https://ollama.com/install.sh | sh

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl ca-certificates jq
curl -fsSL https://ollama.com/install.sh | sh

Alternative: Docker (CPU-only or with NVIDIA GPU)

# CPU-only
docker run -d -p 11434:11434 --name ollama ollama/ollama

# NVIDIA GPU (requires nvidia-container-toolkit)
docker run -d --gpus all -p 11434:11434 --name ollama ollama/ollama

Tip: If you used Docker, exec into the container for commands like pulling models:

docker exec -it ollama bash

---

2) Start Ollama and pull a model

Ollama runs a local API on port 11434.

• If you installed with the script, start the server:

ollama serve

(You can also run it in the background or systemd; for a quick demo, foreground is fine.)

• Pull a model (choose a size that fits your hardware):

ollama pull llama3.1

Try it:

ollama run llama3.1 "Explain what Ollama is in one sentence."

Verify the API:

curl -s http://localhost:11434/api/tags

---

3) Build a minimal Bash agent that can use tools

We’ll use the chat API, define safe tools, and let the model decide whether to call them.

Create a file agent.sh:

#!/usr/bin/env bash
set -euo pipefail

MODEL="${MODEL:-llama3.1}"
API="${API:-http://localhost:11434/api/chat}"

# Tool definitions (JSON Schema). The model can request these.
TOOLS='[
  {
    "type": "function",
    "function": {
      "name": "list_dir",
      "description": "List files in a directory non-recursively.",
      "parameters": {
        "type": "object",
        "properties": {
          "path": { "type": "string", "description": "Directory to list" }
        },
        "required": ["path"]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "read_file",
      "description": "Read a small text file (first 2000 bytes).",
      "parameters": {
        "type": "object",
        "properties": {
          "path": { "type": "string", "description": "Path to file" }
        },
        "required": ["path"]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "run_shell_readonly",
      "description": "Run a safe, read-only shell command. Allowed: ls, cat, head, tail, grep, du, df, uname, whoami, date.",
      "parameters": {
        "type": "object",
        "properties": {
          "command": { "type": "string", "description": "The command to run with arguments" }
        },
        "required": ["command"]
      }
    }
  }
]'

SYSTEM_PROMPT="You are a helpful Linux assistant. Prefer explaining steps. Only call tools if necessary. Never run destructive commands."

if [[ $# -eq 0 ]]; then
  echo "Usage: $0 \"your question or task\"" >&2
  exit 1
fi

USER_PROMPT="$*"

# 1) Ask the model. It may reply with a tool call.
REQUEST=$(jq -n \
  --arg model "$MODEL" \
  --arg sys "$SYSTEM_PROMPT" \
  --arg user "$USER_PROMPT" \
  --argjson tools "$TOOLS" \
  '{model:$model, messages:[{"role":"system","content":$sys},{"role":"user","content":$user}], tools:$tools, stream:false}')

RESPONSE=$(curl -s "$API" -H "Content-Type: application/json" -d "$REQUEST")

# Extract potential tool call.
TOOL_NAME=$(echo "$RESPONSE" | jq -r '.message.tool_calls[0].function.name // empty')
ARGS_STR=$(echo "$RESPONSE"  | jq -r '.message.tool_calls[0].function.arguments // empty')

if [[ -z "$TOOL_NAME" ]]; then
  # No tool call: print the assistant message.
  echo "$RESPONSE" | jq -r '.message.content'
  exit 0
fi

# Parse arguments; they are often a JSON string.
ARGS_JSON=$(echo "$ARGS_STR" | jq -rc 'try fromjson catch {}')

# Implement tools
run_list_dir() {
  local path
  path=$(echo "$ARGS_JSON" | jq -r '.path // "."')
  # Resolve to absolute path for safety; you might restrict to $HOME
  if ! realpath "$path" >/dev/null 2>&1; then
    echo "Invalid path: $path"
    return 1
  fi
  ls -1A --group-directories-first --color=never -- "$path" 2>&1 | head -n 200
}

run_read_file() {
  local path
  path=$(echo "$ARGS_JSON" | jq -r '.path // ""')
  if [[ -z "$path" || ! -f "$path" ]]; then
    echo "File not found or not a regular file: $path"
    return 1
  fi
  head -c 2000 -- "$path" 2>&1
}

run_shell_readonly() {
  local cmd first
  cmd=$(echo "$ARGS_JSON" | jq -r '.command // ""')
  if [[ -z "$cmd" ]]; then
    echo "No command provided"
    return 1
  fi
  # Whitelist commands (first token only)
  read -r first _ <<<"$cmd"
  case "$first" in
    ls|cat|head|tail|grep|du|df|uname|whoami|date)
      eval "$cmd" 2>&1 | head -c 8000
      ;;
    *)
      echo "Command '$first' is not allowed (read-only whitelist)."
      return 1
      ;;
  esac
}

TOOL_OUTPUT=""
case "$TOOL_NAME" in
  list_dir) TOOL_OUTPUT=$(run_list_dir || true) ;;
  read_file) TOOL_OUTPUT=$(run_read_file || true) ;;
  run_shell_readonly) TOOL_OUTPUT=$(run_shell_readonly || true) ;;
  *) TOOL_OUTPUT="Requested unknown tool: $TOOL_NAME" ;;
esac

# 2) Send tool result back to the model for a final answer.
FOLLOW_UP=$(jq -n \
  --arg model "$MODEL" \
  --arg sys "$SYSTEM_PROMPT" \
  --arg user "$USER_PROMPT" \
  --arg tool_name "$TOOL_NAME" \
  --arg tool_out "$TOOL_OUTPUT" \
  '{
    model:$model,
    messages:[
      {"role":"system","content":$sys},
      {"role":"user","content":$user},
      {"role":"tool","name":$tool_name,"content":$tool_out}
    ],
    stream:false
  }')

FINAL=$(curl -s "$API" -H "Content-Type: application/json" -d "$FOLLOW_UP")
echo "$FINAL" | jq -r '.message.content'

Make it executable:

chmod +x agent.sh

Try it:

./agent.sh "What's taking up space under /var/log? Use a safe command and summarize."

The agent should choose a safe command (like du -h /var/log | sort -hr | head), run it via run_shell_readonly, then provide a human-readable summary.

Notes:

• The API response structure above reflects Ollama’s tool-calling pattern at time of writing. If it changes, inspect raw JSON with jq and adjust field names accordingly.

• For multi-step tool calls, expand the script to loop on tool_calls until none remain.

---

4) Real-world examples

• Investigate disk usage:

./agent.sh "Find the top 10 largest directories in my home folder and summarize. Use du safely."

• Skim logs:

./agent.sh "Look at /var/log/syslog and show the last 50 lines; call out any obvious errors."

• Quick system snapshot:

./agent.sh "Tell me my kernel version and disk free space, then explain if I'm low on space."

• Browse a codebase (read-only):

./agent.sh "List the top-level files under ./src and summarize the project layout."

Remember: keep tools read-only, and never run as root for exploratory tasks.

---

5) Tips: models, performance, and security

• Model choice:

  • llama3.1 (good general assistant)
  • mistral, qwen2.5, or coders for programming-heavy work
  • Smaller models (7–8B) fit most CPUs; larger models need more RAM/GPU VRAM.

• Performance knobs:

# Limit GPU layers or concurrency if you hit memory issues
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_GPU_OVERHEAD=0

• Run headless/remote:

# Listen on all interfaces (restrict with firewall/VPN!)
export OLLAMA_HOST=0.0.0.0:11434
ollama serve

• Security:

  • Keep a strict command whitelist (as shown).
  • Don’t run as root; consider a dedicated low-privilege user.
  • Sandbox tools further with containers, chroot, or firejail if needed.
  • Be explicit in the system prompt about safety constraints.

• Troubleshooting:

# Check if the API is alive
curl -s http://localhost:11434/api/tags | jq
# Test a basic chat completion
curl -s http://localhost:11434/api/chat -H "Content-Type: application/json" \
  -d '{"model":"llama3.1","messages":[{"role":"user","content":"Hello!"}], "stream":false}' | jq

---

Conclusion and next steps

You now have a local, private AI agent that can call Linux tools safely—all powered by Bash and Ollama. From here, you can:

• Add more tools (e.g., “search_code”, “git_diff”, “run_ansible_check”).

• Store conversation state to support multi-turn tasks.

• Wrap this in a TUI with fzf or a small web UI.

• Swap in a domain-specific model (coding, ops, docs QA) as needed.

Your move: 1) Install Ollama and jq, pull a model. 2) Run the agent.sh script and try the examples. 3) Extend the toolset to match your workflow.

If you build something cool—or run into snags—share it with the community. Happy hacking!