Posted on
• Artificial Intelligence

AI Agents on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI Agents on Linux: Turn Your Terminal Into a Teammate

What if your Linux shell had a tireless co-worker—one that reads docs, drafts scripts, summarizes logs, and even runs curated commands for you? That’s the promise of AI agents. Most tutorials hand-wave Linux specifics or default to GUI-first workflows. In this guide, we’ll keep it terminal-first, show why Linux is the best home for agents, and walk you through building a practical, local-first agent that runs right on your machine.

You’ll leave with:

  • A clear mental model for AI agents on Linux

  • A working terminal agent script using a local model (Ollama)

  • Steps to add tools safely, run as a service, and sandbox

Why AI Agents on Linux?

  • Local-first control and privacy: Keep sensitive code and logs off third-party clouds by running models locally.

  • Automation-native: Cron, systemd, shell pipelines—Linux is built for orchestrating long-running jobs and background workers.

  • Composability: Agents play nicely with text-first tools like grep, sed, jq, curl, and your editor of choice.

  • Portability: Package managers and reproducible environments make deploying agents to servers and workstations predictable.

Prerequisites: Install Core Utilities

We’ll use curl and jq for HTTP and JSON, Python for optional scripting, and basic build tools for compiling or installing native dependencies later.

Install the following with your package manager.

# apt (Debian/Ubuntu/Mint)
sudo apt update
sudo apt install -y git curl jq python3 python3-venv python3-pip build-essential cmake pkg-config

# dnf (Fedora/RHEL/CentOS Stream)
sudo dnf install -y git curl jq python3 python3-pip python3-virtualenv gcc gcc-c++ make cmake pkgconf-pkg-config

# zypper (openSUSE)
sudo zypper refresh
sudo zypper install -y git curl jq python3 python3-pip python3-virtualenv gcc gcc-c++ make cmake pkg-config

Option A (Local): Install Ollama and a Small Model

Ollama is a simple way to run LLMs locally with CPU or GPU acceleration.

# Install Ollama (Linux)
curl -fsSL https://ollama.com/install.sh | sh

Start the service and pull a lightweight model that runs on most laptops:

# Start the Ollama server (choose one)
ollama serve & disown
# or (if set up by the installer)
systemctl --user start ollama

# Pull a small model (adjust as you like)
ollama pull llama3.2:3b

Quick test:

ollama run llama3.2:3b "In one sentence, tell me what you can do."

Option B (Remote): Use an OpenAI-compatible API

If you prefer cloud models, you can point a script at an OpenAI-compatible endpoint.

Set your environment (example):

export OPENAI_API_KEY="sk-..."
export OPENAI_BASE_URL="https://api.openai.com/v1"   # change if using a compatible provider
export OPENAI_MODEL="gpt-4o-mini"                    # or another available model

We’ll include a tiny agent example for this path below.

Build a Minimal Terminal Agent (Local, via Ollama)

This is a compact Bash agent that talks to your local model through Ollama’s REST API. It’s stateless for simplicity; you can expand it to keep history per task.

#!/usr/bin/env bash
# file: agent.sh
set -euo pipefail

MODEL="${MODEL:-llama3.2:3b}"
API_URL="${API_URL:-http://localhost:11434}"
PROMPT="${*:-}"

if [[ -z "$PROMPT" ]]; then
  echo "Usage: ./agent.sh \"your question or task\""
  exit 1
fi

# Ensure server is up
if ! curl -fsS "$API_URL/api/version" >/dev/null 2>&1; then
  echo "Starting ollama server..."
  nohup ollama serve >/dev/null 2>&1 &
  # wait briefly for server
  for i in {1..20}; do
    sleep 0.2
    curl -fsS "$API_URL/api/version" >/dev/null 2>&1 && break
  done
fi

# Ensure the model exists locally
if ! ollama show "$MODEL" >/dev/null 2>&1; then
  echo "Pulling model $MODEL ..."
  ollama pull "$MODEL"
fi

# Ask the model (non-streaming for easy parsing)
RESP="$(curl -fsS -X POST "$API_URL/api/chat" \
  -H "Content-Type: application/json" \
  -d "$(jq -nc --arg m "$MODEL" --arg p "$PROMPT" \
      '{model:$m, stream:false, messages:[{role:"user", content:$p}]}')" \
)"

echo "$RESP" | jq -r '.message.content'

Try it:

chmod +x agent.sh
./agent.sh "Summarize the top 3 log lines from /var/log/syslog and suggest a next action."

Tip: Pipe inputs for quick transforms.

dmesg | tail -n 200 | ./agent.sh "Summarize this kernel output in bullet points."

Add Safe, Useful “Tools” (Command Execution With Guardrails)

Agents get powerful when they can call tools. But don’t give an LLM full shell access. Start with a tiny allowlist and require confirmation.

#!/usr/bin/env bash
# file: agent_tools.sh
set -euo pipefail

MODEL="${MODEL:-llama3.2:3b}"
API_URL="${API_URL:-http://localhost:11434}"
TASK="${*:-"List large files in the current directory and suggest cleanup."}"

allow() {
  # Allow only a few read-only commands with limited args
  # Expand carefully as you gain confidence.
  case "$1" in
    ls|cat|head|tail|wc|grep|rg|du|df|sed|awk|curl)
      return 0 ;;
    *)
      return 1 ;;
  esac
}

system_prompt='You are a cautious Linux assistant. When you need shell info, propose exactly ONE command prefixed by "CMD:" on a single line, using only safe read-only utilities (ls, cat, head, tail, wc, grep, du, df, sed, awk, curl). Keep commands short and avoid destructive flags. If unsure, ask for clarification.'

# Start server if needed
curl -fsS "$API_URL/api/version" >/dev/null 2>&1 || (nohup ollama serve >/dev/null 2>&1 & sleep 1)

BASE=$(jq -nc --arg m "$MODEL" --arg sp "$system_prompt" --arg t "$TASK" \
  '{model:$m, stream:false, messages:[
    {role:"system", content:$sp},
    {role:"user", content:$t}
  ]}')

RESP=$(curl -fsS -X POST "$API_URL/api/chat" -H "Content-Type: application/json" -d "$BASE")
TEXT=$(echo "$RESP" | jq -r '.message.content')

echo "Agent:"
echo "$TEXT"
CMD=$(grep -m1 '^CMD:' <<<"$TEXT" | sed 's/^CMD:\s*//')

if [[ -n "${CMD:-}" ]]; then
  echo
  echo "Proposed command:"
  echo "  $CMD"
  if allow "$(awk '{print $1}' <<<"$CMD")"; then
    read -rp "Run this command? [y/N] " yn
    if [[ "${yn,,}" == "y" ]]; then
      echo
      echo "----- output -----"
      bash -c "$CMD"
    else
      echo "Skipped."
    fi
  else
    echo "Blocked: command not in allowlist."
  fi
fi

Usage:

chmod +x agent_tools.sh
./agent_tools.sh "Find the 5 largest files under the current directory."

Important: Start strict. Only add commands you really need. Keep human-in-the-loop confirmation for anything that can mutate state.

Prefer Cloud? Minimal OpenAI-compatible Bash Agent

If using a remote endpoint, here’s a tiny script you can point at any OpenAI-compatible API:

#!/usr/bin/env bash
# file: api_agent.sh
set -euo pipefail

: "${OPENAI_API_KEY:?Set OPENAI_API_KEY}"
OPENAI_BASE_URL="${OPENAI_BASE_URL:-https://api.openai.com/v1}"
OPENAI_MODEL="${OPENAI_MODEL:-gpt-4o-mini}"
PROMPT="${*:-"Explain cgroups in one paragraph."}"

curl -fsS -X POST "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -nc --arg m "$OPENAI_MODEL" --arg p "$PROMPT" \
      '{model:$m, temperature:0.2, messages:[{role:"user", content:$p}] }')" \
| jq -r '.choices[0].message.content'

Make It Robust: Services, Limits, and Sandboxing

As agents become more capable, invest in guardrails and ops hygiene.

1) Run with resource limits

  • Quick limits with ulimit:
ulimit -v 2097152   # 2 GiB virtual memory
ulimit -t 60        # 60s CPU time
  • Per-invocation cgroup with systemd-run:
systemd-run --user --scope -p MemoryMax=2G -p CPUQuota=200% ./agent.sh "Summarize auth.log"

2) Don’t run as root

  • Create a dedicated, low-privilege user for agents and store minimal secrets.

3) Sandbox if tools are enabled

  • Podman containers are a great boundary.

Install Podman:

# apt
sudo apt install -y podman

# dnf
sudo dnf install -y podman

# zypper
sudo zypper install -y podman

Run the agent in a locked-down container (example skeleton):

podman run --rm -it --security-opt=no-new-privileges \
  --pids-limit=256 --memory=2g --cpus=2 \
  -v "$PWD":/work -w /work docker.io/library/debian:stable-slim \
  bash -lc 'apt-get update && apt-get install -y curl jq && ./agent.sh "Check disk usage safely"'

4) Optional extra sandbox: Firejail

# apt
sudo apt install -y firejail
# dnf
sudo dnf install -y firejail
# zypper
sudo zypper install -y firejail

Then:

firejail --private --net=none --caps.drop=all ./agent.sh "Local analysis only"

5) Schedule or always-on

  • Use systemd user services or timers to run agents periodically for log triage, backups verification, or report generation.

Example user service:

# ~/.config/systemd/user/agent.service
[Unit]
Description=Local AI Agent (Ollama)

[Service]
ExecStart=%h/bin/agent.sh "Daily: summarize system changes under /etc"
Environment=MODEL=llama3.2:3b
MemoryMax=2G
CPUQuota=200%
NoNewPrivileges=yes

[Install]
WantedBy=default.target

Enable it:

systemctl --user daemon-reload
systemctl --user enable --now agent.service

Real-World Uses You Can Try Today

  • Log whisperer: Pipe logs to your agent and ask for anomalies, summaries, or action items.

  • Ops explainer: Paste a failing CI log or kernel trace and ask for likely causes and next debug steps.

  • Doc scout: Have it draft README sections or inline comments from your existing shell scripts.

  • Disk janitor (with guardrails): Let it propose read-only commands to identify big files and stale caches; you confirm before deletion.

Conclusion and Next Steps

You’ve seen how to stand up a local-first Linux agent, give it carefully scoped tools, and run it safely with limits and sandboxes. From here:

  • Extend the allowlist with the commands you actually use daily.

  • Keep a per-task conversation history to add context.

  • Experiment with different models in Ollama for better reasoning or speed.

  • Move repetitive tasks into systemd timers and collect outputs in a daily digest.

If you build something neat—or run into rough edges—share your scripts and lessons learned. The Linux shell has always been about composing small, sharp tools. AI agents are just the newest one in the drawer.

Related posts