Posted on
• Artificial Intelligence

Beginner's Guide to AI Agents on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Beginner’s Guide to AI Agents on Linux

What if your shell could plan a task, pick the right commands, and then explain what just happened? That’s the promise of AI agents: they don’t just answer questions—they can use tools, iterate on results, and help you automate real-world work right from your terminal.

In this guide, you’ll learn what AI agents are, why they’re valuable on Linux, and how to stand up a minimal, safe agent you can extend. We’ll cover both local models (private, offline) and cloud models (fast, powerful), along with actionable steps and real-world examples. All commands are Linux-friendly, with apt, dnf, and zypper installation instructions where relevant.

Why AI Agents on Linux?

  • They plan and act: Unlike simple chatbots, agents can reason about goals, call tools (like shell commands), inspect outputs, and iterate.

  • They speak Linux: The shell is a toolbox of composable programs. Agents can orchestrate grep, awk, find, journalctl, and more—then summarize results for humans.

  • They save time: Log triage, codebase spelunking, and boilerplate scripting become faster and less error-prone.

  • They’re flexible: Run agents entirely locally for privacy, or connect to cloud models when you need extra capability.

What You’ll Build

A minimal “safe” terminal agent that:

  • Plans steps for a request

  • Proposes bash commands

  • Asks for your confirmation before executing

  • Runs allowed commands, captures outputs

  • Explains what it did

You’ll be able to plug in a local model via Ollama or a cloud model via an OpenAI-compatible API.

1) Prep your Linux box

We’ll install common CLI utilities and Python tooling.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl git python3 python3-venv pipx jq ripgrep
pipx ensurepath

Fedora/RHEL (dnf):

sudo dnf install -y curl git python3 python3-venv pipx jq ripgrep
pipx ensurepath

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl git python3 python3-venv pipx jq ripgrep
pipx ensurepath

If pipx isn’t available in your repo:

python3 -m pip install --user pipx
python3 -m pipx ensurepath

2) Choose your model runtime

Option A — Local (privacy-friendly, offline) with Ollama:

curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
# Pull a compact general-purpose model (CPU-friendly)
ollama pull llama3.2:3b
# You can also try a larger model if you have RAM/GPU:
# ollama pull llama3.1:8b

Option B — Cloud (fast, powerful) with OpenAI-compatible API:

  • Create an API key with your provider and export it:
export OPENAI_API_KEY="sk-yourkey"

Notes:

  • Local is great for privacy and air-gapped systems. Smaller models can run on CPU but will be slower.

  • Cloud models are faster/stronger but require an internet connection and a paid key.

3) Set up a Python environment

Create a dedicated virtual environment and install client libraries for both backends (we’ll auto-detect which one to use):

python3 -m venv ~/agents-venv
source ~/agents-venv/bin/activate
pip install --upgrade pip
pip install ollama openai

4) Build a minimal safe agent

Save this as agent.py:

#!/usr/bin/env python3
import os, re, shlex, subprocess, sys

# Configuration
DEFAULT_LOCAL_MODEL = os.environ.get("AGENT_LOCAL_MODEL", "llama3.2:3b")
DEFAULT_CLOUD_MODEL = os.environ.get("AGENT_CLOUD_MODEL", "gpt-4o-mini")
ALLOWED_CMDS = {
    "ls","cat","head","tail","grep","rg","awk","sed","cut","sort","uniq","wc",
    "find","du","df","journalctl","dmesg","printf","echo","date"
}
DENY_SUBSTRINGS = ["sudo", " rm ", " mv ", " chmod ", " chown ", "dd ", " :>", " > ", " >> ", "| sh", "mkfs", "mount", "umount"]

def backend():
    if os.environ.get("AGENT_BACKEND") in ("ollama","openai"):
        return os.environ["AGENT_BACKEND"]
    return "openai" if os.environ.get("OPENAI_API_KEY") else "ollama"

def model_name():
    return DEFAULT_CLOUD_MODEL if backend() == "openai" else DEFAULT_LOCAL_MODEL

def ask_model(prompt):
    sys_msg = (
        "You are a Linux command-line assistant. "
        "Plan briefly, then propose safe, read-only commands to achieve the goal. "
        "Return commands in a single fenced code block like:\n```bash\n...\n```"
    )
    if backend() == "ollama":
        import ollama
        resp = ollama.chat(
            model=model_name(),
            messages=[
                {"role":"system","content":sys_msg},
                {"role":"user","content":prompt}
            ],
        )
        return resp["message"]["content"]
    else:
        from openai import OpenAI
        client = OpenAI()
        resp = client.chat.completions.create(
            model=model_name(),
            messages=[
                {"role":"system","content":sys_msg},
                {"role":"user","content":prompt}
            ],
            temperature=0.2,
        )
        return resp.choices[0].message.content

def extract_commands(text):
    blocks = re.findall(r"```(?:bash|sh)?\n(.*?)```", text, re.S)
    if not blocks:
        return []
    lines = []
    for block in blocks:
        for raw in block.strip().splitlines():
            line = raw.strip()
            if not line or line.startswith("#"):
                continue
            if line.startswith("$ "):
                line = line[2:].strip()
            lines.append(line)
    # Filter with allowlist/denylist
    safe = []
    for line in lines:
        if any(bad in f" {line} " for bad in DENY_SUBSTRINGS):
            continue
        try:
            cmd = shlex.split(line)
        except Exception:
            continue
        if not cmd:
            continue
        if cmd[0] in ALLOWED_CMDS:
            safe.append(line)
    return safe

def run_commands(cmds):
    transcript = []
    for c in cmds:
        proc = subprocess.run(c, shell=True, capture_output=True, text=True)
        out = (proc.stdout or "") + (("\nSTDERR:\n" + proc.stderr) if proc.stderr else "")
        transcript.append(f"$ {c}\n{out}".rstrip())
    return "\n\n".join(transcript)

def summarize(observation, original_goal):
    followup = (
        "You executed the above commands. Summarize what happened, "
        "call out any anomalies, and answer the user's goal:\n" + original_goal
    )
    return ask_model(followup + "\n\nOBSERVATION:\n" + observation[:6000])

def main():
    if len(sys.argv) < 2:
        print("Usage: python agent.py \"your goal here\"")
        sys.exit(1)
    goal = sys.argv[1]

    print(f"[backend] {backend()}  [model] {model_name()}")
    print("[plan] Asking the model for a plan + commands...")
    plan = ask_model(goal)
    print("\n--- Model Plan/Proposal ---")
    print(plan)
    cmds = extract_commands(plan)
    if not cmds:
        print("\n[warn] No safe commands detected. Exiting.")
        sys.exit(0)
    print("\n--- Candidate Commands (filtered) ---")
    for c in cmds:
        print(c)
    ans = input("\nExecute these commands? [y/N]: ").strip().lower()
    if ans != "y":
        print("Aborted by user.")
        sys.exit(0)

    print("\n[run] Executing...")
    observation = run_commands(cmds)
    print("\n--- Observation ---")
    print(observation[:4000] + ("\n... (truncated)" if len(observation) > 4000 else ""))

    print("\n[summary] Asking the model to summarize results...")
    final = summarize(observation, goal)
    print("\n=== Final Summary ===")
    print(final)

if __name__ == "__main__":
    main()

Make it executable and run:

chmod +x agent.py

Backend selection:

  • Default: Uses OpenAI if OPENAI_API_KEY is set; otherwise uses Ollama.

  • Override explicitly:

export AGENT_BACKEND=ollama   # or: openai
export AGENT_LOCAL_MODEL="llama3.1:8b"  # optional
export AGENT_CLOUD_MODEL="gpt-4o-mini"  # optional

5) Try real-world examples

Before running, ensure your chosen backend is ready:

  • Local: ollama serve & and ollama pull llama3.2:3b

  • Cloud: export OPENAI_API_KEY=...

A) Log triage (largest logs + quick summary)

./agent.py "Find the 5 largest files under /var/log and summarize common issues mentioned inside."

B) Dev spelunking (explain an error seen in your codebase)

./agent.py "Search this repo for EADDRINUSE occurrences and explain typical causes and fixes."

C) Bash helper (generate a safe snippet, then run it)

./agent.py "Show commands to list all systemd services that failed in the last boot and summarize likely root causes."

Tip: The allowlist restricts to read-only commands by default. You can add more tools as you gain confidence (e.g., curl, jq)—but be mindful of security.

Tips, performance, and safety

  • Safety first: This example requires confirmation and filters dangerous commands. Keep destructive operations off the allowlist unless you fully trust the agent and add extra guardrails (dry-runs, sandboxes, or containers).

  • Models:

    • Local: Smaller models like llama3.2:3b run on CPU; bigger models need more RAM and benefit from a GPU.
    • Cloud: Use a cost-effective model for command planning; step up to larger models for complex reasoning.

  • Useful tools to allow later: curl, jq, tar, unzip—but start read-only and iterate cautiously.

  • Troubleshooting:

    • If pipx isn’t found, re-run pipx ensurepath or re-open your shell.
    • If Ollama fails to start, ensure port 11434 is free and your GPU drivers (if any) are set up.
    • To keep the agent’s venv handy, add to your shell profile: alias agent='source ~/agents-venv/bin/activate && ./agent.py'.

Why this approach works

  • Composability: Linux tools excel at one thing each. Agents can chain them to achieve complex tasks while keeping you in control.

  • Feedback loop: The agent proposes, you approve, it executes, then explains—closing the loop in a single workflow.

  • Extensibility: You can add capabilities (e.g., web search via curl+jq, file parsing, local knowledge bases) without rewriting from scratch.

Conclusion and Next Steps

You’ve stood up a minimal, safe AI agent on Linux that plans, runs allowed commands, and explains results—using either a local or cloud model.

Next steps:

  • Extend the allowlist with tools you rely on, and add structured “tools” (functions) the agent can call.

  • Integrate with tmux scripts, wrapper functions, or a systemd user service for quick access.

  • Explore frameworks like LangChain or AutoGen once you want more advanced tool routing and multi-agent patterns.

If this was useful, try wiring the agent into one of your daily workflows and see how much time you save this week. Happy hacking!

Related posts