Posted on
• Artificial Intelligence

End-to-End Artificial Intelligence Automation for Linux Infrastructure

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

End-to-End Artificial Intelligence Automation for Linux Infrastructure

If you manage Linux fleets, you’ve felt the squeeze: more hosts, more logs, more patches, more alerts, less time. What if your estate could collect its own signals, ask an AI to explain what’s wrong, draft a safe remediation, and then hand you a clear, check-mode preview to approve? That’s end-to-end AI automation for Linux—turning raw telemetry into guided, idempotent actions with guardrails.

This post shows you how to stand up a small, auditable pipeline on any modern Linux box using Bash, systemd timers, Python, and Ansible. You’ll go from “signal” to “suggested fix” in minutes, keeping a human firmly in control.

  • You’ll install lightweight, standard tools (apt/dnf/zypper instructions included).

  • You’ll collect OS signals on a timer.

  • You’ll run an AI analysis step that summarizes issues and proposes an Ansible playbook.

  • You’ll preview the remediation in Ansible check mode and decide when to run it.

Why this works (and why now)

  • Linux is scriptable end-to-end: systemd timers, journald, and CLI-first tooling make event-driven ops simple to wire together.

  • Idempotent automation is mature: Ansible and friends let you encode safe, repeatable state changes with diff/preview gates.

  • LLMs are great at text-to-ops: They digest logs and config, propose steps, and explain trade-offs—if you constrain them and add guardrails.

  • Human-in-the-loop keeps it safe: You review, modify, and approve. The AI drafts; you decide.

What you’ll build

A minimal, production-friendly loop:

1) Collect: A systemd timer runs a Bash probe that captures recent errors and system health. 2) Analyze: A Python script sends that bundle to an LLM and gets back: - a short summary, - a risk rating, - a proposed Ansible playbook (YAML) that is safe and idempotent. 3) Preview: Ansible runs in check mode with diff to show what it would change. 4) Approve and apply: You promote the playbook and run it for real.

You can expand this pattern to include CI, GitOps, cloud APIs, and broader telemetry later.

Prerequisites and installation

The following commands install everything you need. Pick the block for your distro.

Required tools:

  • git, curl, jq, make

  • Python 3 + pip

  • Ansible

  • sysstat (for sar)

  • Optional: python packages via pip

Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y git curl jq make python3 python3-venv python3-pip ansible sysstat

Fedora/Rocky/Alma/RHEL (dnf):

sudo dnf install -y git curl jq make python3 python3-virtualenv python3-pip ansible sysstat
# If "ansible" is unavailable, try:
# sudo dnf install -y ansible-core

openSUSE Leap/Tumbleweed (zypper):

sudo zypper refresh
sudo zypper install -y git curl jq make python3 python3-pip ansible sysstat

Python packages (user install; works everywhere):

python3 -m pip install --user openai pyyaml

Environment for AI access (choose an OpenAI-compatible endpoint or vendor):

export OPENAI_API_KEY="your_api_key_here"
# Optional: self-hosted or alternate provider base URL
# export OPENAI_BASE_URL="https://your.endpoint.example/v1"
# Model hint (adjust to what your provider offers)
export OPENAI_MODEL="gpt-4o-mini"

Note: You can use any OpenAI-compatible API. If you run a local model behind an OpenAI-compatible server (e.g., vLLM, text-generation-inference, etc.), just set OPENAI_BASE_URL and a matching model name.

Step 1 — Collect OS signals with a systemd timer

Create a collection script that gathers a concise “signal bundle.”

collect-signals.sh:

#!/usr/bin/env bash
set -euo pipefail

OUT="${1:-/var/tmp/ai-signals.log}"
exec >"$OUT" 2>&1

echo "=== AI Signal Bundle | Host: $(hostname -f) | $(date -Is) ==="
echo
echo "[uptime]"
uptime
echo
echo "[load/memory]"
free -m
echo
echo "[disk]"
df -hT --exclude-type=tmpfs --exclude-type=devtmpfs
echo
echo "[network summary]"
ss -s || true
echo
echo "[systemd failed units]"
systemctl --failed || true
echo
echo "[recent critical journal]"
journalctl -p 0..3 -S -5m -o short-iso --no-pager || true
echo
echo "[top CPU offenders]"
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head -n 15
echo
echo "[run queue (sar)]"
sar -q 1 3 || true

Install and schedule it:

sudo install -m 0755 collect-signals.sh /usr/local/sbin/collect-signals.sh

/etc/systemd/system/ai-signals.service:

[Unit]
Description=Collect OS signals for AI analysis

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/collect-signals.sh /var/tmp/ai-signals.log
User=root

/etc/systemd/system/ai-signals.timer:

[Unit]
Description=Every 5 minutes collect OS signals for AI

[Timer]
OnBootSec=3min
OnUnitActiveSec=5min
Unit=ai-signals.service

[Install]
WantedBy=timers.target

Enable the timer:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-signals.timer
systemctl list-timers ai-signals.timer

You now have a rolling, timestamped snapshot of system health at /var/tmp/ai-signals.log.

Step 2 — AI analysis: turn logs into a plan

The following Python script reads the signal bundle, calls an OpenAI-compatible API, and returns a JSON payload that includes a concise summary, a risk level, and a proposed Ansible playbook (as YAML text).

ai_analyze.py:

#!/usr/bin/env python3
import os, sys, json, textwrap
from openai import OpenAI

def read_input(path=None):
    if path and path != "-":
        with open(path, "r", encoding="utf-8") as f:
            return f.read()
    return sys.stdin.read()

def main():
    data = read_input(sys.argv[1] if len(sys.argv) > 1 else "/var/tmp/ai-signals.log")

    api_key = os.environ.get("OPENAI_API_KEY")
    if not api_key:
        print("ERROR: OPENAI_API_KEY not set", file=sys.stderr)
        sys.exit(2)

    base_url = os.environ.get("OPENAI_BASE_URL")  # optional
    model = os.environ.get("OPENAI_MODEL", "gpt-4o-mini")

    client = OpenAI(api_key=api_key, base_url=base_url) if base_url else OpenAI(api_key=api_key)

    system_msg = textwrap.dedent("""
      You are a Linux SRE assistant. Read Linux telemetry and produce:
      - a short, plain-English summary,
      - a risk rating: low | medium | high,
      - a proposed Ansible playbook in YAML that is safe, idempotent, and uses check-mode-friendly modules.
      Constraints:
      - Prefer Ansible modules (lineinfile, ini_file, service, package, sysctl) over raw commands.
      - If running a command is unavoidable, set changed_when appropriately and keep it safe.
      - Target localhost inventory group by default; do not assume SSH.
      - Keep the playbook minimal and self-contained (handlers allowed).
      - Explain nothing in the YAML; comments are okay but no extra prose outside the YAML.
    """)

    user_msg = f"""
    Linux signal bundle follows. Identify top 1-2 actionable issues and propose the smallest safe fix.
    If no action is warranted, return an empty playbook.
    ---BEGIN---
    {data}
    ---END---
    """

    resp = client.chat.completions.create(
        model=model,
        temperature=0.2,
        response_format={"type":"json_object"},
        messages=[
            {"role":"system","content":system_msg},
            {"role":"user","content":user_msg},
        ],
    )

    content = resp.choices[0].message.content
    try:
        j = json.loads(content)
    except Exception:
        j = {"summary": content.strip(), "risk": "unknown", "proposed_playbook_yml": ""}

    print(json.dumps(j, indent=2))

if __name__ == "__main__":
    main()

Quick test:

python3 ai_analyze.py /var/tmp/ai-signals.log | jq .

Step 3 — Preview remediation with Ansible (check mode)

Create a tiny inventory for local runs:

inventory.ini:

[local]
localhost ansible_connection=local

Now wire the pipeline together. This Bash wrapper collects signals (on-demand), calls the AI, extracts the proposed playbook, and runs Ansible in check mode with diff. Nothing changes on the host unless you later approve and re-run without --check.

pipeline.sh:

#!/usr/bin/env bash
set -euo pipefail

RUN_DIR="${RUN_DIR:-$PWD/runs}"
SIG_FILE="/var/tmp/ai-signals.log"
TS="$(date +%Y%m%d-%H%M%S)"
OUT_JSON="$RUN_DIR/analysis-$TS.json"
OUT_PLAY="$RUN_DIR/proposed-$TS.yml"

mkdir -p "$RUN_DIR"

# 1) Ensure we have fresh signals
sudo /usr/local/sbin/collect-signals.sh "$SIG_FILE"

# 2) Analyze with AI
python3 ai_analyze.py "$SIG_FILE" | tee "$OUT_JSON" >/dev/null

# 3) Extract proposed playbook (if any)
PLAYBOOK_YML=$(jq -r '.proposed_playbook_yml // ""' "$OUT_JSON")
if [[ -z "$PLAYBOOK_YML" || "$PLAYBOOK_YML" == "null" ]]; then
  echo "No remediation proposed. Summary:"
  jq -r '.summary' "$OUT_JSON"
  exit 0
fi

printf "%s\n" "$PLAYBOOK_YML" > "$OUT_PLAY"
echo "Proposed playbook written to: $OUT_PLAY"
echo

# 4) Dry-run with Ansible (check mode + diff)
ANSIBLE_STDOUT_CALLBACK=unixy \
ansible-playbook -i inventory.ini "$OUT_PLAY" --check --diff -v || true

echo
echo "Review the diff above. If acceptable, apply for real:"
echo "ansible-playbook -i inventory.ini $OUT_PLAY -v"

Run the pipeline:

chmod +x pipeline.sh
./pipeline.sh

You’ll see:

  • A human-readable summary of what’s wrong,

  • A proposed playbook file under runs/,

  • Ansible’s check-mode output and unified diffs of what would change.

Approve and apply when ready:

ansible-playbook -i inventory.ini runs/proposed-<timestamp>.yml -v

Tip: Commit runs/ outputs into a Git repo for auditability:

git init
echo -e "*__pycache__*\n*.retry\n" >> .gitignore
git add inventory.ini ai_analyze.py collect-signals.sh pipeline.sh runs/
git commit -m "AI-ops first run: analysis and proposed remediation"

Step 4 — Guardrails you should keep

  • Always start in check mode. Only promote after review and, ideally, in a maintenance window.

  • Keep the blast radius small:

    • Use an inventory group like [canary] and run --limit canary first.
    • Gradually expand to [prod] once you trust the change.

  • Require code review: store proposed playbooks in Git and use a PR to merge into an “approved” directory.

  • Log everything: keep the signal bundle, the AI JSON, and the final Ansible output in your runs/ folder.

Real-world example: noisy logs and runaway journals

Symptom:

  • journalctl shows rapid log growth and disk usage approaching 90%.

  • Critical units are flapping.

Likely AI output:

  • Summary flags log volume and risk: medium.

  • Proposed Ansible playbook to cap journald size and vacuum old logs:

Example playbook (what you might see in runs/proposed-*.yml):


- hosts: local
  gather_facts: false
  tasks:
    - name: Ensure journald size is capped
      ini_file:
        path: /etc/systemd/journald.conf
        section: Journal
        option: SystemMaxUse
        value: 1G
        no_extra_spaces: true
        backup: true
      notify: Restart journald

    - name: Vacuum old journal logs (7 days)
      command: journalctl --vacuum-time=7d
      changed_when: false  # act only if vacuum actually deletes logs; safe in check mode

  handlers:
    - name: Restart journald
      service:
        name: systemd-journald
        state: restarted

Dry-run first:

ansible-playbook -i inventory.ini runs/proposed-<timestamp>.yml --check --diff -v

Then apply:

ansible-playbook -i inventory.ini runs/proposed-<timestamp>.yml -v

This keeps your logs within a predictable footprint and quiets cascading disk alerts.

Where to go next

  • Add more signals: dmesg excerpts, smartctl for disks, package drift, kubelet status.

  • Wire into CI: push runs/ artifacts to a repo; gate “apply” on an approval workflow.

  • Expand the scope: use group_vars and tags so the AI can propose targeted changes per role or environment.

  • Bring metrics: feed Prometheus alerts or sysstat history into the bundle for trend-aware suggestions.

  • Tiered models: use a small, cheap model for triage; escalate tricky cases to a bigger one.

Conclusion and Call to Action

You don’t need a massive platform to get value from AI in Linux ops. With a timer, a few scripts, and Ansible, you can turn endless logs into concise explanations and safe, idempotent fixes—still reviewed and approved by you.

Your next step: 1) Install the prerequisites for your distro (apt/dnf/zypper blocks above). 2) Drop in collect-signals.sh, ai_analyze.py, inventory.ini, and pipeline.sh. 3) Export your OPENAI_API_KEY and run ./pipeline.sh. 4) Review the proposed fix in check mode, then approve and apply.

Ship it—safely, repeatably, and with your hands on the wheel.

Related posts