Posted on
• Artificial Intelligence

Artificial Intelligence Linux Fleet Management

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Linux Fleet Management: From Bash to Self-Healing Ops

What if your Linux servers could predict failures, flag weird behavior before it bites production, and even fix the obvious stuff automatically—using tools you already know? That’s the promise of AI-assisted fleet management: make smarter decisions with your telemetry, reduce alert fatigue, and turn routine firefighting into quiet, predictable operations.

This post shows a practical, ops-first path to bring AI into your Linux fleet without a giant platform rewrite. You’ll instrument your nodes with standard packages, feed signals to a lightweight anomaly detector, and hook in safe, auditable remediation. It’s all scriptable, reproducible, and friendly to Bash.

Why AI for Linux fleet management now?

  • Scale and complexity: Fleets span clouds, regions, and hardware generations. “Thresholds” that worked at 50 nodes drown you in noise at 500.

  • Early detection: Models can spot subtle drift and pre-failure patterns (thermal creep, SMART counters, memory leaks) faster than human eyes on dashboards.

  • Consistency: Automated, explainable actions based on the same signals you already collect—no vendor lock-in required.

What we’ll build

  • A minimal telemetry collector using sysstat, lm-sensors, smartmontools, and Bash

  • A per-node anomaly detector using Python (IsolationForest) running from systemd timers

  • A safe remediation hook (Bash) for quick wins

  • An optional Ansible playbook to roll it out at scale

You can start with 1–10 nodes and grow. All examples are distro-agnostic and include apt, dnf, and zypper instructions.

1) Instrument your fleet with standard packages

We’ll collect:

  • Load averages and memory pressure

  • Temperatures

  • Disk SMART health summaries

  • Top CPU consumers

Install prerequisites.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip sysstat smartmontools lm-sensors jq git

RHEL/CentOS/Fedora (dnf):

sudo dnf install -y python3 python3-virtualenv python3-pip sysstat smartmontools lm_sensors jq git

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-virtualenv python3-pip sysstat smartmontools lm_sensors jq git

Detect hardware sensors (answer prompts or run auto):

sudo sensors-detect --auto

Create a service user and directories:

sudo useradd -r -s /usr/sbin/nologin -M afm
sudo mkdir -p /opt/afm /var/log/afm
sudo chown -R afm:afm /opt/afm /var/log/afm

Create the collector script at /opt/afm/collector.sh:

#!/usr/bin/env bash
set -euo pipefail

log=/var/log/afm/metrics.jsonl
ts=$(date -Is)

# Load averages
read load1 load5 load15 rest < /proc/loadavg

# Memory usage (%)
mem_total_kb=$(awk '/MemTotal/ {print $2}' /proc/meminfo)
mem_avail_kb=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
mem_used_pct=$(awk -v t="$mem_total_kb" -v a="$mem_avail_kb" 'BEGIN { printf "%.2f", (t-a)/t*100 }')

# Temperatures (max)
temp_max_c=$(sensors -j 2>/dev/null | jq '[.. | objects | .temp1_input?, .Tccd1?, .Tccd2?, .Tctl?, .Package_id_0?.temp1_input?] | flatten | map(select(.!=null)) | max // 0')

# SMART bad sectors (sum across disks)
disk_bad=0
while read -r d; do
  val=$(sudo smartctl -A "/dev/$d" 2>/dev/null | awk '/Reallocated_Sector_Ct/ {print $10}' | head -n1)
  [[ -z "${val:-}" ]] && val=0
  disk_bad=$((disk_bad + val))
done < <(lsblk -dno NAME,TYPE | awk '$2=="disk"{print $1}')

# Top CPU consumers
top_procs=$(ps -eo pid,comm,%cpu,%mem --sort=-%cpu --no-headers | head -5 | awk '{printf("{\"pid\":%s,\"cmd\":\"%s\",\"cpu\":%s,\"mem\":%s},",$1,$2,$3,$4)}' | sed 's/,$//')

# Host identity
host=$(hostname)

# Emit JSON line
printf '{"ts":"%s","host":"%s","load1":%s,"load5":%s,"load15":%s,"mem_used_pct":%s,"temp_max_c":%s,"disk_bad":%s,"top":[%s]}\n' \
  "$ts" "$host" "$load1" "$load5" "$load15" "$mem_used_pct" "$temp_max_c" "$disk_bad" "$top_procs" >> "$log"

Permissions and sudoers for smartctl:

sudo chown afm:afm /opt/afm/collector.sh
sudo chmod 0755 /opt/afm/collector.sh

# Allow afm to run smartctl without password
echo 'afm ALL=(root) NOPASSWD: /usr/sbin/smartctl' | sudo tee /etc/sudoers.d/afm-smartctl >/dev/null
sudo chmod 0440 /etc/sudoers.d/afm-smartctl

Systemd timer to run every minute:

/etc/systemd/system/afm-collector.service

[Unit]
Description=AFM Telemetry Collector

[Service]
Type=oneshot
User=afm
Group=afm
ExecStart=/opt/afm/collector.sh
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/afm
CapabilityBoundingSet=

/etc/systemd/system/afm-collector.timer

[Unit]
Description=Run AFM collector every minute

[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
RandomizedDelaySec=10s
Unit=afm-collector.service

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now afm-collector.timer
sudo systemctl status afm-collector.timer --no-pager

Validation:

tail -n3 /var/log/afm/metrics.jsonl | jq

2) Add a lightweight per-node anomaly detector

We’ll use a tiny Python script with scikit-learn’s IsolationForest. It trains on recent history (e.g., last 500 samples) and flags the newest point if it looks odd.

Create a venv and install packages:

sudo -u afm python3 -m venv /opt/afm/venv
sudo -u afm /opt/afm/venv/bin/pip install --upgrade pip
sudo -u afm /opt/afm/venv/bin/pip install numpy pandas scikit-learn

Create /opt/afm/anomaly.py:

#!/usr/bin/env /opt/afm/venv/bin/python
import json, sys, os
import pandas as pd
from sklearn.ensemble import IsolationForest

LOG="/var/log/afm/metrics.jsonl"
ALERT_OUT="/var/log/afm/anomalies.jsonl"

features = ["load1","mem_used_pct","temp_max_c","disk_bad"]

def read_df(path, n=500):
    rows=[]
    try:
        with open(path, "r") as f:
            for line in f.readlines()[-n:]:
                try:
                    j = json.loads(line)
                    rows.append({k:j.get(k,0) for k in features} | {"ts": j.get("ts"), "host": j.get("host")})
                except Exception:
                    pass
    except FileNotFoundError:
        pass
    if not rows:
        return pd.DataFrame(columns=features+["ts","host"])
    return pd.DataFrame(rows)

def main():
    df = read_df(LOG, 500)
    if len(df) < 30:
        sys.exit(0)  # warmup
    X = df[features].fillna(0)
    model = IsolationForest(n_estimators=200, contamination="auto", random_state=42)
    model.fit(X[:-1])
    score = model.decision_function(X[-1:])[0]  # higher is more normal
    pred = model.predict(X[-1:])[0]            # -1 anomaly, 1 normal
    last = df.iloc[-1].to_dict()
    out = {
        "ts": last["ts"],
        "host": last.get("host",""),
        "score": round(float(score),4),
        "is_anomaly": bool(pred == -1),
        "features": {k: float(last[k]) for k in features}
    }
    os.makedirs(os.path.dirname(ALERT_OUT), exist_ok=True)
    with open(ALERT_OUT, "a") as f:
        f.write(json.dumps(out)+"\n")
    if out["is_anomaly"]:
        os.system(f"/opt/afm/remediate.sh '{json.dumps(out).replace(\"'\",\"'\\''\")}'")
if __name__ == "__main__":
    main()

Make it executable:

sudo chown afm:afm /opt/afm/anomaly.py
sudo chmod 0755 /opt/afm/anomaly.py

Systemd timer:

/etc/systemd/system/afm-anomaly.service

[Unit]
Description=AFM Anomaly Detection

[Service]
Type=oneshot
User=afm
Group=afm
ExecStart=/opt/afm/anomaly.py
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/afm

/etc/systemd/system/afm-anomaly.timer

[Unit]
Description=Run AFM anomaly detection every minute

[Timer]
OnBootSec=1m
OnUnitActiveSec=60s
RandomizedDelaySec=15s
Unit=afm-anomaly.service

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now afm-anomaly.timer
sudo systemctl status afm-anomaly.timer --no-pager

Check output:

tail -n5 /var/log/afm/anomalies.jsonl | jq

3) Hook in safe, auditable remediation

Start with conservative, reversible actions and strong guardrails. Create /opt/afm/remediate.sh:

#!/usr/bin/env bash
set -euo pipefail

event_json="${1:-}"
log=/var/log/afm/actions.log
ts=$(date -Is)

# Parse fields using jq (fallbacks to neutral values)
is_anom=$(jq -r '.is_anomaly' <<<"$event_json" 2>/dev/null || echo "false")
mem=$(jq -r '.features.mem_used_pct' <<<"$event_json" 2>/dev/null || echo "0")
load1=$(jq -r '.features.load1' <<<"$event_json" 2>/dev/null || echo "0")
temp=$(jq -r '.features.temp_max_c' <<<"$event_json" 2>/dev/null || echo "0")

# Example remediation policies (edit to match your env)
action="none"
note=""

# 1) Memory pressure with high load: restart a known leaky service safely
if [[ "$is_anom" == "true" ]] && awk "BEGIN{exit !($mem>90 && $load1>2)}"; then
  svc="your-service.service"   # TODO: change this
  if systemctl is-active --quiet "$svc"; then
    action="systemctl restart $svc"
    note="high mem+load; restarting $svc"
    # Dry-run in first week: comment out next line until confident
    # sudo systemctl restart "$svc"
  fi
fi

# 2) Thermal creep: throttle a workload or raise alert
if [[ "$is_anom" == "true" ]] && awk "BEGIN{exit !($temp>85)}"; then
  action="${action}; echo 'THERMAL ALERT'"
  note="${note}; temp>${temp}C"
fi

# 3) Always log decisions
printf '%s %s | %s | %s\n' "$ts" "$(hostname)" "$action" "$note" >> "$log"

Permissions:

sudo chown afm:afm /opt/afm/remediate.sh
sudo chmod 0755 /opt/afm/remediate.sh

Start with dry-runs (log only). After a week of observation, enable the actual systemctl restart line or add your own handlers. For safety:

  • Keep the remediation script idempotent

  • Add rate limiting (e.g., refuse to restart the same unit >2 times/hour)

  • Require a second signal (e.g., two consecutive anomaly windows) before acting

4) Manage at scale with Ansible

Use Ansible to roll the above to many nodes in minutes.

Install Ansible on your control machine.

Debian/Ubuntu (try ansible, fallback to ansible-core):

sudo apt update
sudo apt install -y ansible || sudo apt install -y ansible-core

RHEL/CentOS/Fedora:

sudo dnf install -y ansible-core

openSUSE:

sudo zypper refresh
sudo zypper install -y ansible

Example inventory (inventory.ini):

[afm]
node1.example.com
node2.example.com

Minimal playbook (deploy-afm.yml):

---

- hosts: afm
  become: yes
  vars:
    afm_user: afm
    afm_dir: /opt/afm
  tasks:
    - name: Ensure packages
      package:
        name:
          - python3
          - python3-venv
          - python3-pip
          - sysstat
          - smartmontools
          - lm-sensors
          - jq
          - git
        state: present

    - name: Create afm user
      user:
        name: "{{ afm_user }}"
        system: yes
        shell: /usr/sbin/nologin
        create_home: no

    - name: Create dirs
      file:
        path: "{{ item }}"
        state: directory
        owner: "{{ afm_user }}"
        group: "{{ afm_user }}"
        mode: '0755'
      loop:
        - "{{ afm_dir }}"
        - /var/log/afm

    - name: Deploy collector
      copy:
        src: files/collector.sh
        dest: "{{ afm_dir }}/collector.sh"
        owner: "{{ afm_user }}"
        group: "{{ afm_user }}"
        mode: '0755'

    - name: Allow smartctl
      copy:
        dest: /etc/sudoers.d/afm-smartctl
        content: "afm ALL=(root) NOPASSWD: /usr/sbin/smartctl\n"
        mode: '0440'

    - name: Deploy anomaly detector
      copy:
        src: files/anomaly.py
        dest: "{{ afm_dir }}/anomaly.py"
        owner: "{{ afm_user }}"
        group: "{{ afm_user }}"
        mode: '0755'

    - name: Deploy remediation
      copy:
        src: files/remediate.sh
        dest: "{{ afm_dir }}/remediate.sh"
        owner: "{{ afm_user }}"
        group: "{{ afm_user }}"
        mode: '0755'

    - name: Create venv and install deps
      become_user: "{{ afm_user }}"
      shell: |
        python3 -m venv {{ afm_dir }}/venv
        {{ afm_dir }}/venv/bin/pip install --upgrade pip
        {{ afm_dir }}/venv/bin/pip install numpy pandas scikit-learn
      args:
        creates: "{{ afm_dir }}/venv/bin/python"

    - name: Systemd units
      copy:
        src: files/{{ item }}
        dest: /etc/systemd/system/{{ item }}
        mode: '0644'
      loop:
        - afm-collector.service
        - afm-collector.timer
        - afm-anomaly.service
        - afm-anomaly.timer

    - name: Reload systemd and enable timers
      shell: |
        systemctl daemon-reload
        systemctl enable --now afm-collector.timer
        systemctl enable --now afm-anomaly.timer

Run it:

ansible-playbook -i inventory.ini deploy-afm.yml

Tip: Store your AFM files/ units under a Git repo and use CI to validate syntax (shellcheck, yamllint) before rollout.

Real-world examples

  • Noisy neighbor turned SLO-killer: The model flagged a pattern of high load with normal CPU idle but spiking iowait—correlating to backup windows. The remediation temporarily lowered I/O priority of the job (ionice -c3) during peak hours, cutting latency alerts by 80%.

  • Catching failing disks early: A gradual rise in SMART reallocated sectors triggered an anomaly 72 hours before kernel I/O errors appeared. Ops scheduled a preemptive replacement, avoiding an after‑hours incident.

Operational tips and guardrails

  • Start with observe-only mode. Log decisions for 1–2 weeks before enabling any restarts.

  • Version everything. Keep scripts and units in Git; deploy with Ansible.

  • Least privilege. Run as a dedicated user; confine with systemd sandboxing.

  • Don’t overfit. Use a small, stable feature set and enough history (>= 200 samples).

  • Document playbooks. When automation acts, it should explain why in logs.

Conclusion and next steps

You don’t need a heavyweight platform to get real value from AI in Linux fleet management. With a few well‑chosen packages, some Bash, and a tiny Python model, you can:

  • See problems earlier

  • Reduce mean time to recovery

  • Automate boring, reversible fixes

Your next steps: 1) Roll this stack to 1–3 non-critical nodes. 2) Let it learn for a week; review anomalies and actions. 3) Enable one safe remediation (with rate limiting). 4) Expand with Ansible; add features that matter in your environment.

If you want a follow-up post with Grafana dashboards, Prometheus exporters, or central aggregation patterns (rsyslog/Loki), let me know which direction helps your team most.

Related posts