Artificial Intelligence Service Failure Analysis: A Bash-First Playbook for Linux

When your AI service goes dark at 2 a.m., logs are thin, dashboards are red, and pressure is high. The fastest path to daylight is a reproducible, bash-driven playbook you can run on any Linux system—bare metal, VM, or container—to turn a mysterious “it’s broken” into a concrete root cause.

This guide shows you how to analyze and fix production AI service failures using core Linux tooling. You’ll get an actionable checklist, real-world examples, and installation commands for apt, dnf, and zypper wherever tools are cited.

Why AI services fail (and why this matters)

AI stacks amplify typical Linux failure modes:

Multiple layers: systemd, containers, CUDA/ROCm, Python envs, kernels, and drivers
Resource volatility: GPU/CPU/memory/file descriptors spike under load
Divergent environments: minor drift breaks native deps, TLS, DNS, or glibc
Non-obvious signals: the “error” might be a restart backoff, OOM killer, or cert chain issue

A small, well-understood toolkit plus a repeatable sequence gives you speed and certainty.

Tools you’ll need (install with your package manager)

strace: inspect syscalls when apps hang/crash
lsof: see open files/sockets/pipes
jq: shape JSON logs/configs
dig: troubleshoot DNS
curl: quick HTTP(S)/health/TLS checks
ca-certificates: fix missing trust stores

Install commands:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y strace lsof jq dnsutils curl ca-certificates

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y strace lsof jq bind-utils curl ca-certificates

openSUSE (zypper):

sudo zypper install -y strace lsof jq bind-utils curl ca-certificates

Optional networking capture:

apt:
```
sudo apt install -y tcpdump
```
dnf:
```
sudo dnf install -y tcpdump
```
zypper:
```
sudo zypper install -y tcpdump
```

Note: systemd, journalctl, ss, ip/ifconfig, and openssl are usually preinstalled.

A 5-step Linux playbook to diagnose AI service failures

1) Fast triage with systemd and logs

Start by answering: Is it running? Has it been restart-throttled? What changed?

# Replace ai.service with your unit name
systemctl status ai.service
systemctl show ai.service -p ActiveState,SubState,ExecMainStatus,MainPID,Restart,RestartSec,StartLimitIntervalSec,StartLimitBurst
journalctl -u ai.service --since -1h -p notice..alert
journalctl -u ai.service -b  # logs since last boot

If it’s in a restart loop:

journalctl -u ai.service --since -30m | tail -n +1

Check kernel events (OOM, segfaults, driver issues):

dmesg --ctime | egrep -i 'oom|killed process|segfault|gpu|nv|amdgpu'

Export the service environment (often overlooked):

systemctl show ai.service -p Environment,EnvironmentFile

2) Resource sanity checks (CPU, RAM, disk, FDs, GPU)

Many mysterious failures are simple exhaustion.

Memory and swap:

free -h
cat /sys/fs/cgroup/memory.max 2>/dev/null || echo "cgroup v1 or unlimited"
dmesg --ctime | grep -i oom | tail

Disk space and inodes:
```
df -h /
df -i /
```

File descriptors:

ulimit -n
cat /proc/sys/fs/file-max
lsof -p $(systemctl show ai.service -p MainPID --value) 2>/dev/null | wc -l

Inotify limits (model watchers, file-based triggers):
```
cat /proc/sys/fs/inotify/max_user_watches
```

GPU visibility (if applicable):

command -v nvidia-smi >/dev/null && nvidia-smi || echo "nvidia-smi not found"
command -v rocm-smi   >/dev/null && rocm-smi   || echo "rocm-smi not found"

Temporarily raise a limit (ephemeral until reboot):

sudo sysctl -w fs.inotify.max_user_watches=524288

3) Network, DNS, and TLS verification

Failures that look like “app bugs” are often name resolution or TLS trust chain issues.

DNS resolution:

getent hosts api.example.com
dig +short A api.example.com
dig +short AAAA api.example.com

Connectivity and TLS:

curl -v https://api.example.com/healthz
openssl s_client -connect api.example.com:443 -servername api.example.com -brief </dev/null

Listening sockets and conflicts:
```
ss -tulpn | grep -E ':(80|443|8080)\b'
```

CA trust (inside minimal containers this is often missing):

python3 -c "import ssl,urllib.request as u;print(u.urlopen('https://example.com').status)" || echo "Likely CA/trust issue"

Install CA bundle if missing:

apt:

sudo apt install -y ca-certificates && sudo update-ca-certificates

dnf:

sudo dnf install -y ca-certificates && sudo update-ca-trust

zypper:

sudo zypper install -y ca-certificates && sudo update-ca-certificates

4) Dependency and environment drift

Minor mismatches in native deps, CUDA/ROCm, glibc, or SSL can break AI runtimes.

Record versions:

python3 --version 2>/dev/null || true
pip3 list --format=freeze 2>/dev/null | sort | tee /tmp/pip-freeze.txt
ldd --version | head -n1
openssl version -a | head -n3
uname -a

Check binary deps:

ldd /usr/local/bin/ai-server 2>&1 | tee /tmp/ldd-ai.txt

CUDA/ROCm presence:

ldconfig -p | egrep -i 'cuda|cudnn|roc|miopen' || true

Container vs host drift (run inside and outside):

printf "HOST glibc: "; ldd --version | head -n1
docker exec ai-container ldd --version | head -n1

If you see “No such file or directory” for an existing binary, suspect a missing ELF interpreter or incompatible glibc. Rebuild in a base image matching your runtime, or install the matching runtime libs.

5) Deep-dive with strace and lsof (only when needed)

Attach to a stuck process to see exactly what it’s waiting on.

Identify PID:

PID=$(systemctl show ai.service -p MainPID --value); echo $PID

Trace syscalls (short window):

sudo strace -f -p "$PID" -tt -o /tmp/ai.strace.log -s 200 -e trace=%network,%file -r -qq -f -T -v -y -Z -w
sleep 5; sudo kill -INT $(pgrep -P $$ strace) 2>/dev/null || true
tail -n 50 /tmp/ai.strace.log

Open files/sockets:

sudo lsof -p "$PID" | head -n 50
sudo lsof -i -P -n | grep $PID

Look for repeating open() failures, permission denies (EACCES), DNS timeouts, or blocked connects.

Real-world mini-cases (and quick fixes)

1) CrashLoop due to file descriptors

Symptom: Works under light load, crashes under heavy concurrency. Too many open files.

Evidence:

journalctl -u ai.service -b | grep -i 'too many open files'
ulimit -n

Fix (systemd override):

sudo systemctl edit ai.service

Add:

[Service]
LimitNOFILE=1048576

Then:

sudo systemctl daemon-reload
sudo systemctl restart ai.service

2) TLS handshake fails only in container

Symptom: curl on host OK; inside container: SSL certificate problem: unable to get local issuer certificate.

Evidence:

docker exec -it ai-container curl -v https://api.example.com/healthz

Fix: install CA bundle in the image/runtime.

apt:

apt-get update && apt-get install -y ca-certificates && update-ca-certificates

dnf:

dnf install -y ca-certificates && update-ca-trust

zypper:

zypper install -y ca-certificates && update-ca-certificates

3) DNS timeouts under load

Symptom: Sporadic timeouts to internal services; logs show Temporary failure in name resolution.

Evidence:

dig +retry=1 +timeout=2 internal.service.local
getent hosts internal.service.local

Fix: ensure resolv.conf points to the correct resolver for the container/namespace, or configure explicit DNS servers. For systemd-resolved setups, use the real file:
```
cat /run/systemd/resolve/resolv.conf
```

A reusable “sos” collector you can paste in

Use this to snapshot the state for incident reports or offline analysis.

#!/usr/bin/env bash
# ai-svc-sos.sh: Collects quick AI service diagnostics
set -euo pipefail
SVC="${1:-ai.service}"
OUT="${2:-/tmp/ai-sos-$(date +%Y%m%d-%H%M%S)}"
mkdir -p "$OUT"

echo "[*] Collecting systemd/journal info..."
systemctl status "$SVC" &> "$OUT/systemctl-status.txt" || true
systemctl show "$SVC" &> "$OUT/systemctl-show.txt" || true
journalctl -u "$SVC" -b &> "$OUT/journal-b.txt" || true
journalctl -u "$SVC" --since -2h &> "$OUT/journal-2h.txt" || true

echo "[*] Kernel and dmesg..."
dmesg --ctime &> "$OUT/dmesg.txt" || true

echo "[*] Resources..."
free -h &> "$OUT/free.txt" || true
df -h &> "$OUT/df-h.txt" || true
df -i &> "$OUT/df-i.txt" || true
ulimit -a &> "$OUT/ulimit.txt" || true
cat /proc/sys/fs/file-max > "$OUT/file-max.txt" || true
cat /proc/sys/fs/inotify/max_user_watches > "$OUT/inotify-max.txt" || true

echo "[*] Network..."
ss -tulpn &> "$OUT/ss.txt" || true
getent hosts localhost &> "$OUT/getent-sanity.txt" || true

echo "[*] Versions..."
{ uname -a; ldd --version | head -n1; openssl version -a | head -n3; } &> "$OUT/versions.txt" || true

PID=$(systemctl show "$SVC" -p MainPID --value 2>/dev/null || echo "")
if [[ -n "${PID}" && "${PID}" != "0" ]]; then
  echo "[*] Process introspection for PID=$PID..."
  ps -p "$PID" -o pid,ppid,cmd &> "$OUT/ps.txt" || true
  lsof -p "$PID" &> "$OUT/lsof.txt" || true
  command -v strace >/dev/null && timeout 5s sudo strace -f -p "$PID" -tt -s 200 -e trace=%network,%file -o "$OUT/strace.txt" || true
fi

tar -czf "$OUT.tar.gz" -C "$(dirname "$OUT")" "$(basename "$OUT")"
echo "Collected: $OUT.tar.gz"

Run:

chmod +x ai-svc-sos.sh
./ai-svc-sos.sh ai.service

Pro tips

Stabilize restarts to see real errors:

sudo systemctl set-property ai.service StartLimitBurst=50 StartLimitIntervalSec=300

Persist sysctl changes by writing to /etc/sysctl.d/*.conf instead of using sysctl -w.
For GPU issues, verify host driver/runtime versions match container/toolkit requirements; small mismatches cause silent failures.

Conclusion and next steps

AI services fail for predictable reasons: limits, dependencies, and environment drift. With this bash-first playbook, you can:

Rapidly triage with systemd and journals
Prove or eliminate resource exhaustion
Validate DNS/TLS in seconds
Catch version and runtime drift
Use strace/lsof only when the basics don’t explain it

Your next step: 1) Install the core tools using your package manager (commands above). 2) Save the ai-svc-sos.sh script and commit it to your ops repo. 3) Create a runbook per AI service with exact unit names, ports, and known-good versions.

When the next outage hits, you’ll be running a checklist—not guessing.

Artificial Intelligence Service Failure Analysis: A Bash-First Playbook for Linux

Why AI services fail (and why this matters)

Tools you’ll need (install with your package manager)

A 5-step Linux playbook to diagnose AI service failures

1) Fast triage with systemd and logs

2) Resource sanity checks (CPU, RAM, disk, FDs, GPU)

3) Network, DNS, and TLS verification

4) Dependency and environment drift

5) Deep-dive with strace and lsof (only when needed)

Real-world mini-cases (and quick fixes)

A reusable “sos” collector you can paste in

Pro tips

Conclusion and next steps

Related posts