Artificial Intelligence Troubleshooting Checklists for Bash-First Linux Users

When an AI job fails at 2 a.m., it’s rarely one thing—it’s everything, everywhere, all at once: environment quirks, brittle dependencies, sketchy data, tired GPUs, flakey networks. The fix? A Bash-first, repeatable set of checklists you can run with your eyes half-closed and your logs fully open.

This post gives you practical, copy-pasteable diagnostics and remediation steps for the most common failure modes in AI work on Linux. You’ll get a lightweight toolbox, five focused checklists, and small scripts you can drop into any repo.

Why this matters

• AI stacks are multi-layered: kernel, drivers, Python, native libs, CUDA/ROCm, data formats, APIs, and orchestration. A disciplined checklist finds the right layer fast.

• Root-cause analysis beats guesswork: you’ll learn how to validate assumptions and surface the exact failure (permissions, OOM, missing symbol, bad JSON, throttled network).

• Reproducibility saves time: capturing environment state and inputs now prevents “works-on-my-machine” purgatory later.

---

Quick toolbox install (apt, dnf, zypper)

These are lean, cross-distro tools that cover environment, data, performance, and networking. Install with your distro’s package manager.

Apt (Debian/Ubuntu and derivatives):

sudo apt update
sudo apt install -y \
  git git-lfs \
  python3 python3-venv python3-pip \
  jq curl htop tmux lsof strace ripgrep fzf sysstat nvtop \
  build-essential pkg-config python3-dev

Dnf (Fedora/RHEL/CentOS Stream):

sudo dnf install -y \
  git git-lfs \
  python3 python3-pip \
  jq curl htop tmux lsof strace ripgrep fzf sysstat nvtop \
  gcc gcc-c++ make pkg-config python3-devel

Zypper (openSUSE Leap/Tumbleweed):

sudo zypper refresh
sudo zypper install -y \
  git git-lfs \
  python3 python3-pip \
  jq curl htop tmux lsof strace ripgrep fzf sysstat nvtop \
  gcc gcc-c++ make pkg-config python3-devel

Tip:

git lfs install

Use Git LFS when working with large model weights or datasets to avoid partial checkouts and mysterious “file looks like text” issues.

---

Checklist 1: Environment sanity (prove the machine is what you think it is)

Make sure the OS, resources, Python, and optional GPU stack are correct before looking anywhere else.

# system & OS
uname -a
. /etc/os-release && echo "$PRETTY_NAME"

# disk and memory at a glance
df -h .
free -h || true

# python layout and isolation
command -v python3
python3 --version
python3 -c 'import sys,platform; print(sys.executable, platform.platform())'

# recommended: use a fresh venv per project
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Optional GPU presence (NVIDIA):

if command -v nvidia-smi >/dev/null 2>&1; then
  nvidia-smi
else
  echo "No nvidia-smi found (no NVIDIA driver or not on PATH)."
fi

Live GPU view (optional):

nvtop

Why this works:

• Early detection of tiny mismatches (Python minor versions, disk full, no driver) prevents hours of Python-level flailing.

• Always isolate via venv—dependency pinning without isolation isn’t reproducible.

---

Checklist 2: Dependencies and native builds (prove the wheels are on)

Python dependency problems often masquerade as runtime crashes, symbol errors, or mysterious hangs.

# within your virtualenv:
pip install -r requirements.txt
pip check  # detects broken or incompatible deps

# snapshot for reproducibility
pip list --format=freeze > requirements.lock

# is the compiler toolchain present (when native wheels aren’t available)?
gcc --version
pkg-config --version

Import checks for heavyweight libs:

python - <<'PY'
try:
    import torch
    print("torch:", torch.__version__)
    print("CUDA available:", getattr(torch, "cuda", None) and torch.cuda.is_available())
except Exception as e:
    print("Torch import failed:", e)
PY

Why this works:

• Many ML libraries ship prebuilt wheels, but fall back to source builds silently if a wheel isn’t available for your platform/Python—then fail due to missing compilers/headers.

• pip check quickly surfaces incompatible version graphs.

---

Checklist 3: Data sanity (prove your inputs won’t poison the run)

Bad data kills runs more often than bad models. Validate early, fail fast.

Basic size and count:

# largest files (helps catch a single corrupt giant)
find data/ -type f -printf '%s %p\n' | sort -n | tail -5

# line counts for quick sanity on text datasets
wc -l data/*.txt 2>/dev/null || true

Detect control characters and weird bytes:

rg -n --stats '[\x00-\x08\x0B\x0C\x0E-\x1F]' data/ || true

Validate JSON/JSONL:

# does every line parse as JSON?
jq -c . < data/train.jsonl > /dev/null
# quickly sample
shuf -n 3 data/train.jsonl | jq .

CSV numeric sanity (NaN/inf hunting):

awk -F, 'NR>1 { for (i=1;i<=NF;i++) if ($i=="NaN"||$i=="inf"||$i=="+inf"||$i=="-inf") {print "Bad num @ line " NR; exit 1}}' data/train.csv

Why this works:

• Structural data errors cause silent truncation or loader crashes later; cheap up-front checks save GPU-hours.

• Control characters in text corpora can explode tokenizers.

---

Checklist 4: Runtime and performance (prove where time and memory go)

If it’s slow, blocked, or OOM’ing, measure before you guess.

CPU/memory/GPU monitoring:

htop

I/O and system pressure:

# requires sysstat
iostat -xz 1
vmstat 1

Import-time profiling (Python):

python3 -X importtime -c "import your_model_package" 2> importtime.log
sort -nrk2 importtime.log | head

Trace syscalls to catch missing files, timeouts, or permission errors:

strace -f -tt -o trace.log python3 train.py --epochs 1
rg -n 'ENOENT|ETIMEDOUT|EACCES|ECONN' trace.log | head

File descriptor leaks:

pidof python3 | xargs -r -I{} lsof -p {} | wc -l

Optional GPU memory pressure:

if command -v nvidia-smi >/dev/null; then
  watch -n 1 nvidia-smi
fi

Why this works:

• Many “stalls” are I/O-bound (slow network/NFS) or waiting on locks; strace and iostat reveal it immediately.

• Import-time hotspots can dwarf your actual compute; shaving seconds off imports pays back in iterations.

---

Checklist 5: Networking and APIs (prove the wire is fine)

When your run hits remote endpoints for data, tokens, or model hosting, verify connectivity and payloads.

Connectivity and TLS:

curl -fsS https://api.example.com/health | jq .

Verbose diagnostics:

curl -v https://api.example.com/health 2>&1 | tee curl.log

Proxy/DNS sanity:

env | rg -n 'HTTP_PROXY|HTTPS_PROXY|NO_PROXY' || true
getent hosts api.example.com

POST a minimal JSON payload:

curl -fsS -H 'Content-Type: application/json' \
  -d '{"ping":"pong"}' \
  https://api.example.com/echo | jq .

Why this works:

• Most API errors reduce to TLS/proxy/DNS/environment issues; starting with curl and jq isolates network from application code.

---

Real-world mini playbooks

1) Model won’t load: CUDA out-of-memory

• Confirm memory pressure:

nvidia-smi

• Reduce batch size and disable unnecessary gradients:

python3 train.py --batch-size 8 --grad-accum 4 --precision bf16

• Free zombie processes:

pkill -f python  # careful: stops all Python runs in your session

• If using multiple workers, reduce num_workers or pin memory to limit host-GPU thrash.

2) Training stuck at first batch (I/O-bound data)

• Check disk and I/O:

iostat -xz 1

• Is data on a slow mount or NFS?

df -hT data/

• Mitigate: cache locally (TMPDIR), reduce compression, pre-shard, or increase prefetch_factor in your dataloader.

3) ImportError: cannot open shared object file (native dep mismatch)

• Identify missing libs:

strace -f -e file -o files.trace python -c "import your_native_lib"
rg 'ENOENT.*\.so' files.trace

• Reinstall matching wheels or add system headers (e.g., python3-devel, gcc, pkg-config), then reinstall the package:

pip install --force-reinstall your_native_lib

---

Drop-in script: ai-env-diag.sh

Add this to your repo to capture environment facts for bug reports and reproducibility.

#!/usr/bin/env bash
set -euo pipefail

echo "=== OS ==="
uname -a
. /etc/os-release && echo "$PRETTY_NAME"

echo "=== Resources ==="
df -h .
free -h || true

echo "=== Python ==="
command -v python3
python3 --version
python3 - <<'PY'
import sys, platform
print("executable:", sys.executable)
print("version:", sys.version.replace("\n"," "))
print("venv active:", (hasattr(sys, "base_prefix") and sys.prefix != sys.base_prefix))
print("platform:", platform.platform())
PY

echo "=== Pip freeze (top 50) ==="
python3 -m pip list --format=freeze 2>/dev/null | head -n 50 || true

echo "=== GPU ==="
if command -v nvidia-smi >/dev/null 2>&1; then nvidia-smi || true; else echo "No nvidia-smi"; fi

echo "=== I/O hotpaths ==="
iostat -xz 1 1 2>/dev/null || true

echo "=== Recent kernel messages (errors only) ==="
dmesg --ctime --level=err 2>/dev/null | tail -n 20 || true

echo "=== Git status (if repo) ==="
git rev-parse --is-inside-work-tree >/dev/null 2>&1 && { git status -sb; git rev-parse HEAD; } || true

Usage:

bash ai-env-diag.sh | tee diag.txt

---

Conclusion and next steps

AI troubleshooting doesn’t have to be guesswork. With a small, cross-distro toolbox and five focused checklists—environment, dependencies, data, runtime, and network—you can turn late-night mysteries into reproducible, fixable tickets.

• Start by installing the toolbox (apt/dnf/zypper commands above).

• Commit ai-env-diag.sh to your repos and include diag.txt in bug reports.

• Turn the checklists into project-specific runbooks—pin versions, validate data upfront, and trace first, guess later.

If you want a printable one-pager of these checklists or a minimal Bash “AI doctor” script tailored to your stack, ask and I’ll generate it.