AI on Linux: Troubleshooting Case Studies for Bash‑First Engineers

If your model trained fine yesterday but today the GPU is “missing,” inference got slow, or your service keeps throwing 5xx, you’re not alone. Modern AI stacks depend on drivers, kernels, Python wheels, thread libraries, IO paths, and APIs that can fail in surprising ways. The fastest path to a fix starts in the shell: measure, isolate, and only then change.

This article walks through real troubleshooting case studies you can reproduce with Bash. You’ll get a small, dependable toolbox, concrete symptoms to watch for, and actionable commands that lead to root causes quickly.

• Value: Save hours of guesswork by checking the right things in the right order.

• Scope: GPU detection, CUDA/wheel mismatches, OOM kills, slow inference, and flaky APIs.

• Target: Linux users comfortable with Bash and system tools.

Quick setup: install the troubleshooting toolbox

Tools we’ll use: curl, jq, git, Python 3 + venv/pip, strace, lsof, pciutils (lspci), nvtop (GPU monitor), htop, iproute2 (ss), sysstat (iostat), numactl.

Apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y \
  curl jq git python3 python3-venv python3-pip \
  strace lsof pciutils nvtop htop iproute2 sysstat numactl

Dnf (Fedora/RHEL/CentOS Stream):

sudo dnf install -y \
  curl jq git python3 python3-pip \
  strace lsof pciutils nvtop htop iproute sysstat numactl

Zypper (openSUSE):

sudo zypper refresh
sudo zypper install -y \
  curl jq git python3 python3-virtualenv python3-pip \
  strace lsof pciutils nvtop htop iproute2 sysstat numactl

Optional but useful:

• NVIDIA driver utilities (nvidia-smi). These ship with the proprietary driver.
  • Ubuntu/Debian: sudo ubuntu-drivers autoinstall sudo reboot
  • Fedora: enable RPM Fusion and install akmod-nvidia sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \ https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm sudo dnf install -y akmod-nvidia sudo reboot
  • openSUSE (example for Tumbleweed; pick the repo matching your distro/driver gen): sudo zypper ar -f https://download.nvidia.com/opensuse/tumbleweed/ nvidia sudo zypper install -y nvidia-driver-G06 sudo reboot

Create a fresh Python virtual environment for isolation:

python3 -m venv ~/venvs/ai-debug
source ~/venvs/ai-debug/bin/activate
python -m pip install --upgrade pip

Case study 1: “GPU missing” or Torch falls back to CPU

Symptoms:

• nvidia-smi shows nothing or errors.

• Python says torch.cuda.is_available() -> False.

• Your process runs but the GPU is idle.

Checklist: 1) Confirm the device is on the PCIe bus:

lspci | grep -i nvidia

2) Confirm the kernel driver and runtime:

nvidia-smi

• If this fails, the driver is not loaded or mismatched with the kernel. Reinstall/update driver, ensure Secure Boot isn’t blocking it, and reboot.

3) Confirm Torch sees CUDA:

python - <<'PY'
import torch, platform
print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("CUDA version (torch):", torch.version.cuda)
print("Device count:", torch.cuda.device_count())
print("Device name:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "N/A")
print("Python:", platform.python_version())
PY

4) Fix CUDA/wheel mismatches:

• Match Torch wheels to your CUDA runtime. nvidia-smi shows “CUDA Version: 12.x” which maps to a wheel tag like cu121/cu122.

• Example installs (pick one; do this inside your venv):

CUDA 12.1:

pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision torchaudio

CUDA 12.4:

pip install --index-url https://download.pytorch.org/whl/cu124 torch torchvision torchaudio

CPU-only (no NVIDIA driver):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

5) Pin the visible GPU if you run multi-GPU boxes:

CUDA_VISIBLE_DEVICES=0 python your_script.py

Why this works: 90% of “GPU missing” bugs are driver not loaded, incompatible kernel modules, or a wheel compiled for a different CUDA runtime. Checking PCIe → driver → Python narrows it fast.

Case study 2: CUDA OOM, fragmentation, and “it fails after a few epochs”

Symptoms:

• RuntimeError: CUDA out of memory.

• Fails intermittently (e.g., only after a few validation runs).

• Memory appears “free” but allocation fails due to fragmentation.

Observe first:

watch -n1 nvidia-smi

Quick mitigations:

• Reduce peak memory: smaller batch, shorter sequence length, use fp16/bf16.

• Enable gradient checkpointing:

# PyTorch
from torch.utils.checkpoint import checkpoint
# wrap parts of your model forward with checkpoint(...)

• Accumulate gradients instead of large batches:

# Pseudocode
loss = model(batch) / accum_steps
loss.backward()
if step % accum_steps == 0:
    optimizer.step(); optimizer.zero_grad()

• Free caches between validation/training:

import torch
torch.cuda.empty_cache()

• Limit workspace sizes and determinism if needed:

import torch
torch.backends.cudnn.benchmark = True

Repro and isolate:

• Build a minimal script that allocates tensors to verify if the issue is model-specific or environmental.

Why this works: Many OOMs are about the peak working set and allocator fragmentation. Lowering peak usage or restructuring computation reduces fragmentation pressure.

Case study 3: Training killed by the kernel OOM killer or file-handle exhaustion

Symptoms:

• Process “just dies.”

• dmesg shows Killed process N (python).

• Dataloaders crash under load, or “Too many open files.”

Check kernel logs:

dmesg -T | grep -Ei 'out of memory|killed process'
journalctl -k -b | grep -Ei 'oom|killed process'

Check memory and swap:

free -h
vmstat 1

Check open files and limits:

ulimit -n
lsof -p $(pgrep -n python) | wc -l

Raise file descriptor limits (session-level):

ulimit -n 1048576

Persistent raise via systemd unit override (for a service mytrainer.service):

sudo systemctl edit mytrainer.service
# Then add:
# [Service]
# LimitNOFILE=1048576
# LimitNPROC=131072
sudo systemctl daemon-reload
sudo systemctl restart mytrainer

If tmp fills up (huggingface cache, dataloader spills), redirect temp:

df -h
du -sh ~/.cache/huggingface
export TMPDIR=/mnt/fast/tmp

Overcommit tuning (be careful and test):

sudo sysctl -w vm.overcommit_memory=1

Why this works: Many “mystery exits” are kernel OOM kills, not Python exceptions. Logs plus limits reveal whether you’re hitting system ceilings, not code bugs.

Case study 4: Slow inference that’s actually I/O, threads, or NUMA

Symptoms:

• GPU utilization is low; end-to-end latency is high.

• CPU pinned or one core overloaded; data loader or decode bound.

Measure:

iostat -xz 1    # from sysstat
vmstat 1
htop            # look for CPU hotspots, steal, migration
ss -tnp | head  # look for network stalls

Common fixes:

• Threading for BLAS/NumPy/PyTorch (avoid oversubscription):

export OMP_NUM_THREADS=4
export MKL_NUM_THREADS=4
python - <<'PY'
import torch, os
print("intra-op before:", torch.get_num_threads())
torch.set_num_threads(int(os.getenv("OMP_NUM_THREADS","4")))
print("intra-op after:", torch.get_num_threads())
PY

• Pin to a NUMA node to reduce cross‑socket traffic:

numactl --hardware
numactl --cpunodebind=0 --membind=0 python infer.py

• Preprocess on the right device: decode/resize on CPU in batches, keep GPU fed.

• Warm up models to trigger autotuning:

python - <<'PY'
import torch
m=torch.nn.Linear(4096,4096).cuda()
x=torch.randn(64,4096,device='cuda'); 
for _ in range(10): m(x)  # warmup
PY

• Profile dataloaders: ensure pinned memory, nonblocking transfers, appropriate num_workers.

Why this works: Most “slow GPU” incidents are starved GPUs. The bottleneck is data movement, CPU threads, or NUMA penalties—not the kernel you think is slow.

Case study 5: API timeouts, 429s, and flaky upstreams

Symptoms:

• HTTP 429/5xx from model gateways.

• SSL errors or intermittent DNS failures.

Reproduce with curl and log everything:

curl -v https://api.example.com/health

Apply retries and backoff:

curl --retry 5 --retry-all-errors --max-time 30 https://api.example.com/generate \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"hello"}' | jq .

Pin DNS and verify resolution:

getent hosts api.example.com

Check system trust store if TLS fails:

• Apt:

  sudo apt install -y ca-certificates
  sudo update-ca-certificates
  

• Dnf:

  sudo dnf install -y ca-certificates
  sudo update-ca-trust
  

• Zypper:

  sudo zypper install -y ca-certificates
  sudo update-ca-certificates
  

Trace a stuck client to see socket states:

pid=$(pgrep -n python)
sudo strace -f -e trace=network -p "$pid"

Observe service logs:

journalctl -u your-api.service --since "1 hour ago"

Why this works: Treat APIs like any flaky network dependency—retry, backoff, verify DNS/TLS, and only then dig into app logic.

A tiny, reusable GPU sanity check

Drop this into scripts to fail fast:

python - <<'PY'
import torch, sys
ok = torch.cuda.is_available()
print("CUDA available:", ok)
if ok:
    print("Device:", torch.cuda.get_device_name(0))
    a = torch.randn(4096,4096, device='cuda')
    b = torch.mm(a,a)
    print("Sanity matmul:", b.shape)
else:
    sys.exit("No CUDA device detected")
PY

Pulling it together: a practical triage order

• Hardware/driver first:

  • lspci → nvidia-smi → dmesg/journalctl

• Python/runtime:

  • venv → correct torch wheel for CUDA → minimal repro

• Resources:

  • watch nvidia-smi → free/vmstat → ulimit/df

• Performance:

  • iostat/vmstat/htop → thread env vars → NUMA pinning

• Network/API:

  • curl -v with retries → DNS/TLS store → strace

Conclusion and next steps

AI outages rarely start in your model code. Bash gives you the fastest route to root cause: measure, isolate, then change. Save this checklist, build a minimal repro for every incident, and automate the sanity checks into your CI and service health endpoints.

Call to action:

• Install the toolbox on your machines now and commit a gpu_sanity.sh and api_sanity.sh to your repos.

• Add a runbook with the triage order above for your on-call rotation.

• Consider wrapping these checks in a GitHub Actions or systemd health unit so failures surface early.

If you want a ready-made script pack that implements these checks end-to-end, say the word and I’ll generate one tailored to your distro and GPU stack.