Posted on
• Artificial Intelligence

Artificial Intelligence Boot Troubleshooting

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Boot Troubleshooting: Get Your AI Stack to Start Cleanly on Linux

You reboot after a driver or CUDA update, and your GPU fans spin—but your model server never comes up. Or worse, your display goes black and SSH is dead right when you need to demo. If you run AI workloads on Linux, “boot” trouble often isn’t about GRUB; it’s about the fragile chain of kernel, drivers, containers, and services that must line up perfectly. This post shows you how to quickly diagnose and fix AI boot/startup issues using Bash, with pragmatic steps you can script and automate.

What you’ll get:

  • A mental model for why AI stacks fail to start on boot

  • 3–5 actionable checks and fixes you can run now

  • Real-world examples you can adapt to your environment

  • Exact commands for Debian/Ubuntu (apt), Fedora/RHEL (dnf), and openSUSE (zypper)

Note: Commands are safe to paste, but read comments and outputs. For production, test on non-critical nodes first.

Why AI “boot” troubleshooting is its own thing

AI nodes combine layers that are all version- and timing-sensitive:

  • Kernel, initramfs, microcode, and Secure Boot

  • GPU kernel modules (NVIDIA, AMD), firmware, and DKMS

  • CUDA/ROCm userland, cuDNN, NCCL, and BLAS stacks

  • Container runtime (Podman/Docker), cgroups, GPU passthrough hooks

  • Your model server/service and its on-boot ordering

A tiny mismatch—new kernel but old driver, module blocked by Secure Boot, service starting before network or GPU is ready—can bring your AI stack down even if Linux itself “boots.” The fix is usually to align versions and boot order, not to reinstall everything.

Quick tooling setup (recommended)

Install basic hardware and log inspection tools so you can see what’s going on.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y pciutils lshw inxi dmidecode mokutil podman curl git

Fedora/RHEL (dnf):

sudo dnf install -y pciutils lshw inxi dmidecode mokutil podman curl git

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y pciutils lshw inxi dmidecode mokutil podman curl git

Optional GPU monitors (helpful if in repos):

  • Debian/Ubuntu: sudo apt install -y nvtop

  • Fedora/RHEL: sudo dnf install -y nvtop

  • openSUSE: sudo zypper install -y nvtop

1) Prove the hardware-driver chain loads cleanly at boot

Before blaming your app or container, confirm the kernel and GPU stack are healthy.

  • Kernel, microcode, and Secure Boot:
uname -r
mokutil --sb-state
dmesg -T | grep -iE 'secure boot|sb|efi'
  • Check GPU presence and drivers:
lspci -nn | grep -iE 'vga|3d|display'
lsmod | grep -iE 'nvidia|amdgpu|nouveau'
dmesg -T | grep -iE 'nvidia|amdgpu|nouveau|firmware' | tail -n +1
  • If you use NVIDIA and the proprietary module won’t load after a kernel update, it’s often:
    • DKMS didn’t rebuild the module for the new kernel
    • Secure Boot blocking unsigned modules

You can try booting your previous kernel from the GRUB menu to confirm it’s a driver/kernel mismatch. If Secure Boot is enabled and you use unsigned modules, either enroll a Machine Owner Key (MOK) to sign the module or install distro-signed drivers. Disabling Secure Boot is a last resort and a security trade-off—document and justify it if you must.

  • Quick PCI and system inventory:
sudo lshw -short -C display
inxi -Gxx
sudo dmidecode -t bios

If you see firmware or microcode complaints in dmesg, update your system firmware and microcode packages through your regular package manager.

2) Keep kernel, drivers, and CUDA/ROCm in lockstep

Most “worked yesterday, broken today” issues trace back to a version skew introduced by an update.

  • See what kernel(s) you have and which you’re running:
uname -r

  • If a new kernel broke your GPU driver:

    • Temporarily boot the previous kernel from GRUB to restore service.
    • Rebuild/initramfs as needed:
    • Debian/Ubuntu: sudo update-initramfs -u -k all sudo update-grub
    • Fedora/RHEL/openSUSE (dracut-based): sudo dracut --force
    • Reinstall or rebuild GPU drivers for the current kernel. If using DKMS, verify:
    sudo dkms status

  • Pin or stage updates to avoid surprise kernel bumps that strand your drivers:

    • Debian/Ubuntu (example hold; adapt to your packages):
    sudo apt-mark hold linux-image-generic linux-headers-generic
# Later, unhold when you’re ready:
# sudo apt-mark unhold linux-image-generic linux-headers-generic
    • Fedora/RHEL and openSUSE can use versioned kernels side-by-side; schedule maintenance windows and validate DKMS after updates.

  • Confirm runtime-toolkit alignment:

    • CUDA/ROCm userland must match your driver’s supported versions.
    • For NVIDIA, nvidia-smi shows driver and CUDA runtime if installed:
    nvidia-smi

3) Make your AI service start at the right time (systemd)

Many AI daemons fail at boot simply because they start before the network is fully online or before the GPU runtime is ready. Use systemd ordering and health logging.

Example: systemd unit for a containerized model server (Podman). Adjust ExecStart for your app.

Create /etc/systemd/system/ai-model.service:

[Unit]
Description=AI Model Server
Wants=network-online.target
After=network-online.target podman.service
# If NVIDIA GPUs and persistence daemon are used:
# After=nvidia-persistenced.service

[Service]
User=ai
Group=ai
Restart=always
RestartSec=5
Environment=CUDA_VISIBLE_DEVICES=0
# Example Podman run (rootless or rootful). Replace image/args.
ExecStart=/usr/bin/podman run --rm --name ai-model --network host \
  --device nvidia.com/gpu=all \
  ghcr.io/example/ai-model:latest --port 8000
ExecStop=/usr/bin/podman stop -t 10 ai-model

# Ensure logs land in journal
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Enable network-wait and your service:

sudo systemctl enable NetworkManager-wait-online.service
sudo systemctl daemon-reload
sudo systemctl enable --now ai-model.service
sudo systemctl status ai-model.service
sudo journalctl -u ai-model.service -b -e

If you don’t use NetworkManager, substitute the appropriate “wait online” service for your stack (e.g., systemd-networkd-wait-online).

4) Containers, GPUs, and cgroups sanity checks

If you run your AI stack in containers, confirm the runtime can actually see the GPU and the right cgroup version.

  • Check cgroups version:
stat -fc %T /sys/fs/cgroup
# cgroup2fs means unified (v2). Many recent distros default to v2.
  • Podman is generally cgroup-v2 friendly out of the box. If you use Docker, ensure your GPU toolkit (e.g., NVIDIA container toolkit) matches your driver and cgroup mode. When in doubt, test a simple GPU container:
podman run --rm --device nvidia.com/gpu=all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

If that fails, logs will often point to missing hooks or permissions.

Install Podman (if you skipped earlier):

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y podman

Fedora/RHEL (dnf):

sudo dnf install -y podman

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y podman

Tip: For rootless Podman to access GPUs, ensure the appropriate device nodes and container hooks are present for your vendor. If using NVIDIA, follow the vendor’s container toolkit instructions corresponding to your distro, driver, and cgroup version.

5) Safe recovery modes when the box won’t come up

When a GUI or driver blocks normal boot, get back in cleanly so you can fix it.

  • Temporarily boot to multi-user (text) mode from GRUB:

    • Edit the kernel line and append:
    systemd.unit=multi-user.target
    • Then troubleshoot drivers, DKMS, and services without the display stack.

  • Roll back the last kernel:

    • From GRUB’s Advanced options, choose the previous working kernel. If it works, rebuild drivers and initramfs for the current kernel before trying again.

  • Rebuild initramfs:

    • Debian/Ubuntu:
    sudo update-initramfs -u -k all
sudo update-grub
    • Fedora/RHEL/openSUSE:
    sudo dracut --force

  • If nouveau or amdgpu is colliding with your proprietary driver, you can temporarily blacklist it to verify the hypothesis, then pursue a signed, supported driver path.

Real-world examples you can copy

1) Secure Boot blocked the NVIDIA module after a kernel update

  • Symptoms: nvidia module missing; dmesg shows module verification errors; nvidia-smi fails.

  • Checks:

    mokutil --sb-state
dmesg -T | grep -i 'module.*nvidia.*signature'

  • Fix options:

    • Install distro-signed GPU drivers so modules load under Secure Boot.
    • Or enroll your own MOK and sign the module (advanced).
    • As a last resort for labs (not recommended for prod), disable Secure Boot in firmware, document the change, and re-enable later with signed modules.

2) Model service starts before network or GPU is ready

  • Symptoms: system boots, but your service exits early with “connection refused” or “no GPU devices found.”

  • Fix:

    • Add Wants=network-online.target and After=network-online.target to your unit.
    • Enable the wait-online service:
    sudo systemctl enable NetworkManager-wait-online.service
    • If using NVIDIA, ensure nvidia-persistenced is enabled so device nodes are ready early (package/enablement varies by distro/driver packaging).
    • Validate by rebooting and checking:
    sudo journalctl -u ai-model.service -b -e

3) Container cannot access GPU after distro upgrade (cgroup v2)

  • Symptoms: Container runs, but nvidia-smi inside container fails; host nvidia-smi works.

  • Checks:

    stat -fc %T /sys/fs/cgroup
podman run --rm --device nvidia.com/gpu=all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

  • Fix:

    • Update GPU container toolkit and hooks to versions that support your cgroup mode.
    • Restart container runtime and retry.

A short checklist you can automate

  • After every kernel or driver update:

    • Reboot during a maintenance window.
    • Verify lsmod, dmesg, and nvidia-smi (or AMD equivalents).
    • Rebuild initramfs if needed.

  • For services:

    • Ensure correct systemd dependencies: network-online, container runtime, GPU persistence.
    • Log to journal and set Restart=always.

  • For containers:

    • Validate GPU passthrough with a tiny test image.
    • Confirm cgroup version and toolkit compatibility.

  • For security:

    • Prefer signed, distro-packaged drivers with Secure Boot.
    • Document exceptions.

Conclusion and next steps (CTA)

Your AI node doesn’t have to be fragile at boot. By verifying the driver chain early, keeping kernel and runtime versions aligned, and controlling service start order, you can make AI services come up predictably every time.

Next steps:

  • Put the checks above into a single “post-update health” script and run it on every AI node after updates.

  • Convert your model startup into a systemd unit with proper dependencies and logging.

  • If you use containers, standardize on Podman or Docker with a known-good GPU toolkit version and test image.

If you want a ready-to-adapt systemd unit and health-check script, reply with your distro, GPU vendor, and how you deploy (bare-metal vs container), and I’ll tailor a drop-in template.

Related posts