Posted on
• Artificial Intelligence

Artificial Intelligence Infrastructure Dashboards

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Infrastructure Dashboards: From Zero to Insight on Linux (with Bash)

You wouldn’t fly a plane without instruments—so why train or serve AI models without live visibility into GPUs, CPUs, memory, disks, and network? If your training jobs are slow, your inference QPS dips, or your GPUs idle while data pipelines thrash disks, the quickest win is a clear, reliable dashboard. This guide shows you how to build an AI infrastructure dashboard on Linux using Prometheus, Node Exporter, and Grafana—end to end, from install to first panels—using Bash-friendly steps and distro-agnostic package commands.

Why dashboards for AI infra are non‑negotiable

  • AI workloads are resource hungry and spiky. Without observability, you’ll misattribute bottlenecks (e.g., blaming the model when the data loader is starved).

  • Dashboards unify signals across compute, storage, and network—and (optionally) GPUs—so you can correlate utilization, throughput, and saturation in one place.

  • The stack below is free, proven, and scriptable with Bash, so you can standardize monitoring across workstations, bare-metal, and clusters.

What we’ll build

  • Prometheus: time-series database and scraper

  • Node Exporter: per-node system metrics (CPU, RAM, disk, network)

  • Grafana: dashboards

  • Optional: GPU metrics using NVIDIA’s DCGM exporter (pointer provided)

You’ll install Prometheus and Grafana on a “monitoring” host, and Node Exporter on every node you want to watch.

1) Install the monitoring stack (Prometheus, Node Exporter, Grafana)

Run these on your Linux hosts using your distro’s package manager. Commands are grouped by role.

Tip: For repeatability, drop these into Bash scripts or Ansible.

A. Monitoring server (Prometheus + Grafana)

Debian/Ubuntu (apt):

sudo apt update
# Prometheus
sudo apt install -y prometheus
# Add official Grafana repo for up-to-date Grafana
sudo apt install -y curl gnupg
curl -fsSL https://packages.grafana.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/grafana.gpg
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana
# Start services
sudo systemctl enable --now prometheus
sudo systemctl enable --now grafana-server

RHEL/CentOS/Fedora (dnf):

sudo dnf install -y epel-release
# Prometheus (on some RHEL-family distros the package is prometheus2)
sudo dnf install -y prometheus || sudo dnf install -y prometheus2
# Add official Grafana repo
sudo tee /etc/yum.repos.d/grafana.repo >/dev/null <<'EOF'
[grafana]
name=Grafana OSS
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
EOF
sudo dnf makecache
sudo dnf install -y grafana
# Start services
sudo systemctl enable --now prometheus
sudo systemctl enable --now grafana-server

openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y prometheus
# Add official Grafana repo
sudo rpm --import https://packages.grafana.com/gpg.key
sudo zypper ar -f https://packages.grafana.com/oss/rpm grafana
sudo zypper refresh
sudo zypper install -y grafana
# Start services
sudo systemctl enable --now prometheus
sudo systemctl enable --now grafana-server

Open firewall ports (Prometheus 9090, Grafana 3000):

# firewalld
sudo firewall-cmd --permanent --add-port=9090/tcp --add-port=3000/tcp
sudo firewall-cmd --reload
# or UFW
sudo ufw allow 9090,3000/tcp

Grafana UI will be at:

http://<monitoring-host>:3000
# default login: admin / admin (you'll be prompted to change it)

B. All nodes (Node Exporter)

Install and start Node Exporter on every node (including the monitoring server if you want its metrics too).

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y prometheus-node-exporter
sudo systemctl enable --now prometheus-node-exporter

RHEL/CentOS/Fedora (dnf):

sudo dnf install -y epel-release
sudo dnf install -y node_exporter
sudo systemctl enable --now node_exporter

openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y prometheus-node_exporter
sudo systemctl enable --now prometheus-node_exporter

Open Node Exporter port (9100):

# firewalld
sudo firewall-cmd --permanent --add-port=9100/tcp
sudo firewall-cmd --reload
# or UFW
sudo ufw allow 9100/tcp

Sanity check from the monitoring host:

curl -s http://<node-ip>:9100/metrics | head

2) Point Prometheus at your nodes

Edit Prometheus’ scrape config on the monitoring server at /etc/prometheus/prometheus.yml. Add your node IPs or DNS names under a job named node:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Scrape Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Scrape Node Exporter on every node
  - job_name: 'node'
    static_configs:
      - targets:
        - '10.0.0.11:9100'
        - '10.0.0.12:9100'
        - '10.0.0.13:9100'
        # add more here

Reload Prometheus:

sudo systemctl reload prometheus
# or if reload is unsupported by your build:
sudo systemctl restart prometheus

Quick verify in Prometheus UI:

http://<monitoring-host>:9090/targets
# All "node" targets should be "UP"

Bash tip to append many nodes:

NODES=("10.0.0.11" "10.0.0.12" "10.0.0.13")
for n in "${NODES[@]}"; do
  echo "        - '${n}:9100'" | sudo tee -a /etc/prometheus/prometheus.yml
done
sudo systemctl reload prometheus

3) Visualize: import ready-made dashboards in Grafana

  • Add Prometheus as a data source in Grafana:

    • URL: http://localhost:9090 (from Grafana’s perspective, if on same host)

  • Import these battle-tested dashboards (Grafana.com > Dashboards > Import by ID):

    • Node Exporter Full (ID: 1860) – host-level CPU, memory, disks, network
    • Prometheus 2.0 Stats (ID: 3662) – health of your metrics pipeline

Optional: Use the API to add the Prometheus data source via Bash:

curl -s -X POST http://admin:admin@localhost:3000/api/datasources \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Prometheus",
    "type": "prometheus",
    "access": "proxy",
    "url": "http://localhost:9090",
    "basicAuth": false
  }'

After importing “Node Exporter Full,” filter by host to see per-node health. Correlate:

  • CPU% + load averages + process counts (are you CPU-bound?)

  • Memory working set + page faults (are you swapping?)

  • Disk read/write latency + throughput (is I/O the bottleneck?)

  • Network drops + bandwidth (is data ingress starving GPUs?)

4) Add AI-specific signals (GPU, jobs, and storage) when ready

Start simple and iterate. Here are pragmatic add-ons that pay off quickly.

  • GPU metrics (NVIDIA):

    • Use NVIDIA’s DCGM Exporter container to expose rich GPU telemetry on each GPU node (utilization, memory, ECC, PCIe). Prometheus scrapes it; Grafana dashboard ID 12239 is a good start.
    • See: NVIDIA DCGM Exporter docs for installation and run flags on your environment.

  • Workload/job metrics:

    • Export queue depth, throughput (samples/sec), latency percentiles, and error rates from your training/inference services via a Prometheus client library. Expose on a /metrics HTTP endpoint and add a scrape_config in Prometheus.

  • Storage and data pipeline:

    • Add exporters for NFS/SMB, object storage S3 gateways, or your caching layer. Watch I/O latency and cache hit ratios to keep GPUs fed.

Small, Bash-friendly example of adding a custom service to Prometheus:

# Add a job that scrapes your inference server metrics on port 8000
sudo sed -i '/scrape_configs:/a\  - job_name: "inference"\n    static_configs:\n      - targets: ["10.0.0.21:8000","10.0.0.22:8000"]' /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

Real-world win:

  • A team saw GPU utilization stuck at ~30% while disk I/O latency spiked and network RX stayed low. Dashboards confirmed dataloaders were I/O bound. They added local SSD caching, batched reads, and async prefetch—GPU utilization jumped to >85% and epoch time halved.

5) Hardening, housekeeping, and scale

  • Authentication: Secure Grafana with SSO or at least strong credentials; firewall off Prometheus from public networks.

  • Retention and storage: Tune Prometheus with --storage.tsdb.retention.time=30d and ensure enough disk IOPS for your scrape volume.

  • Alerting: Add basic alerts (node down, high load, low disk) and route to email/Slack; Grafana Alerting or Alertmanager both work well.

  • Config as code: Keep /etc/prometheus/prometheus.yml and dashboard JSON in Git; roll changes with scripts.

Troubleshooting quick hits

  • A target is DOWN: Test with curl http://<target>:<port>/metrics from the monitoring host; check firewalls and service status with systemctl status.

  • Grafana can’t query: Verify the Prometheus data source URL from Grafana’s perspective (localhost vs. IP).

  • High Prometheus CPU: Reduce scrape interval or number of time series; consider federation for large fleets.

Call to Action

Don’t wait for the next training slowdown to guess at the root cause. Install the stack today, import the Node Exporter Full dashboard, and get eyes on your fleet in under an hour. Then iterate: add GPU and workload metrics, wire in alerts, and standardize your config with Bash scripts so every new node is observable on day one.

If you want a ready-to-run snippet pack (prometheus.yml templates, firewall rules, and a basic onboarding script), tell me your distro mix and node count—I'll tailor a starter kit you can paste straight into your shell.

Related posts