Artificial Intelligence Metrics That Matter: A Bash‑Friendly Guide

You shipped a model. It “works.” But is it good? Is it fast enough? Fair enough? Cheap enough? Without the right metrics, AI work becomes guesswork—and guesswork doesn’t scale.

This post shows you how to measure what matters for AI using tools you can run directly from your Linux shell. You’ll get practical commands, real examples, and distro‑specific install instructions (apt, dnf, zypper) so you can start benchmarking today.

Why metrics matter

• They reduce risk: AUC, F1, BLEU/ROUGE, latency, and P99s catch regressions before users do.

• They align cost with value: Throughput and memory footprints determine your infra bill.

• They accelerate iteration: Reproducible, automatable measurements let you compare baselines quickly.

• They support trust: Transparent metrics help teams and stakeholders agree on “good enough.”

Before you start: install the tools

We’ll use a few CLI and Python tools. Choose your distro and run:

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-sklearn python3-pip jq time hyperfine sysstat powertop
# Optional: NVIDIA GPU users – install the proprietary driver, then nvidia-smi becomes available:
# sudo ubuntu-drivers autoinstall && sudo reboot
# After reboot: nvidia-smi

• Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y python3 python3-scikit-learn python3-pip jq time hyperfine sysstat powertop
# Optional: NVIDIA GPU users – install the proprietary driver per vendor/RPM Fusion docs; then run `nvidia-smi`.

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-scikit-learn python3-pip jq time hyperfine sysstat powertop
# Optional: NVIDIA GPU users – install the proprietary driver per SUSE/NVIDIA docs; then run `nvidia-smi`.

For text generation metrics, install two Python packages (after python3-pip is installed):

pip3 install --user sacrebleu rouge-score

Tip: add ~/.local/bin to PATH if needed:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

---

1) Classification quality: accuracy is not enough

Accuracy can hide problems (e.g., class imbalance). Track precision, recall, and F1—plus a confusion matrix—to understand trade‑offs.

Example: given a CSV file predictions_cls.csv with headers:

y_true,y_pred
1,0
0,0
1,1
...

Compute core metrics right from the shell:

python3 - <<'PY'
import csv
from sklearn.metrics import classification_report, confusion_matrix, f1_score, precision_score, recall_score

y_true, y_pred = [], []
with open('predictions_cls.csv') as f:
    for row in csv.DictReader(f):
        y_true.append(int(row['y_true']))
        y_pred.append(int(row['y_pred']))

print(classification_report(y_true, y_pred, digits=4))
print("Confusion matrix:")
print(confusion_matrix(y_true, y_pred))
PY

Why it matters:

• Precision: how often positive predictions are correct.

• Recall: how many actual positives you caught.

• F1: balances precision and recall, useful when classes are imbalanced.

Real‑world tip:

• For fraud, safety, or medical triage, prioritize recall or F1 over raw accuracy.

---

2) Regression error: measure the miss, not just the mean

For numeric predictions (prices, demand, time estimates), track MAE, RMSE, and R². RMSE penalizes large errors; MAE is easier to interpret in “units.”

Prepare predictions_reg.csv:

y_true,y_pred
10.0,9.5
5.2,6.0
...

Run:

python3 - <<'PY'
import csv, math
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

y_true, y_pred = [], []
with open('predictions_reg.csv') as f:
    for row in csv.DictReader(f):
        y_true.append(float(row['y_true']))
        y_pred.append(float(row['y_pred']))

mae = mean_absolute_error(y_true, y_pred)
rmse = math.sqrt(mean_squared_error(y_true, y_pred))
r2 = r2_score(y_true, y_pred)
print(f"MAE : {mae:.4f}")
print(f"RMSE: {rmse:.4f}")
print(f"R²  : {r2:.4f}")
PY

Why it matters:

• If your users feel large misses more than small ones, optimize for RMSE.

• If interpretability matters (“off by $X on average”), MAE is your friend.

---

3) Generative quality: BLEU/ROUGE you can run on a laptop

For translation/summarization and similar tasks, you can compute BLEU and ROUGE quickly.

Assume line‑aligned files:

• refs.txt (reference/ground truth)

• hyps.txt (your model’s outputs)

BLEU and ChrF with sacreBLEU:

sacrebleu refs.txt -i hyps.txt -m bleu chrf -w 2

ROUGE (F1) via rouge-score:

python3 - <<'PY'
from rouge_score import rouge_scorer
refs = [l.strip() for l in open('refs.txt')]
hyps = [l.strip() for l in open('hyps.txt')]
scorer = rouge_scorer.RougeScorer(['rouge1','rouge2','rougeL'], use_stemmer=True)

n = min(len(refs), len(hyps))
tot = {'rouge1':0.0,'rouge2':0.0,'rougeL':0.0}
for r,h in zip(refs[:n], hyps[:n]):
    s = scorer.score(r, h)
    for k in tot:
        tot[k] += s[k].fmeasure

print(f"ROUGE-1 F1: {tot['rouge1']/n:.4f}")
print(f"ROUGE-2 F1: {tot['rouge2']/n:.4f}")
print(f"ROUGE-L F1: {tot['rougeL']/n:.4f}")
PY

Why it matters:

• BLEU is common for MT; ROUGE‑L correlates well with summary overlap.

• Always sanity‑check with human review; automated metrics are guides, not judges.

---

4) Latency, throughput, and resource usage: speed is a feature

Users notice delays and tails. Measure mean, variance, and P95/P99 latency, plus memory/CPU/GPU.

Benchmark a local inference command (e.g., ./infer.sh "prompt"):

hyperfine -w 3 -r 15 './infer.sh "What is the capital of France?"'

Get detailed timing and memory (Max RSS) for a single run:

/usr/bin/time -v ./infer.sh "What is the capital of France?"

If your model runs on NVIDIA GPUs, sample utilization during a workload:

# Start GPU sampling in the background (optional)
nvidia-smi dmon -s pucmem -o DT -f gpu_metrics.csv & MON=$!

# Run your workload
hyperfine -w 2 -r 5 './infer.sh "Generate a haiku about the ocean."'

# Stop sampling
kill $MON
echo "GPU metrics saved to gpu_metrics.csv"

CPU utilization with sysstat:

# System-wide CPU stats (1s interval, 10 samples)
mpstat 1 10

# Per-process CPU/mem I/O for a PID (replace $PID)
pidstat -r -u -d -p $PID 1

Why it matters:

• P95/P99 latencies drive perceived quality.

• Memory spikes can cause OOM kills; track Max RSS.

• GPU utilization helps you right-size batch sizes and hardware.

---

5) Production health: compute tail latencies from logs

Store each request’s duration. If you have latencies.jsonl like:

{"ts":"...","latency_ms":112}
{"ts":"...","latency_ms":87}
{"ts":"...","latency_ms":240}
...

Compute P50/P95/P99 with jq:

jq -s '
  [.[].latency_ms] as $a
  | ($a|length) as $n
  | $a|sort
  | {count:$n,
     p50: .[($n*0.50|floor)],
     p95: .[($n*0.95|floor)],
     p99: .[($n*0.99|floor)]}
' latencies.jsonl

Why it matters:

• Means hide pain. Tails (P95/P99) are what users complain about.

• Put this in a cronjob or CI step to catch regressions automatically.

---

Putting it together: a minimal, automatable checklist

• Classifiers: precision, recall, F1, confusion matrix

• Regressors: MAE, RMSE, R²

• Generators: BLEU/ChrF, ROUGE‑L

• Performance: mean latency, P95/P99, Max RSS, GPU/CPU utilization

• Continuous checks: scripts wired into CI comparing “new vs baseline”

Example CI diff (bash exit non‑zero on regression):

new=$(hyperfine -i -r 5 './infer.sh "x"' --export-json new.json >/dev/null 2>&1; jq '.results[0].mean' new.json)
base=$(jq '.results[0].mean' baseline.json)
awk -v n="$new" -v b="$base" 'BEGIN { if (n > b*1.05) exit 1 }' || { echo "Latency regressed >5%"; exit 1; }

---

Conclusion and next steps

Good AI isn’t just clever—it’s measurable. Start small: 1) Pick 2–3 quality metrics that reflect user value. 2) Add latency and memory measurements to your run scripts. 3) Automate a baseline comparison in CI.

Then grow:

• Track tails and errors in production logs.

• Version your datasets and metrics next to your code.

• Share a short “model card” with top‑line metrics on every release.

If you want a ready‑to‑run starter pack (scripts, sample CSV/JSONL, and Make targets for the commands above), ask me for the “AI Metrics Bash Starter Kit,” and I’ll generate it tailored to your stack and hardware.