Artificial Intelligence on Rocky Linux: From Zero to Local Inference

You don’t need a GPU farm or a managed cloud to get real value from AI. If you run Rocky Linux for its stability and RHEL compatibility, you already have an excellent foundation for reliable, compliant, on‑prem AI. In this guide, you’ll go from a clean Rocky Linux node to running and serving real AI workloads—entirely locally—with clear, Bash-friendly steps.

What you’ll get:

• Why Rocky Linux is a strong AI platform for on‑prem and edge

• System prep that works on Rocky and translates to apt/dnf/zypper

• 3–5 concrete, reproducible examples: from text classification to local LLMs and a production-grade REST service

• Sensible performance and security notes for SELinux environments

Why AI on Rocky Linux?

• Enterprise stability and binary compatibility with RHEL makes Rocky ideal for production AI where uptime and reproducibility matter.

• Built-in SELinux and long support windows align with governed, on‑prem deployments.

• Predictable packaging via AppStream, CRB (CodeReady Builder), and EPEL provide the wide ecosystem AI likes (compilers, Python tooling, etc.).

• Works great with containers (Podman/Docker), schedulers, and air‑gapped workflows.

Core setup: prepare your node

The commands below include apt, dnf, and zypper so you can translate easily across environments. Rocky Linux users should run the dnf versions.

1) Update and enable developer repos

• dnf (Rocky/RHEL/Fedora-like):

  sudo dnf -y update
  sudo dnf -y install epel-release dnf-plugins-core
  sudo dnf config-manager --set-enabled crb   # Rocky 9: enable CodeReady Builder
  sudo dnf groupinstall -y "Development Tools"
  sudo dnf -y install git cmake pkgconfig python3 python3-pip python3-virtualenv
  

• apt (Debian/Ubuntu):

  sudo apt update
  sudo apt -y install build-essential git cmake pkg-config python3 python3-pip python3-venv curl wget
  

• zypper (openSUSE/SLE):

  sudo zypper refresh
  sudo zypper -n install -t pattern devel_basis
  sudo zypper -n install git cmake pkg-config python3 python3-pip python3-venv curl wget
  

Optional utilities you’ll likely want:

• dnf:

  sudo dnf -y install jq unzip tar
  

• apt:

  sudo apt -y install jq unzip tar
  

• zypper:

  sudo zypper -n install jq unzip tar
  

2) (Optional) Install GPU drivers and toolkits

• NVIDIA: Install the NVIDIA driver and CUDA following the official docs for your OS and kernel. After installation and reboot:

  nvidia-smi
  nvcc --version  # if CUDA toolkit installed
  

  Docs: https://docs.nvidia.com/cuda/

• AMD ROCm: For supported RHEL/Rocky releases, use AMD’s repository instructions, then verify:

  /opt/rocm/bin/rocminfo
  

  Docs: https://roCm.docs.amd.com/

Tip: Keep host driver versions matched to container/toolkit versions you plan to use.

3) Create a Python environment and install core AI libraries

Create a dedicated virtualenv to isolate AI dependencies.

python3 -m venv ~/.venvs/ai
source ~/.venvs/ai/bin/activate
python -m pip install --upgrade pip wheel setuptools

Install baseline libraries (CPU-friendly):

pip install "torch==2.3.1" --index-url https://download.pytorch.org/whl/cpu
pip install torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install "transformers>=4.41" "accelerate>=0.31" "huggingface-hub[cli]" sentencepiece safetensors
pip install onnxruntime

If you have NVIDIA CUDA 12.1 drivers/toolkit working:

pip install "torch==2.3.1" torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install onnxruntime-gpu

Note: For TensorFlow GPU on Linux, prefer the official install matrix; PyTorch and ONNX Runtime tend to be simpler to get running first.

---

Actionable example 1: Run a sentiment classifier (CPU or GPU)

1) Create a tiny inference script

cat > sentiment.py << 'PY'
from transformers import pipeline

clf = pipeline("sentiment-analysis",
               model="distilbert-base-uncased-finetuned-sst-2-english",
               device_map="auto")  # picks GPU if available, else CPU

texts = [
    "Rocky Linux is rock-solid for production.",
    "This deployment was painfully slow."
]

for t in texts:
    out = clf(t)[0]
    print(f"{t} -> {out['label']} (score={out['score']:.3f})")
PY

2) Run it

source ~/.venvs/ai/bin/activate
python sentiment.py

3) Bash one-liner for batch files

while IFS= read -r line; do
  python - <<'PY'
from transformers import pipeline
import sys
clf = pipeline("sentiment-analysis",
               model="distilbert-base-uncased-finetuned-sst-2-english",
               device_map="auto")
text = sys.stdin.read().strip()
res = clf(text)[0]
print(f"{text}\t{res['label']}\t{res['score']:.3f}")
PY
<<< "$line"
done < inputs.txt > outputs.tsv

---

Actionable example 2: Local LLM with llama.cpp (works great on Rocky)

llama.cpp is a fast C/C++ inference engine for GGUF models (LLaMA-family, Mistral, TinyLlama, etc.). You can use CPU or GPU (CUDA/ROCm/Metal).

1) Install build prerequisites

• dnf:

  sudo dnf -y install cmake git gcc gcc-c++ make
  

• apt:

  sudo apt -y install build-essential cmake git
  

• zypper:

  sudo zypper -n install gcc gcc-c++ make cmake git
  

2) Build llama.cpp

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
# CPU build:
make -j
# For NVIDIA GPU (if CUDA toolkit is present):
# make -j LLAMA_CUBLAS=1

3) Fetch a small quantized model (TinyLlama) with Hugging Face CLI

mkdir -p ~/models/tinyllama
huggingface-cli download --repo-type model \
  TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF \
  --include "*Q4_K_M.gguf" \
  --local-dir ~/models/tinyllama

4) Run a prompt

./main -m ~/models/tinyllama/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
  -p "In one sentence, explain why Rocky Linux is good for AI."

Tip: For larger models, add -t $(nproc) for threads and experiment with -ngl (GPU layers) if you built with CUDA.

---

Actionable example 3: A simple REST API for inference (FastAPI + systemd)

1) Create a minimal API

mkdir -p ~/apps/ai-svc
cat > ~/apps/ai-svc/app.py << 'PY'
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import pipeline

app = FastAPI(title="AI on Rocky Linux")
clf = pipeline("sentiment-analysis",
               model="distilbert-base-uncased-finetuned-sst-2-english",
               device_map="auto")

class Item(BaseModel):
    text: str

@app.post("/predict")
def predict(item: Item):
    res = clf(item.text)[0]
    return {"label": res["label"], "score": float(res["score"])}
PY

pip install fastapi uvicorn[standard]

2) Test locally

uvicorn app:app --reload --host 0.0.0.0 --port 8000
# In another shell:
curl -s http://127.0.0.1:8000/predict -X POST -H 'Content-Type: application/json' \
  -d '{"text":"Rocky Linux keeps my stack stable."}'

3) Create a systemd service (Rocky/openSUSE/Ubuntu all use systemd)

sudo tee /etc/systemd/system/ai-svc.service > /dev/null << 'UNIT'
[Unit]
Description=AI Inference Service (FastAPI)
After=network.target

[Service]
Type=simple
User=%i
WorkingDirectory=/home/%i/apps/ai-svc
Environment="PATH=/home/%i/.venvs/ai/bin"
ExecStart=/home/%i/.venvs/ai/bin/uvicorn app:app --host 0.0.0.0 --port 8000
Restart=on-failure

[Install]
WantedBy=multi-user.target
UNIT

Enable and start (replace $USER if running for a different user):

sudo systemctl daemon-reload
sudo systemctl enable ai-svc.service
sudo systemctl start ai-svc.service
sudo systemctl status ai-svc.service

4) Open the firewall (if enabled)

• firewalld (common on Rocky/openSUSE):

  sudo firewall-cmd --add-port=8000/tcp --permanent
  sudo firewall-cmd --reload
  

• ufw (common on Ubuntu):

  sudo ufw allow 8000/tcp
  

---

Optional: Ollama (quick LLMs with one command)

Ollama is a single-binary LLM runner with model management.

Install (works across apt/dnf/zypper systems):

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Pull and run a small model:

ollama pull tinyllama
ollama run tinyllama

Create a non-interactive prompt:

ollama run tinyllama "Give me three shell commands every Rocky Linux admin should know."

---

Practical tips for Rocky Linux AI

• Enable EPEL and CRB to get modern compilers and tools that AI projects expect.

• SELinux: Keep enforcing; label custom model directories if served by daemons.

  sudo semanage fcontext -a -t usr_t "/home/USER/models(/.*)?"
  sudo restorecon -Rv /home/USER/models
  

• Performance on CPU:

  export OMP_NUM_THREADS=$(nproc)
  export MKL_NUM_THREADS=$(nproc)
  

  Test with different thread counts to avoid oversubscription.

• Containers: Podman works great on Rocky.

  • dnf:

  sudo dnf -y install podman
  

  • apt:

  sudo apt -y install podman
  

  • zypper:

  sudo zypper -n install podman
  

  Use NVIDIA Container Toolkit (if using GPUs in containers) per vendor docs.

• Reproducibility: Pin versions in requirements.txt and export hardware/software info.

  pip freeze > requirements.txt
  python -c "import torch,platform;print(torch.__version__, platform.platform())"
  

---

Real-world example workflow (putting it together)

1) Prep the node with dnf steps and EPEL/CRB. 2) Create a Python venv, install Torch + Transformers. 3) Validate with the sentiment script. 4) Choose a path: - Need a small, fast LLM? Build llama.cpp and run TinyLlama in GGUF. - Need simplicity and model selection? Use Ollama. 5) Serve your model behind FastAPI, manage with systemd, open port 8000. 6) Benchmark and tune threads, batch sizes, and quantization as needed.

---

Conclusion and next steps (CTA)

You’ve seen that Rocky Linux can power practical, private AI—without cloud lock-in. From a simple text classifier to a local LLM and a production-style REST service, you now have a repeatable blueprint you can adapt to your org.

Next steps:

• Containerize your API with Podman and add CI/CD.

• Try larger GGUF models or quantized vision models via ONNX Runtime.

• Add observability (Prometheus metrics) and request logging.

• Lock down with SELinux policies, non-root services, and firewalld zones.

If you found this useful, try the examples on a fresh Rocky Linux 9 VM today. Share what you build—and what you want to see next!