Posted on
• Artificial Intelligence

Artificial Intelligence Edge Computing on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence at the Edge on Linux: A Practical, Bash-First Guide

The smartest place for your AI might be nowhere near the cloud. When milliseconds matter, connectivity is spotty, or privacy is paramount, running AI at the edge—right on your Linux device—wins on latency, reliability, cost, and control. This article shows you how to stand up a lean, reproducible edge AI pipeline on Linux using Bash-friendly steps and common package managers, with actionable examples you can copy-paste today.

What you’ll get:

  • Why edge AI on Linux is a high-value move right now.

  • A minimal-but-powerful toolchain you can install with apt, dnf, and zypper.

  • A working CPU-based inference pipeline (with optional GPU/NPU paths).

  • Optimization and deployment steps (systemd service, MQTT telemetry, container option).

  • Real-world examples you can model your project on.

Why Edge AI on Linux is Worth Your Time

  • Latency and bandwidth: Sensory data is large; shipping it to the cloud burns time and money. Infer near the sensor and only send what’s meaningful.

  • Privacy and compliance: Keeping raw data on-prem reduces exposure and simplifies compliance.

  • Reliability: Edge works even during network outages.

  • Cost and control: Linux gives you driver flexibility, performance tuning (cgroups, systemd), and predictable bill-of-materials—even on low-power devices.

Linux is the de facto standard for edge devices (x86, ARM), and its package ecosystem makes repeatable deployment straightforward.

What We’ll Build

  • A portable Python-based inference pipeline using ONNX Runtime (CPU by default).

  • Optional acceleration paths:

    • Intel iGPU/NPU via OpenVINO.
    • NVIDIA GPU via CUDA/TensorRT + ONNX Runtime (advanced).

  • A clean service deployment (systemd) and telemetry with MQTT.

  • Containerized alternative via Docker/Podman.

1) Prepare Your Linux System

Install baseline packages. Choose your package manager.

A) Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y \
  python3 python3-venv python3-pip git curl wget \
  build-essential cmake pkg-config \
  ffmpeg v4l-utils \
  mosquitto mosquitto-clients \
  docker.io podman
# Optional: add your user to docker group (log out/in after)
sudo usermod -aG docker "$USER"

B) Fedora/RHEL/CentOS Stream (dnf):

sudo dnf -y update
sudo dnf -y install \
  python3 python3-virtualenv python3-pip git curl wget \
  gcc gcc-c++ make cmake pkgconf-pkg-config \
  ffmpeg v4l-utils \
  mosquitto mosquitto-clients \
  moby-engine podman
# Enable and start Docker if using moby-engine
sudo systemctl enable --now docker
# Optional: add your user to docker group (log out/in after)
sudo usermod -aG docker "$USER"

C) openSUSE/SUSE (zypper):

sudo zypper refresh
sudo zypper install -y \
  python3 python3-virtualenv python3-pip git curl wget \
  gcc gcc-c++ make cmake pkgconf-pkg-config \
  ffmpeg v4l-utils \
  mosquitto mosquitto-clients \
  docker podman
# Enable and start Docker
sudo systemctl enable --now docker
# Optional: add your user to docker group (log out/in after)
sudo usermod -aG docker "$USER"

Notes:

  • Docker and Podman are both provided so you can pick your preference.

  • If you’re on a minimal system, ensure your user has permission to access cameras: add to the video group if needed (sudo usermod -aG video "$USER").

2) Create a Python Environment and Install AI Runtimes

We’ll use a virtual environment to keep things clean.

mkdir -p ~/edge-ai && cd ~/edge-ai
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools

Install common packages:

pip install numpy onnx onnxruntime opencv-python paho-mqtt

Optional accelerators (install only what you need):

  • Intel iGPU/NPU (OpenVINO):
pip install openvino-dev
  • NVIDIA GPU (advanced): Requires recent NVIDIA driver + CUDA. Then:
pip install onnxruntime-gpu

Tip: Verify GPU visibility with nvidia-smi and match the CUDA version expected by the onnxruntime-gpu wheel.

3) Run Your First Inference (CPU by default)

We’ll download a small ONNX classification model (SqueezeNet) and run it on a sample image. You can swap in a live camera feed later.

Get a model and a test image:

mkdir -p models data
cd models
wget -O squeezenet1.1-7.onnx https://github.com/onnx/models/raw/main/vision/classification/squeezenet/model/squeezenet1.1-7.onnx
cd ../data
wget -O cat.jpg https://raw.githubusercontent.com/opencv/opencv/master/samples/data/baboon.jpg
cd ..

Minimal inference script (CPU with ONNX Runtime):

cat > infer.py << 'EOF'
import cv2, numpy as np, onnxruntime as ort, sys
img_path = sys.argv[1] if len(sys.argv) > 1 else "data/cat.jpg"
model_path = "models/squeezenet1.1-7.onnx"

# Preprocess to 224x224, BGR->RGB, normalize (ImageNet)
img = cv2.imread(img_path)
if img is None:
    raise SystemExit(f"Could not read {img_path}")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = cv2.resize(img, (224, 224), interpolation=cv2.INTER_AREA)
img = img.astype(np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std  = np.array([0.229, 0.224, 0.225], dtype=np.float32)
img = (img - mean) / std
img = np.transpose(img, (2,0,1))[None, :]  # NCHW

sess = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
input_name = sess.get_inputs()[0].name
output_name = sess.get_outputs()[0].name

prob = sess.run([output_name], {input_name: img})[0][0]
top5 = prob.argsort()[-5:][::-1]
print("Top-5 class indices:", top5.tolist())
EOF

python infer.py

Optional: Use OpenVINO for Intel acceleration (same model):

cat > infer_openvino.py << 'EOF'
import cv2, numpy as np, sys
from openvino.runtime import Core

img_path = sys.argv[1] if len(sys.argv) > 1 else "data/cat.jpg"
model_path = "models/squeezenet1.1-7.onnx"

img = cv2.imread(img_path)
if img is None:
    raise SystemExit(f"Could not read {img_path}")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = cv2.resize(img, (224, 224), interpolation=cv2.INTER_AREA)
img = img.astype(np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std  = np.array([0.229, 0.224, 0.225], dtype=np.float32)
img = (img - mean) / std
img = np.transpose(img, (2,0,1))[None, :]

ie = Core()
model = ie.read_model(model=model_path)
compiled = ie.compile_model(model=model, device_name="AUTO")  # CPU/GPU
infer_request = compiled.create_infer_request()
res = infer_request.infer({"data": img})  # Some models use different input names
out = list(res.values())[0][0]
print("Top-5 class indices:", out.argsort()[-5:][::-1].tolist())
EOF

python infer_openvino.py

Camera input (V4L2) swap-in is easy: replace image loading with cv2.VideoCapture(0) and grab frames in a loop.

4) Optimize for the Edge (Quantization + Profiling)

Start with dynamic quantization (fast, low risk) to reduce model size and improve latency on CPU:

cat > quantize.py << 'EOF'
from onnxruntime.quantization import quantize_dynamic, QuantType
import os
in_model = "models/squeezenet1.1-7.onnx"
out_model = "models/squeezenet1.1-7-int8.onnx"
quantize_dynamic(in_model, out_model, weight_type=QuantType.QInt8)
print(f"Quantized model written to {out_model} ({os.path.getsize(out_model)} bytes)")
EOF

python quantize.py

Test latency differences:

cat > bench.py << 'EOF'
import time, numpy as np, cv2, onnxruntime as ort, sys

def load_img(p):
    img = cv2.imread(p); img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    img = cv2.resize(img, (224,224), interpolation=cv2.INTER_AREA)
    img = img.astype(np.float32) / 255.0
    mean = np.array([0.485, 0.456, 0.406], np.float32)
    std  = np.array([0.229, 0.224, 0.225], np.float32)
    img = (img - mean) / std
    return np.transpose(img, (2,0,1))[None,:]

img = load_img("data/cat.jpg")

def run(model):
    sess = ort.InferenceSession(model, providers=["CPUExecutionProvider"])
    x = sess.get_inputs()[0].name; y = sess.get_outputs()[0].name
    # Warmup
    for _ in range(5): sess.run([y], {x: img})
    t0 = time.perf_counter()
    for _ in range(50): sess.run([y], {x: img})
    t1 = time.perf_counter()
    return (t1 - t0) / 50

fp = bench_fp = run("models/squeezenet1.1-7.onnx")
int8 = bench_int8 = run("models/squeezenet1.1-7-int8.onnx")
print(f"Avg latency FP32: {fp*1000:.2f} ms | INT8: {int8*1000:.2f} ms")
EOF

python bench.py

For deeper gains:

  • Batch small models where appropriate.

  • Use provider-specific accelerators (OpenVINO on Intel, TensorRT on NVIDIA).

  • Pin CPU governor to performance for consistent timing: sudo cpupower frequency-set -g performance.

5) Deploy as a Service with Telemetry (systemd + MQTT)

Install and run Mosquitto if you haven’t already.

A) Debian/Ubuntu (apt):

sudo systemctl enable --now mosquitto

B) Fedora/RHEL/CentOS Stream (dnf):

sudo systemctl enable --now mosquitto

C) openSUSE/SUSE (zypper):

sudo systemctl enable --now mosquitto

Create a simple inference+MQTT script:

cat > edge_infer_mqtt.py << 'EOF'
import time, json, cv2, numpy as np, onnxruntime as ort, paho.mqtt.client as mqtt

cam = cv2.VideoCapture(0)  # or use a file path
sess = ort.InferenceSession("models/squeezenet1.1-7-int8.onnx", providers=["CPUExecutionProvider"])
x = sess.get_inputs()[0].name
y = sess.get_outputs()[0].name

client = mqtt.Client(client_id="edge-ai-1")
client.connect("localhost", 1883, 60)
client.loop_start()

def preprocess(frame):
    img = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    img = cv2.resize(img, (224,224), interpolation=cv2.INTER_AREA)
    img = img.astype(np.float32)/255.0
    mean = np.array([0.485,0.456,0.406], np.float32)
    std  = np.array([0.229,0.224,0.225], np.float32)
    img = (img-mean)/std
    return np.transpose(img, (2,0,1))[None,:]

while True:
    ok, frame = cam.read()
    if not ok:
        time.sleep(0.5)
        continue
    inp = preprocess(frame)
    t0 = time.perf_counter()
    out = sess.run([y], {x: inp})[0][0]
    dt = (time.perf_counter() - t0)*1000.0
    payload = {"ts": time.time(), "top1": int(out.argmax()), "lat_ms": dt}
    client.publish("edge/vision/classifier", json.dumps(payload), qos=0, retain=False)
    # Throttle if needed
    time.sleep(0.1)
EOF

Create a systemd service:

cat > ~/.config/systemd/user/edge-ai.service << 'EOF'
[Unit]
Description=Edge AI Inference Service
After=network-online.target

[Service]
Type=simple
WorkingDirectory=%h/edge-ai
Environment=PYTHONUNBUFFERED=1
ExecStart=%h/edge-ai/.venv/bin/python %h/edge-ai/edge_infer_mqtt.py
Restart=on-failure

[Install]
WantedBy=default.target
EOF

systemctl --user daemon-reload
systemctl --user enable --now edge-ai.service
journalctl --user -u edge-ai.service -f

Test MQTT messages:

mosquitto_sub -t 'edge/vision/classifier' -v

Containerized Alternative (Docker/Podman)

Dockerfile (CPU, ONNX Runtime):

cat > Dockerfile << 'EOF'
FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
    ffmpeg v4l-utils && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY models/ models/
COPY edge_infer_mqtt.py .
RUN pip install --no-cache-dir numpy onnx onnxruntime opencv-python paho-mqtt
ENV PYTHONUNBUFFERED=1
CMD ["python", "edge_infer_mqtt.py"]
EOF

Build and run with camera access:

  • Docker:
docker build -t edge-ai:cpu .
# Expose /dev/video0 and host network for MQTT to localhost
docker run --rm --device /dev/video0 --network host edge-ai:cpu
  • Podman:
podman build -t edge-ai:cpu .
podman run --rm --device /dev/video0 --network host edge-ai:cpu

Note: For NVIDIA GPU, use the NVIDIA Container Toolkit and the onnxruntime-gpu wheel in your Dockerfile.

Real-World Examples You Can Emulate

  • Smart camera at retail entrance (CPU only): Count entries, detect line buildup, publish counts over MQTT to trigger staff alerts. Privacy preserved—no raw frames leave the device.

  • Industrial fan monitoring (Intel iGPU with OpenVINO): Classify abnormal vibrations or spectrograms from microphones; send only anomalies to a central dashboard.

  • Wildlife camera trap (battery/solar, ARM Linux): Run a quantized model to wake a radio or take high-res images only when specific species are detected.

Troubleshooting Essentials

  • Camera not found? Check device:
v4l2-ctl --list-devices
  • Performance swings? Pin CPU governor:
sudo cpupower frequency-set -g performance
  • MQTT not receiving? Verify broker:
systemctl status mosquitto
mosquitto_sub -t '#' -v
  • GPU path: Ensure nvidia-smi works and CUDA/toolkit versions match your runtime.

Conclusion and Next Steps (CTA)

You now have a reproducible, Linux-native edge AI stack: install, infer, optimize, and deploy as a service—optionally in a container. Next:

  • Swap the demo model for your own (convert to ONNX), and try dynamic quantization immediately.

  • Choose your accelerator: stick with CPU for simplicity, add OpenVINO for Intel hardware, or step up to NVIDIA GPU.

  • Wire events to your existing message bus or dashboard via MQTT, and scale out with containers.

If you found this useful, try adapting the service to your camera feed, benchmark INT8 vs FP32 on your device, and share your latency gains. Edge AI on Linux is fast, private, and yours to control—start shipping it today.

Related posts