Posted on
• Artificial Intelligence

Artificial Intelligence Without Internet: Practical Linux Examples

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Without Internet: Practical Linux Examples

Ever wished you could run powerful AI workloads on a laptop in the field, on a server in an air‑gapped lab, or on a homelab that never phones home? You can. AI doesn’t have to live in the cloud. With Linux, a few small tools, and some planning, you can do language modeling, speech recognition, OCR, and image classification fully offline.

This guide shows you why offline AI is worth it, how to prepare, and 4 practical Linux examples you can run without an internet connection. All commands are shell-friendly, and installation steps are provided for apt, dnf, and zypper.

Why run AI offline on Linux?

  • Privacy and compliance: Keep data on-prem, satisfy regulatory or contractual constraints, and eliminate third-party processing of sensitive content.

  • Uptime and latency: No network round-trips, no service outages. Results arrive predictably and fast.

  • Cost control: Avoid metered API calls. Once models are downloaded, you can run them as much as you like.

  • Reproducibility: Pin your models and code. Same inputs, same outputs, every time.

  • Edge and air-gapped environments: Field sites, labs, factories, or ships—no internet required.

Constraints to consider:

  • Compute and memory: Choose model sizes your CPU/GPU and RAM can handle. Quantized models help.

  • Storage: Models are large (hundreds of MB to several GB). Plan for disk space and backups.

  • Licensing: Verify model and data licenses before redistribution.

What you’ll build

Four fully offline workflows you can prep on an online machine, then run on an offline Linux box: 1) Local language model with llama.cpp (LLM chat/Q&A) 2) Speech-to-text with Vosk 3) OCR with Tesseract 4) Image classification with ONNX Runtime

You’ll also see a small “offline-first” checklist that saves you time and bandwidth.

Before you go offline: a simple pattern

  • Stage on a connected machine:

    • Download models, wheels, and assets.
    • Save checksums and verify integrity.
    • Copy to USB or your secure transfer method.

  • Keep things tidy:

    • Use a dedicated workspace (for example, ~/ai-offline).
    • Store model versions with clear names.

Create the workspace:

mkdir -p ~/ai-offline/{models,wheels,projects}

Optional: build a local Python wheelhouse for offline installs:

python3 -m venv ~/ai-offline/venv
. ~/ai-offline/venv/bin/activate
pip install --upgrade pip
pip download -d ~/ai-offline/wheels vosk onnxruntime numpy pillow

Later, offline:

. ~/ai-offline/venv/bin/activate
pip install --no-index --find-links ~/ai-offline/wheels vosk onnxruntime numpy pillow

If you need system packages offline, prefetch them on a similar distro:

  • apt:

    apt-get download <pkg>

  • dnf:

    dnf download <pkg>

  • zypper:

    zypper download <pkg>

1) Run a local language model with llama.cpp

Great for: shell helpers, local Q&A, drafting, code assistance—no network required.

Install build dependencies (choose your distro):

  • apt (Debian/Ubuntu):

    sudo apt update
sudo apt install -y git build-essential cmake libopenblas-dev

  • dnf (Fedora/RHEL/CentOS Stream):

    sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git cmake openblas-devel

  • zypper (openSUSE):

    sudo zypper install -y git gcc-c++ make cmake openblas-devel

Build llama.cpp with BLAS for better CPU performance:

cd ~/ai-offline/projects
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
LLAMA_BLAS=1 make -j"$(nproc)"

Get an offline model:

  • On a connected machine, download a quantized GGUF model compatible with llama.cpp (e.g., a 7B “instruct” model). Save its checksum.

    # Example placeholders; use a trusted source and actual filenames/URLs.
wget -O mistral-7b-instruct.Q4_K_M.gguf <URL_TO_GGUF_MODEL>
sha256sum mistral-7b-instruct.Q4_K_M.gguf > SHA256SUMS

  • Transfer the .gguf file (and SHA256SUMS) to ~/ai-offline/models on the offline box, then verify:

    cd ~/ai-offline/models
sha256sum -c SHA256SUMS

Run the model:

cd ~/ai-offline/projects/llama.cpp
./llama-cli -m ~/ai-offline/models/mistral-7b-instruct.Q4_K_M.gguf -p "In 3 bullet points, explain how to find large files in Linux."
# If your build produced 'main' instead of 'llama-cli':
# ./main -m ~/ai-offline/models/mistral-7b-instruct.Q4_K_M.gguf -p "..."

Tips:

  • Start with 4–7B parameter models on CPU. 8–16 GB RAM recommended for comfort; quantized models reduce memory.

  • For NVIDIA GPU acceleration, compile with CUDA flags per llama.cpp docs on an online machine, then transfer the build artifacts.

2) Offline speech-to-text with Vosk

Vosk provides accurate offline ASR (Automatic Speech Recognition) with small-footprint models.

Install audio tools and Python:

  • apt:

    sudo apt update
sudo apt install -y python3 python3-venv python3-pip sox alsa-utils

  • dnf:

    sudo dnf install -y python3 python3-virtualenv python3-pip sox alsa-utils

  • zypper:

    sudo zypper install -y python3 python3-venv python3-pip sox alsa-utils

Create a venv and install Vosk (use your offline wheelhouse if you built one):

python3 -m venv ~/ai-offline/venv
. ~/ai-offline/venv/bin/activate
pip install --upgrade pip
pip install vosk

Get an offline Vosk model:

  • On a connected machine, download a small English model (or your language), for example “vosk-model-small-en-us-0.15”. Verify checksum and transfer to ~/ai-offline/models.

  • Extract it to: ~/ai-offline/models/vosk-model-small-en-us-0.15

Record audio (16 kHz mono is ideal):

# 5-second 16kHz mono WAV via sox
rec -r 16000 -b 16 -c 1 ~/ai-offline/hello.wav trim 0 5
# Or via ALSA:
arecord -f S16_LE -r 16000 -c 1 -d 5 ~/ai-offline/hello.wav

Transcribe with Python:

cat > ~/ai-offline/projects/vosk_transcribe.py << 'PY'
import wave, json, sys
from vosk import Model, KaldiRecognizer

audio_path = sys.argv[1] if len(sys.argv) > 1 else "hello.wav"
model_dir = sys.argv[2] if len(sys.argv) > 2 else "../models/vosk-model-small-en-us-0.15"

wf = wave.open(audio_path, "rb")
assert wf.getnchannels() == 1 and wf.getsampwidth() == 2 and wf.getframerate() == 16000, \
    "Use 16kHz mono 16-bit PCM WAV"
model = Model(model_dir)
rec = KaldiRecognizer(model, 16000)

text = []
while True:
    data = wf.readframes(4000)
    if len(data) == 0:
        break
    if rec.AcceptWaveform(data):
        j = json.loads(rec.Result())
        text.append(j.get("text",""))
j = json.loads(rec.FinalResult())
text.append(j.get("text",""))
print(" ".join([t for t in text if t]))
PY

cd ~/ai-offline/projects
. ../venv/bin/activate
python vosk_transcribe.py ../hello.wav ../models/vosk-model-small-en-us-0.15

3) OCR entirely offline with Tesseract

Turn scans and images into searchable text.

Install Tesseract and ImageMagick:

  • apt:

    sudo apt update
sudo apt install -y tesseract-ocr tesseract-ocr-eng imagemagick

  • dnf:

    sudo dnf install -y tesseract tesseract-langpack-eng ImageMagick

  • zypper:

    sudo zypper install -y tesseract tesseract-ocr-traineddata-eng ImageMagick

Preprocess for better accuracy (deskew, grayscale, sharpen):

convert scan.jpg -deskew 40% -resize 200% -colorspace Gray -density 300 -units PixelsPerInch -sharpen 0x1 prepped.png

Run OCR:

tesseract prepped.png stdout -l eng --oem 1 --psm 6

Notes:

  • For other languages, install the right traineddata package (eng=English, deu=German, spa=Spanish, etc.). On apt: tesseract-ocr-<lang>, on dnf: tesseract-langpack-<lang>, on zypper: tesseract-ocr-traineddata-<lang>.

  • For multi-page PDFs, combine with pdftoppm or convert and then OCR in a loop.

4) Offline image classification with ONNX Runtime

Run a pre-trained ImageNet model (e.g., MobileNetV2) with zero internet.

Install Python dependencies (use offline wheelhouse if available):

python3 -m venv ~/ai-offline/venv
. ~/ai-offline/venv/bin/activate
pip install --upgrade pip
pip install onnxruntime numpy pillow

Get offline assets on a connected machine:

  • A MobileNetV2 ONNX model (e.g., mobilenetv2-7.onnx).

  • ImageNet class labels (e.g., imagenet-labels.txt).

  • Transfer to: ~/ai-offline/models/ and verify checksums.

Run inference:

cat > ~/ai-offline/projects/onnx_classify.py << 'PY'
import onnxruntime as rt
import numpy as np
from PIL import Image

model_path = "../models/mobilenetv2-7.onnx"
labels_path = "../models/imagenet-labels.txt"
image_path = "../models/example.jpg"  # replace with your image

sess = rt.InferenceSession(model_path, providers=["CPUExecutionProvider"])
input_name = sess.get_inputs()[0].name
output_name = sess.get_outputs()[0].name

img = Image.open(image_path).convert("RGB").resize((224, 224))
x = np.array(img).astype("float32") / 255.0
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std  = np.array([0.229, 0.224, 0.225], dtype=np.float32)
x = (x - mean) / std
x = np.transpose(x, (2, 0, 1))  # HWC -> CHW
x = np.expand_dims(x, 0)        # NCHW

probs = sess.run([output_name], {input_name: x})[0][0]
top5_idx = probs.argsort()[-5:][::-1]

labels = [l.strip() for l in open(labels_path, "r", encoding="utf-8").readlines()]
for i in top5_idx:
    print(f"{labels[i]}: {probs[i]:.4f}")
PY

cd ~/ai-offline/projects
. ../venv/bin/activate
python onnx_classify.py

Tips:

  • Many ONNX models expect specific preprocessing; confirm the right resize, normalization, and layout.

  • ONNX Runtime supports CPU out of the box. For GPU, prepare the appropriate provider builds before going offline.

Performance and reliability tips

  • Prefer quantized models (e.g., GGUF Q4/Q5 for LLMs) to cut RAM and compute.

  • Use BLAS (OpenBLAS) for CPU math. Already covered with LLAMA_BLAS=1.

  • Keep swap enabled if RAM is tight, but expect slower performance.

  • Version-lock your models and scripts. Save SHA256 checksums.

  • Test with the network disabled to prove you’re fully offline:

    sudo systemctl stop NetworkManager
# or temporarily disable the interface
# nmcli radio all off

Your offline-first checklist

  • Pick one workflow above and run it end-to-end.

  • Build a “kit”:

    • Models (.gguf, Vosk, ONNX) + checksums
    • Python wheelhouse (pip download)
    • Bash/Python scripts in a git repo

  • Document your steps in a README so teammates can reproduce them.

  • Iterate: measure speed and memory; try smaller or quantized models; add GPU acceleration if available.

Conclusion and next steps

AI without the internet is not just possible—it’s powerful. You’ve seen four practical examples that respect privacy, work anywhere, and cost nothing per call. Your next step:

  • Choose one example and implement it today.

  • Package your assets into a reusable offline kit.

  • Extend it: add a local RAG workflow (document embeddings + a local LLM), add multi-language OCR, or automate audio transcription pipelines.

When you’re ready, share your scripts and results with your team—or contribute improvements back to the community. Your Linux box just became an AI edge device.

Related posts