Posted on
• Artificial Intelligence

Resource Optimisation for Local Artificial Intelligence Models

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Resource Optimisation for Local Artificial Intelligence Models

Make small machines feel big with a handful of Bash-friendly tweaks

Running LLMs or embeddings locally is equal parts empowering and frustrating. You download a model, hit run, and then… fans spin up, tokens trickle out, RAM spikes, and your terminal stutters. The good news: a few targeted system and runtime tweaks can double your tokens-per-second, slash memory usage, and make your machine feel purpose-built for AI.

This article explains why local inference feels “heavy,” then walks you through 3–5 high-impact optimisations you can apply today—with concrete commands and distro-specific installs for apt, dnf, and zypper.

Why optimisation matters (and why it works)

  • AI inference is memory- and bandwidth-bound. The model’s weights and Key/Value (KV) cache must be fed to compute units as efficiently as possible. Locality and parallelism matter.

  • General-purpose schedulers aren’t model-aware. Out-of-the-box, the OS may schedule hot threads across NUMA nodes or let background I/O contend with weight paging.

  • Model choice changes the game. A good quantisation can shrink memory 3–5× with minimal quality hit, which often translates into higher throughput on the same hardware.

These constraints mean small, deliberate changes—pinning threads, right-sizing context, enabling BLAS, selecting the right quantisation—can deliver outsized gains.

Quick-install: tools you’ll use (apt, dnf, zypper)

Install common build, math, and monitoring tools.

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y git build-essential cmake \
  libopenblas-dev numactl htop btop nvtop sysstat lm-sensors \
  cpufrequtils

  • Fedora/RHEL (dnf):

    sudo dnf install -y git gcc gcc-c++ make cmake \
  openblas-devel numactl htop btop nvtop sysstat lm_sensors \
  kernel-tools

  • openSUSE (zypper):

    sudo zypper refresh
sudo zypper install -y git gcc gcc-c++ make cmake \
  openblas-devel numactl htop btop nvtop sysstat sensors \
  cpupower

Notes:

  • cpufrequtils (apt), kernel-tools (dnf), and cpupower (zypper) give you CPU governor utilities.

  • OpenBLAS accelerates CPU math for many local-inference engines.

  • sysstat provides iostat/pidstat; lm-sensors/sensors help watch thermals.

Actionable optimisation levers

1) Start with the right model and quantisation

  • Pick a model size that fits in memory with headroom for the KV cache. For LLMs, the KV cache grows with context length; oversizing context can tank performance on low RAM.

  • Prefer GGUF/GGML quantised weights for CPU or light GPU offload. Example sizes for a 7B-class LLM:

    • FP16: ~13–14 GB
    • Q5_K_M: ~5–6 GB
    • Q4_K_M: ~4–5 GB
    • Q3_K_M: ~3–4 GB

  • Real-world: Moving from FP16 to Q4_K_M on a 16 GB RAM laptop often turns “swapping and stalling” into smooth, 10–20+ tok/s CPU inference.

Example: run a quantised 7B with llama.cpp (see build below):

./build/bin/llama-cli \
  -m ~/models/llama-7b-Q4_K_M.gguf \
  -p "Summarize the following text:" \
  -t 12 \
  -c 2048 \
  -b 64 \
  --no-mmap

Tips:

  • -t: threads (start with physical core count).

  • -c: context. Use only what you need; large contexts inflate KV cache memory.

  • -b: batch size (tokens processed in parallel). Tune for your CPU cache and RAM.

2) Use BLAS and build your runtime for your machine

Many local engines can link against BLAS to accelerate matrix multiplies.

Build llama.cpp with OpenBLAS:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Ensure OpenBLAS dev is installed (see install section above)
cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build -j

# Run a quick benchmark (no model needed)
./build/bin/llama-bench

Expect higher tokens/sec over the non-BLAS build on CPUs with decent vector units.

3) Pin threads and be NUMA-aware

Keep hot threads on the same CPU/NUMA node and avoid cross-node memory ping-pong. Pinning often yields a measurable, steady tokens/sec increase.

Find your physical core count:

PHYS=$(lscpu -p=core | grep -v '^#' | cut -d, -f1 | sort -u | wc -l)
echo "Physical cores: $PHYS"

Run with thread and NUMA pinning:

export OMP_NUM_THREADS=$PHYS
export OMP_PROC_BIND=close
export GOMP_CPU_AFFINITY="0-$((PHYS-1))"

# Bind to first NUMA node, local memory allocation
numactl --cpunodebind=0 --membind=0 \
  taskset -c 0-$((PHYS-1)) \
  ./build/bin/llama-cli -m ~/models/llama-7b-Q4_K_M.gguf -t $PHYS -c 2048 -b 64

If you have a dual-socket server, try running separate processes per socket, each pinned to its own NUMA node, instead of one giant process spanning both.

4) Tame memory: right-size context, leverage huge pages carefully, and keep swaps sane

  • Right-size context length (-c). Bigger contexts balloon the KV cache. If you don’t need 8k tokens, don’t pay for it.

  • Transparent Huge Pages (THP): madvise typically reduces TLB pressure without risking stalls from aggressive defrag.

    echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

    To persist across boots, add to a systemd tmpfiles.d rule or sysfs init script.

  • Provide some swap, but keep swappiness modest (10–20) to prefer RAM while remaining crash-resistant:

    # Create a 8G swapfile (adjust to your machine)
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo swapon -a

# Set swappiness to 10
echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-ai-tuning.conf
sudo sysctl --system

  • Keep models on fast storage (NVMe). If disk is slow and RAM tight, preload models to the page cache before runs:

    # Warm the page cache (harmless; just reads the file)
cat ~/models/llama-7b-Q4_K_M.gguf > /dev/null

5) Dial in the CPU governor and manage background noise

Background tasks and conservative governors reduce steady-state throughput.

  • Check your current governor:

    • apt (cpufrequtils):
    cpufreq-info | grep 'current policy'
sudo cpufreq-set -g performance
    • dnf (kernel-tools) and zypper (cpupower):
    sudo cpupower frequency-info | grep 'current policy'
sudo cpupower frequency-set -g performance

  • Lower I/O priority of background jobs while keeping inference responsive:

    ionice -c2 -n7 nice -n 10 tar -czf backups.tgz ~/bigfolder &

  • Monitor hot spots:

    htop            # per-thread CPU, pinning sanity check
btop            # CPU/mem/disk/pressure overview
nvtop           # if you offload layers to a GPU
pidstat -dur 1  # per-process CPU/mem/I/O every second
sensors         # temps and throttling clues

End-to-end example: build + run a quantised 7B on CPU

1) Install prerequisites (choose one set):

  • apt:

    sudo apt update
sudo apt install -y git build-essential cmake libopenblas-dev numactl

  • dnf:

    sudo dnf install -y git gcc gcc-c++ make cmake openblas-devel numactl

  • zypper:

    sudo zypper install -y git gcc gcc-c++ make cmake openblas-devel numactl

2) Build llama.cpp with OpenBLAS:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build -j

3) Download a GGUF-quantised 7B model (example placeholder; pick a reputable source and license-compatible model):

mkdir -p ~/models && cd ~/models
# Example: using huggingface-cli, or just download via browser/wget
# wget https://huggingface.co/.../llama-7b.Q4_K_M.gguf -O llama-7b-Q4_K_M.gguf

4) Run with sensible defaults and pinning:

cd ~/llama.cpp
PHYS=$(lscpu -p=core | grep -v '^#' | cut -d, -f1 | sort -u | wc -l)
export OMP_NUM_THREADS=$PHYS
export OMP_PROC_BIND=close
numactl --localalloc taskset -c 0-$((PHYS-1)) \
  ./build/bin/llama-cli \
  -m ~/models/llama-7b-Q4_K_M.gguf \
  -p "Write a 4-sentence summary of the performance profile of this machine." \
  -t $PHYS -c 2048 -b 64 --no-mmap

Expect steady, repeatable tokens/sec without wild oscillations.

Real-world snapshots

  • Laptop CPU-only (12 physical cores): Q4_K_M 7B at 12–20 tok/s after BLAS + pinning; prior was 6–10 tok/s unpinned.

  • Small server (dual-socket, NVMe): Two pinned processes, each bound to a socket, outperformed a single process by ~25% aggregate with fewer latency spikes.

  • Memory-constrained mini PC: Dropping context from 4096 to 2048 cut KV cache pressure, eliminated swap thrash, and improved throughput ~1.6×.

Your mileage will vary, but the pattern holds: quantise, use BLAS, pin threads, right-size memory, and stabilise the governor.

Conclusion and next steps

Better local AI isn’t magic—it’s resource discipline. If you:

  • choose quantised weights that fit memory,

  • compile with BLAS,

  • pin threads to cores/NUMA,

  • right-size context and tame swap,

  • and stabilise the CPU governor,

you’ll turn choppy inference into a smooth, predictable workflow.

Call to Action:

  • Benchmark before/after with the steps above and note your tokens/sec.

  • Build a tiny run script that sets OMP vars and pinning for each model.

  • Share your results and hardware in your team chat or issue tracker; small, reproducible wins accumulate quickly.

If you want a follow-up deep dive—GPU offload layers, ROCm/CUDA specifics, or multi-process serving on NUMA—let me know which hardware you run and I’ll tailor a guide.

Related posts