Posted on
• Artificial Intelligence

Artificial Intelligence Coding with Local Models

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Coding with Local Models: Supercharge Your Bash Workflow (Offline and Private)

If you’ve ever hesitated to paste code or logs into a cloud AI, you’re not alone. Secrets, compliance, and latency concerns make many teams wish they had an AI pair-programmer that runs locally. Good news: modern open models can run on your Linux box and slot neatly into your Bash workflow—private, scriptable, and fast.

This article shows you why local models are worth it, how to set them up on Linux, and how to wire them into practical coding tasks from the command line.

Why code with local AI models?

  • Privacy by default: Your code, logs, and credentials never leave your machine.

  • Predictable costs: No per-token cloud bills. Use your own CPU/GPU.

  • Latency and availability: Works offline; results come back quickly.

  • DevOps-friendly: HTTP APIs and CLIs are easy to script in Bash.

  • Good-enough accuracy: 7B–13B parameter models (quantized) are surprisingly capable for code generation, refactoring, and explanation tasks.

Step 1) Prepare your Linux environment

Install common build tools and utilities you’ll need. Choose your distro’s package manager.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl jq git build-essential cmake pkg-config
# Optional for faster CPU math:
sudo apt install -y libopenblas-dev
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y curl jq git gcc gcc-c++ make cmake pkgconfig
# Optional for faster CPU math:
sudo dnf install -y openblas-devel
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y curl jq git gcc gcc-c++ make cmake pkg-config
# Optional for faster CPU math:
sudo zypper install -y openblas-devel

Tip: If you’re CPU-only, set threads to your core count:

export OMP_NUM_THREADS=$(nproc)

Step 2) Pick a local runtime (two great options)

You only need one of these. Ollama is the fastest way to start; llama.cpp is a flexible, build-from-source alternative.

Option A: Ollama (quick start, HTTP API + CLI)

1) Install:

curl -fsSL https://ollama.com/install.sh | sh

2) Pull a coding-friendly model (choose one):

# General + code capable
ollama pull llama3.1:8b
# Code-focused (instruction-tuned)
ollama pull codellama:7b-instruct

3) Smoke test:

ollama run codellama:7b-instruct "Write a Bash function that extracts a tar.gz to a target dir."

4) Use the local HTTP API:

curl -s http://localhost:11434/api/chat -d '{
  "model": "codellama:7b-instruct",
  "messages": [{"role": "user", "content": "Generate a safe Bash script that creates a backup of /etc to ~/etc-backups with timestamps."}],
  "options": {"temperature": 0.2}
}' | jq -r '.message.content'

Notes:

  • Default host/port: 127.0.0.1:11434

  • Ollama runs as a system service after install.

Option B: llama.cpp (build your own, OpenAI-compatible server)

1) Build llama.cpp:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j

2) Obtain a GGUF model (quantized). Place it somewhere (example path shown):

mkdir -p ~/models
# Put your GGUF file here, e.g. ~/models/codellama-7b-instruct.Q4_K_M.gguf

3) Run the server:

./server -m ~/models/codellama-7b-instruct.Q4_K_M.gguf -c 4096 --host 127.0.0.1 --port 8080

4) Call it via OpenAI-style API:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role":"user","content":"Write a Bash script that cleans logs older than 14 days from /var/log/myapp safely."}],
    "temperature": 0.2
  }' | jq -r '.choices[0].message.content'

Step 3) Wire AI into your Bash toolbox

Create small, composable wrappers you can call from any repo or CI job.

1) Ask for help (general prompt)

ai() {
  local prompt="$*"
  curl -s http://localhost:11434/api/chat -d "$(jq -n \
    --arg m "codellama:7b-instruct" \
    --arg p "$prompt" \
    '{model:$m, messages:[{role:"user", content:$p}], options:{temperature:0.2}}')" \
  | jq -r '.message.content'
}
# Usage:
ai "Explain what this awk one-liner does: awk -F: '\''{print $1}'\'' /etc/passwd"

For llama.cpp server, swap the endpoint and payload:

ai() {
  local prompt="$*"
  curl -s http://127.0.0.1:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg p "$prompt" \
      '{model:"local", messages:[{role:"user", content:$p}], temperature:0.2}')" \
  | jq -r '.choices[0].message.content'
}

2) Review a file (lint/explain/refactor)

ai-review() {
  local file="$1"
  [[ -f "$file" ]] || { echo "File not found: $file" >&2; return 2; }
  local content
  content=$(sed 's/\\/\\\\/g;s/"/\\"/g' "$file")
  ai "Review this file for bugs, safety, and POSIX-compatibility. Provide actionable suggestions with examples:\n\n$content"
}
# Usage:
ai-review ./deploy.sh

3) Generate a patch (ask the model to propose a diff)

ai-patch() {
  local file="$1"
  [[ -f "$file" ]] || { echo "File not found: $file" >&2; return 2; }
  local content
  content=$(sed 's/\\/\\\\/g;s/"/\\"/g' "$file")
  ai "Propose a unified diff (patch) against the following file to improve robustness, add set -euo pipefail, and handle errors gracefully. Output only the diff:\n\n--- $file (original)\n+++ $file (proposed)\n\n$content"
}
# Usage:
ai-patch ./backup.sh

Tip: Always review and test generated changes before applying.

Step 4) Real-world coding examples

1) Generate a safe Bash script scaffold

ai "Create a robust Bash script template for a CLI tool. Include: set -euo pipefail, a usage() function, getopts parsing (-i input, -o output), a main() function, and meaningful error messages."

2) Explain a failing build log

make 2>&1 | tee build.log
ai "Explain the root cause of these build errors and suggest a minimal fix. Keep answers to bullet points:\n\n$(tail -n 200 build.log)"

3) Design a one-liner and verify it

ai "Write a portable grep/awk one-liner that prints lines where column 3 is an integer > 100 from a CSV with headers."

Then create a quick test dataset and validate:

cat > /tmp/data.csv <<'EOF'
name,age,score
alice,30,120
bob,28,90
carol,25,180
EOF

# Try the suggested command on /tmp/data.csv and manually spot-check output.

4) Refactor with constraints

ai "Refactor this Bash into POSIX sh, avoid arrays and bashisms, and keep comments intact:\n\n$(cat ./script.sh)"

Performance and safety tips

  • Pick the right size: 7B models are fast on CPUs; 13B+ improves reasoning but needs more RAM. Use quantized GGUF for llama.cpp.

  • Control verbosity: For coding, temperature 0.0–0.3 and top_p/top_k defaults usually work well.

  • Constrain outputs: Ask for “only the diff” or “only code in a fenced block” to simplify parsing.

  • Validate everything: Never run AI-generated shell blindly. Test in a temp dir or container.

  • Cache prompts: Save useful prompts as scripts to keep workflows consistent.

  • Hardware acceleration: If you have a supported GPU, consult your runtime’s docs to enable CUDA/ROCm/OpenCL builds for big speedups.

Conclusion and next step (CTA)

Local AI unlocks private, scriptable coding assistance right inside your terminal. Pick a runtime (start with Ollama for speed), pull a coding-tuned model, and drop the ai(), ai-review(), and ai-patch() helpers into your shell profile. Then iterate: tune prompts, validate outputs, and let the model handle boilerplate while you focus on the hard problems.

Your move: 1) Install Ollama and pull a model:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull codellama:7b-instruct

2) Paste the ai() function into your shell and try your first prompt. 3) Turn your most common coding tasks into repeatable Bash scripts powered by your local model.

Happy hacking—offline and in control.

Related posts