Running Artificial Intelligence Locally with Ollama: A Linux Bash Guide

If you’ve ever wished you could use powerful AI models without sending your data to the cloud, this post is for you. Running AI locally means lower latency, predictable costs, and data that never leaves your machine. Ollama makes this surprisingly easy on Linux—no sprawling MLOps stack required.

Below you’ll get:

• Why local AI is worth your time

• Step‑by‑step install instructions (apt, dnf, zypper and a one‑liner)

• 3–5 actionable examples you can copy/paste into your Bash shell

• Tips for managing models and tuning performance

• A simple call to action to keep going

Why run AI locally?

• Privacy and compliance: Keep sensitive logs, documents, and code on your own hardware.

• Cost and control: No per‑token fees or surprise bills; your hardware, your rules.

• Latency and reliability: Answers in milliseconds, even offline.

• Simplicity: Ollama wraps model download, quantization, and serving into one clean CLI + REST API.

Models like Llama 3, Mistral, and Phi 3 now run well on consumer CPUs and NVIDIA GPUs. The 7–8B parameter class fits on many dev laptops/desktops with 8–16GB RAM, plenty for summarization, Q&A, and coding help.

---

1) Install Ollama

Ollama provides a single binary and a systemd service that listens on localhost:11434. Choose the path that fits your distro.

Note:

• You’ll need curl and systemd.

• For GPUs, ensure NVIDIA drivers are installed. You can check with:

nvidia-smi

If this shows your GPU and driver versions, Ollama can typically use it automatically.

Option A: One‑liner (works on most Linux distros)

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
systemctl status ollama

Option B: Debian/Ubuntu (apt)

1) Download the latest .deb from Ollama (substitute the actual filename you download):

curl -LO https://ollama.com/download/ollama_<version>_linux_amd64.deb

2) Install via apt:

sudo apt update
sudo apt install ./ollama_<version>_linux_amd64.deb

3) Start and enable the service:

sudo systemctl enable --now ollama
systemctl status ollama

Option C: Fedora/RHEL/CentOS (dnf)

1) Download the latest .rpm (substitute the actual filename you download):

curl -LO https://ollama.com/download/ollama-<version>-1.x86_64.rpm

2) Install via dnf:

sudo dnf install -y ./ollama-<version>-1.x86_64.rpm

3) Start and enable the service:

sudo systemctl enable --now ollama
systemctl status ollama

Option D: openSUSE (zypper)

1) Download the latest .rpm (substitute the actual filename you download):

curl -LO https://ollama.com/download/ollama-<version>-1.x86_64.rpm

2) Install via zypper:

sudo zypper install ./ollama-<version>-1.x86_64.rpm

3) Start and enable the service:

sudo systemctl enable --now ollama
systemctl status ollama

Test the API is up:

curl http://localhost:11434/api/tags

You should get JSON (possibly an empty list if no models are pulled yet).

---

2) Pull and run your first model

Ollama hosts an open model library. Start with a compact general model like Llama 3 8B or Phi‑3 mini.

• Pull a model:

ollama pull llama3:8b

• Chat interactively:

ollama run llama3:8b

Type your prompt, press Enter. Ctrl+C to exit.

• One‑off prompt:

ollama run llama3:8b "Explain the difference between grep and ripgrep."

• Model housekeeping:

ollama list
ollama show llama3:8b
ollama rm <model-name>

Tip: Models are multiple GB; ensure you have disk space. If you have a smaller machine, try:

ollama pull phi3:mini

---

3) Use Ollama from Bash with the REST API

The local API streams responses at http://localhost:11434. Here’s a minimal example to generate text:

curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3:8b",
    "prompt": "Write a 2-sentence summary of Linux namespaces.",
    "stream": false
  }' | jq -r '.response'

Wrap that into a tiny shell function in your ~/.bashrc for quick prompts:

ai() {
  local prompt="$*"
  curl -s http://localhost:11434/api/generate \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"llama3:8b\",\"prompt\":\"$prompt\",\"stream\":false}" \
  | jq -r '.response'
}

Use it:

ai "Create a find command to list files larger than 100MB, excluding node_modules."

Process stdin (great for summarizing logs or docs):

ai_summarize() {
  local text
  text="$(cat)"
  curl -s http://localhost:11434/api/generate \
    -H "Content-Type: application/json" \
    -d "$(jq -nc --arg t "$text" '{
      model: "llama3:8b",
      prompt: ("Summarize concisely:\n\n" + $t),
      stream: false,
      options: { temperature: 0.2 }
    }')" | jq -r '.response'
}

journalctl -u sshd --since "1 hour ago" | ai_summarize

---

4) Real‑world Bash workflows

• Summarize a long README:

pandoc -t plain README.md | ai_summarize

• Extract tasks from a Markdown spec:

ai_tasks() {
  cat "$1" | ai_summarize | sed 's/^/- /'
}
ai_tasks SPEC.md

• Answer “how do I” questions for shell one‑liners:

ai 'Show me a one-liner to replace tabs with 4 spaces across all *.py files.'

• Ask models to explain error logs:

tail -n 200 /var/log/syslog | ai_summarize

• Non-interactive CLI run with a system prompt:

ollama run llama3:8b -p "You are a terse Linux assistant. Answer in one line: best way to diff two directories recursively?"

If your ollama version doesn’t support -p, use the REST API’s "system" field:

curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3:8b",
    "system": "You are a terse Linux assistant.",
    "prompt": "Best way to diff two directories recursively?",
    "stream": false
  }' | jq -r '.response'

---

5) Model management and performance tips

• Pick the right size:

  • Smaller (e.g., phi3:mini, llama3:8b) = faster, lower RAM, great for summaries and quick Q&A.
  • Larger (e.g., mistral:instruct, llama3:70b on a powerful workstation) = better reasoning, higher requirements.

• Inspect models and disk use:

ollama list
du -h ~/.ollama -d 1

• Tune generation (via API "options"):

curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3:8b",
    "prompt": "List three hardening steps for an SSH server.",
    "stream": false,
    "options": {
      "temperature": 0.2,
      "num_ctx": 4096
    }
  }' | jq -r '.response'

• Check GPU usage (if available):

watch -n1 nvidia-smi

If you see utilization spike during inference, Ollama is using your GPU.

• Run as a background service (already enabled in install):

sudo systemctl status ollama
sudo journalctl -u ollama -f

If you need to temporarily stop it:

sudo systemctl stop ollama

Security note: The service binds to localhost by default. If you expose it, put it behind a reverse proxy and add authentication.

---

Troubleshooting quick hits

• Port busy or API down:

ss -ltnp | grep 11434
systemctl status ollama
sudo journalctl -u ollama -n 200

• CUDA not detected: Confirm drivers and CUDA runtime with:

nvidia-smi

Then restart the service:

sudo systemctl restart ollama

• Out of memory or slow: Try a smaller model, reduce num_ctx, or close other RAM‑heavy apps.

---

Conclusion and next steps

You now have a local AI stack you can script from Bash, automate with cron/systemd, and keep entirely on‑prem. Next:

• Pull a couple of models and A/B test them on your real workloads.

• Add the ai() and ai_summarize functions to your shell profile.

• Wire Ollama into your tooling: log triage, doc summaries, CLI explainers, and code reviews.

Useful links to explore:

• curl http://localhost:11434/api/tags to list local models via API

• ollama list / ollama pull / ollama rm for model lifecycle

• Your distro’s GPU driver docs (for best acceleration)

Local AI isn’t just possible—it’s practical. Start small, iterate, and make your Linux box your own private LLM.