Posted on
• Artificial Intelligence

Running Private Artificial Intelligence with Ollama

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Running Private Artificial Intelligence on Linux with Ollama

If you’ve ever hesitated to paste confidential logs, code, or customer data into a cloud AI chatbot, you’re not alone. Latency, cost, and compliance headaches only make it worse. The good news: you can run capable large language models entirely on your own Linux machine with Ollama—no account, no internet, and no data leaves your box.

This guide shows you how to install Ollama on Debian/Ubuntu, Fedora, and openSUSE; pull a model; use it from Bash; call its local HTTP API; and customize a model for your workflow. You’ll have a private AI endpoint in minutes.

Why local AI is worth your time

  • Privacy and compliance by default: your prompts and documents never leave your machine.

  • Predictable cost: no per-token billing, no surprise invoices.

  • Low latency and offline: answers even without the internet.

  • Portable and scriptable: one local endpoint you can pipe, cron, and integrate into your Bash tools.

What you’ll build

  • A local Ollama service running on port 11434

  • A first model (for example llama3) you can chat with in the terminal

  • Bash one-liners to summarize logs and draft text

  • Curl examples for the Ollama HTTP API

  • A custom model persona using a simple Modelfile

1) Install Ollama (Debian/Ubuntu, Fedora, openSUSE)

Ollama provides a single-binary runtime and a systemd service. First, install prerequisites with your distro’s package manager, then run the official installer.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl ca-certificates jq
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Fedora (dnf):

sudo dnf install -y curl ca-certificates jq
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl ca-certificates jq
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Check status and logs:

systemctl status ollama
journalctl -u ollama -f

Notes:

  • This installs and starts ollama serve on localhost:11434.

  • Models are stored locally. Defaults are either /usr/share/ollama (system) or ~/.ollama/models (user). Override with OLLAMA_MODELS=/path.

  • NVIDIA GPUs are supported when drivers/CUDA are present; otherwise Ollama runs on CPU. AMD/Intel GPU support is evolving—check Ollama’s docs for current status.

Alternative: run on demand from your shell without systemd:

ollama serve

2) Pull your first model and chat locally

List available models at https://ollama.com/library. Pull and run one (example: llama3):

ollama pull llama3
ollama run llama3

You’ll get an interactive REPL. Type a prompt and press Enter. Exit with Ctrl+C.

Non-interactive prompt:

ollama run llama3 -p "Write a 3-bullet summary of the benefits of local AI."

See what’s installed:

ollama list

Remove a model:

ollama rm llama3

3) Use Ollama from Bash like any other CLI

Because Ollama speaks stdin/stdout, it fits right into pipelines.

Summarize your recent system logs:

journalctl -n 200 | ollama run llama3 -p "Summarize the key events in these logs:"

Draft a commit message from a diff:

git diff | ollama run llama3 -p "Write a concise, imperative Git commit message for these changes:"

Turn a README outline into prose:

cat OUTLINE.md | ollama run llama3 -p "Expand this outline into a clear README in Markdown:"

Tip: Keep inputs reasonable; LLMs have context limits. For big files, chunk them or ask targeted questions.

4) Call the local HTTP API with curl

The server exposes a simple JSON API on localhost:11434. This makes scripting and integrations easy.

Basic generation:

curl -s http://localhost:11434/api/generate \
  -d '{"model":"llama3","prompt":"Give me three Bash tips."}' | jq

Streamed output (read as it’s generated):

curl -sN http://localhost:11434/api/generate \
  -d '{"model":"llama3","prompt":"Explain pipes and redirection in Bash, briefly.","stream":true}'

Embeddings (for search/RAG workflows):

curl -s http://localhost:11434/api/embeddings \
  -d '{"model":"nomic-embed-text","prompt":"linux process management"}' | jq

Expose on your LAN (optional and only on trusted networks):

export OLLAMA_HOST=0.0.0.0:11434
ollama serve

Security reminder: Ollama does not include built-in auth. If you expose it beyond localhost, put it behind a reverse proxy (nginx, Caddy, Traefik) with TLS and authentication.

Open the port if your firewall is enabled.

Debian/Ubuntu with UFW:

sudo ufw allow 11434/tcp

Fedora/openSUSE with firewalld:

sudo firewall-cmd --add-port=11434/tcp --permanent
sudo firewall-cmd --reload

5) Customize a model with a Modelfile

You can package a system prompt, parameters, and assets into a re-usable local model.

Create Modelfile:

cat > Modelfile <<'EOF'
FROM llama3
SYSTEM "You are a precise Linux and Bash assistant. Prefer POSIX-compatible solutions and explain briefly."
PARAMETER temperature 0.2
EOF

Build and run:

ollama create bash-assistant -f Modelfile
ollama run bash-assistant -p "Show a portable way to find the 10 largest files in /var/log."

This lets you standardize tone, defaults, and constraints for your team.

Operational tips

  • Verify GPU use: if you have an NVIDIA GPU and drivers installed, ollama run will typically report GPU usage in its startup logs. You can also check nvidia-smi during a generation.

  • Control storage: set OLLAMA_MODELS=/path/to/disk before starting the service:

    sudo systemctl stop ollama
sudo mkdir -p /srv/ollama-models && sudo chown $USER:$USER /srv/ollama-models
export OLLAMA_MODELS=/srv/ollama-models
ollama serve

    To make it permanent for systemd, create an override:

    sudo systemctl edit ollama

    Then add:

    [Service]
Environment=OLLAMA_MODELS=/srv/ollama-models

    And restart:

    sudo systemctl daemon-reload
sudo systemctl restart ollama

  • Keep models tidy: ollama list to view, ollama rm <name> to free disk space.

  • Stay current: re-run the installer to upgrade Ollama, then restart the service.

Real-world examples

  • Private log triage: pipe journalctl or app logs to a local model for quick summaries and remediation steps.

  • Secure coding assistant: ask for Bash one-liners or sed/awk refactors entirely offline.

  • Documentation drafting: turn outlines or CLI --help text into readable docs.

  • Local RAG: generate embeddings with /api/embeddings, index them, and answer questions against your own documents without sending them to the cloud.

Conclusion and next steps

With Ollama, private AI on Linux is just another local service—simple to install, easy to script, and safe for sensitive workflows.

Your next steps: 1) Install Ollama and pull a starter model:

ollama pull llama3 && ollama run llama3

2) Wire it into your shell:

git diff | ollama run llama3 -p "Draft a commit message:"

3) Expose the HTTP API locally and experiment with automation:

curl -s http://localhost:11434/api/generate -d '{"model":"llama3","prompt":"Make a daily shell task checklist."}' | jq

When you’re ready, build a custom Modelfile to codify your team’s voice and defaults. Your AI, your machine, your rules.

Related posts