Posted on
• Artificial Intelligence

Getting Started with Ollama on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Getting Started with Ollama on Linux: Run Local LLMs from Your Bash Prompt

Want to run a state‑of‑the‑art language model like Llama 3 locally on your Linux machine in minutes—no cloud, no API keys, and scriptable from Bash? That’s exactly what Ollama enables. It brings local, privacy‑friendly LLMs to your terminal and lets you integrate them into real Linux workflows.

In this guide you’ll:

  • Install Ollama on Debian/Ubuntu, Fedora/RHEL, and openSUSE

  • Run your first model and manage model files

  • Call Ollama from Bash and via its HTTP API

  • Configure the systemd service and storage location

  • Create a small custom model with a Modelfile

By the end, you’ll be able to experiment interactively and automate real tasks with local models.

Why Ollama?

  • Local and private: Your prompts and data never leave your machine.

  • Scriptable: It’s a CLI with a simple HTTP API—perfect for Bash pipelines.

  • Fast to start: One command to install, one command to chat with a model.

  • Flexible: CPU or GPU, multiple model families (Llama, Mistral, and more).

1) Install Ollama

Ollama provides an official install script that detects your distro and sets up a systemd service. First, install prerequisites (curl, certificates) for your package manager, then run the installer.

Prerequisites on Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl ca-certificates

Prerequisites on Fedora/RHEL (dnf):

sudo dnf install -y curl ca-certificates

Prerequisites on openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl ca-certificates

Recommended install (official script):

curl -fsSL https://ollama.com/install.sh | sh

If you prefer to inspect the script before running:

curl -fsSLO https://ollama.com/install.sh
less install.sh
sudo sh install.sh

Verify:

ollama --version
sudo systemctl status ollama

The service listens on 127.0.0.1:11434 by default.

Optional: Run via Docker instead of a system install:

docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama

Tip: If you have an NVIDIA or AMD GPU and drivers installed, Ollama will attempt GPU acceleration automatically. For CPU‑only, set:

export OLLAMA_NO_GPU=1

2) Your First Model in the Terminal

Run Llama 3 interactively (Ollama will download it the first time):

ollama run llama3

After the model loads, type a prompt and press Enter. Exit with Ctrl+C or by typing /bye.

Useful model commands:

# Pull a model without running it (good for prefetching)
ollama pull llama3

# See what models you have locally
ollama list

# Remove a model to reclaim space
ollama rm llama3

Real‑world example: use a smaller model for quick tasks:

ollama run mistral

3) Call Ollama from Bash and the HTTP API

Ollama exposes a simple HTTP API on localhost:11434. That means you can wire it into shell scripts easily.

One‑off completion with curl:

curl -s http://localhost:11434/api/generate \
  -d '{"model":"llama3","prompt":"Give me 3 tips to improve Bash scripts."}'

If you prefer JSON lines streaming off, add "stream": true to the payload.

Summarize a long file using a Bash pipeline:

content="$(sed -n '1,200p' /var/log/syslog | tail -n 200)"
curl -s http://localhost:11434/api/generate \
  -d "$(jq -n --arg c "$content" --arg m "llama3" \
        '{model:$m, prompt:("Summarize this log in bullet points:\n\n" + $c)}')"

Quick health/version checks:

curl -s http://localhost:11434/api/version
curl -s http://localhost:11434/api/tags

4) Manage the Service and Configure Ollama

Ollama installs a systemd service. Common operations:

# Status and logs
sudo systemctl status ollama
journalctl -u ollama -f

# Start/stop/enable/disable
sudo systemctl start ollama
sudo systemctl stop ollama
sudo systemctl enable ollama
sudo systemctl disable ollama

Change the bind address/port (for local network access). Use a systemd drop‑in:

sudo systemctl edit ollama

Add:

[Service]
Environment=OLLAMA_HOST=0.0.0.0:11434

Then:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Move the model storage directory (default is under your home in ~/.ollama):

sudo mkdir -p /srv/ollama
sudo chown "$USER":"$USER" /srv/ollama

Persist via systemd drop‑in:

[Service]
Environment=OLLAMA_MODELS=/srv/ollama

Then reload and restart the service.

Handy diagnostics:

ollama info

5) Create a Custom Model with a Modelfile

You can “wrap” a base model with a system prompt or template using a Modelfile.

Create a file named Modelfile:

FROM llama3
SYSTEM You are a concise Linux assistant. Prefer terse explanations, show commands first.
PARAMETER temperature 0.4

Build and run:

ollama create linux-assistant -f Modelfile
ollama run linux-assistant

Now every prompt uses your system instructions by default—useful for standardizing behavior across scripts and teams.

Troubleshooting Tips

  • GPU not used when expected:

    ollama info

    Ensure proprietary GPU drivers are installed. For CPU‑only fallback:

    export OLLAMA_NO_GPU=1

  • Large downloads or low disk space:

    du -sh ~/.ollama
ollama rm <model-name>

  • Firewall note (if binding to 0.0.0.0): Open TCP 11434 only to trusted networks.

  • Uninstall (system install):

    sudo systemctl disable --now ollama
sudo rm -f /usr/local/bin/ollama
# Optional: remove models (this deletes all downloaded models)
rm -rf ~/.ollama

Conclusion and Next Steps

You now have a fast, private LLM stack that fits right into your Linux workflow. Try this next:

  • Pull a few models and benchmark them on your hardware:

    ollama pull llama3
ollama pull mistral

  • Wire Ollama into a Bash script that summarizes logs, explains errors, or generates docs.

  • Package a team‑wide Modelfile to standardize prompts and defaults.

When you’re ready to go deeper, explore the REST API endpoints, set up GPU tuning variables, and build small internal tools—powered entirely on your own Linux box.

Related posts