Getting Started with Ollama on Linux: Run Local LLMs from Your Bash Shell

If you’ve ever wished you could use powerful AI models without sending your data to the cloud—or when the internet is flaky—this guide is for you. Ollama lets you run large language models (LLMs) locally, right from your Linux terminal. That means private prompts, predictable performance, and no API bills for every token.

In this post, you’ll learn what Ollama is, why it’s worth your time, and exactly how to install and use it from Bash on Debian/Ubuntu, Fedora/RHEL, and openSUSE systems. You’ll finish with working commands and real-world examples you can drop straight into your shell.

Why Ollama?

• Privacy by default: Your prompts and outputs never leave your machine.

• Cost control: No per-request API fees; just your local compute.

• Speed and reliability: Low latency and fully offline-friendly.

• Unix-friendly: A simple CLI and a local HTTP API that slot right into existing shell workflows.

1) Install Ollama (Debian/Ubuntu, Fedora/RHEL, openSUSE)

The easiest and recommended way to install Ollama on Linux is via the official install script. Below are distro-specific steps using apt, dnf, and zypper to install prerequisites and enable the service.

Note: Ollama will detect your CPU/GPU and download the right binary. Ensure you have recent NVIDIA/AMD drivers if you want GPU acceleration.

Debian/Ubuntu (apt)

sudo apt update
sudo apt install -y curl ca-certificates

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Start on boot and run now
sudo systemctl enable --now ollama

# Verify
ollama --version

Fedora/RHEL/CentOS (dnf)

sudo dnf install -y curl ca-certificates

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Start on boot and run now
sudo systemctl enable --now ollama

# Verify
ollama --version

openSUSE (zypper)

sudo zypper refresh
sudo zypper install -y curl ca-certificates

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Start on boot and run now
sudo systemctl enable --now ollama

# Verify
ollama --version

Alternative (manual package install):

• Download the latest .deb (Debian/Ubuntu) or .rpm (Fedora/openSUSE) from the official Ollama releases page.

• Install with your package manager:

  • Debian/Ubuntu:

  sudo apt install ./ollama_<version>_amd64.deb
  sudo systemctl enable --now ollama
  

  • Fedora/RHEL:

  sudo dnf install -y ./ollama-<version>.x86_64.rpm
  sudo systemctl enable --now ollama
  

  • openSUSE:

  sudo zypper install -y ./ollama-<version>.x86_64.rpm
  sudo systemctl enable --now ollama
  

Tip: If you ever need to restart the service:

sudo systemctl restart ollama

2) Pull Your First Model

Models are fetched on-demand. Start with a solid general-purpose model:

ollama pull llama3

You can list your local models anytime:

ollama list

And remove one if you need disk space:

ollama rm llama3

Note: Models can be several GB—ensure you have sufficient free space.

3) Chat and Generate from the Terminal

Interactive session:

ollama run llama3

Type your prompt, press Enter, and hit Ctrl+C or type /bye to exit.

One-off prompt:

ollama run llama3 -p "Write a haiku about Bash."

Pipe input into a model:

echo "Summarize why shell scripting is powerful." | ollama run llama3

Real-world example: generate a concise Git commit message from your diff

git diff | ollama run llama3 -p "Write a clear, conventional commit message for these changes:"

4) Automate Everyday Tasks with Bash

• Summarize recent logs:

  journalctl -u ssh --no-pager -n 300 | ollama run llama3 -p "Summarize the top issues and suggest fixes:"
  

• Turn a rough note into a script comment block:

  printf "turn this into a professional bash script header:\n\n%s\n" \
  "backup script: copies /var/www to /backup, keeps 7 days" | ollama run llama3
  

• Explain a command before you run it:

  echo "Explain what this command does: find . -type f -name '*.sh' -print0 | xargs -0 wc -l" | ollama run llama3
  

Pro tip: Redirect model output to files or pass through tools like tee, sed, or jq to fit your workflow.

5) Use the Local HTTP API with curl

Ollama provides a local REST API on port 11434. Make sure the service is running:

sudo systemctl status ollama

Generate text via curl:

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "Write a Bash one-liner to count total lines in all .sh files recursively."
}'

For nicely formatted output, pipe to jq:

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "Write a Bash one-liner to count total lines in all .sh files recursively."
}' | jq -r '.response'

Run the server on a custom host/port (e.g., for LAN use):

export OLLAMA_HOST=0.0.0.0:11434
ollama serve

Be mindful of security if exposing beyond localhost.

Troubleshooting Quick Wins

• Command not found: Reopen your terminal or ensure your shell PATH includes the Ollama binary installed by the script. You can also try hash -r in Bash or restart your login shell.

• Service not running:

  sudo systemctl enable --now ollama
  sudo journalctl -u ollama -f
  

• GPU not used: Update your GPU drivers; Ollama will auto-detect. You can temporarily force CPU by setting:

  export OLLAMA_NO_GPU=1
  

  then run your command again.

Conclusion and Next Steps

You now have a private, local AI workstation you can drive entirely from Bash. With Ollama installed and a model pulled, try:

• Wiring an LLM into your daily shell tasks (log triage, doc generation, quick code snippets).

• Building tiny CLI helpers around ollama run and curl to standardize prompts for your team.

• Exploring additional models with ollama pull mistral, ollama pull phi3, and comparing outputs.

Call to action:

• Install Ollama using the commands above for your distro.

• Pull llama3 and run your first prompt today:

  ollama pull llama3 && ollama run llama3 -p "What should I automate next in my Linux shell?"
  

  Then iterate—every useful one-liner you create with Ollama is another sharp tool in your Bash toolbox.