MCP with Ollama: Bring Local AI Into Your Linux Toolchain

If you’ve ever wished you could keep the speed and privacy of local LLMs while still giving them safe, auditable access to your files, terminals, and services, this is your moment. The Model Context Protocol (MCP) makes tools and data sources discoverable to AI assistants in a standardized, permissioned way. Pair that with Ollama—your favorite local LLM runtime—and you can wire up a powerful, private, Linux-native workflow that you actually control.

This post explains why MCP + Ollama is worth your time, and walks you through practical, bash-friendly steps to get started on Ubuntu/Debian, Fedora/RHEL, and openSUSE.

Why MCP + Ollama?

• Privacy by default: Ollama runs models locally; MCP lets you expose only the tools and data you intend, with consent prompts and auditability.

• Cost and speed: No per-token bills. Latency is your CPU/GPU, not the network.

• Composable architecture: MCP is a protocol, not a product. Mix-and-match servers for filesystem access, shell commands, git, search, databases—and route generation through Ollama.

• Works with real editors: MCP-enabled clients (e.g., desktop assistants and IDE extensions) can call tools, read files, and use local models in one place.

What you’ll build

• Ollama serving a local model (e.g., llama3)

• An MCP-aware client that can call a lightweight MCP server for Ollama (a tiny bridge that forwards generation requests to Ollama’s HTTP API)

• A safe, Linux-first workflow you can expand with more MCP servers over time

1) Install prerequisites (curl, jq, git, Python/Node)

You only need a few basics to script, test, and glue things together. Install with your distro’s package manager:

• Ubuntu/Debian (apt)

  • sudo apt update && sudo apt install -y curl jq git python3 python3-pip pipx nodejs npm

• Fedora/RHEL (dnf)

  • sudo dnf install -y curl jq git python3 python3-pip pipx nodejs npm

• openSUSE (zypper)

  • sudo zypper install -y curl jq git python3 python3-pip pipx nodejs npm

Tip: If pipx isn’t available on your distro, use python3 -m pip install --user pipx then ~/.local/bin/pipx ensurepath.

2) Install Ollama and pull a model

Ollama provides a simple shell installer. After install, enable the service and pull a model.

• Install Ollama:

  • curl -fsSL https://ollama.com/install.sh | sh

• Start on boot and now:

  • sudo systemctl enable --now ollama

• Pull a model (example: Llama 3):

  • ollama pull llama3

• Quick test (streamlined with jq):

  • curl -s http://localhost:11434/api/generate -d '{"model":"llama3","prompt":"Say hello from Ollama."}' | jq -r '.response'

You should see a response in your terminal; if not, confirm the service is running with systemctl status ollama.

3) Add a minimal MCP server that forwards to Ollama

MCP is transport-agnostic and typically uses stdio. In practice, you run a small MCP server process that exposes one or more “tools.” For Ollama, that means a tool like ollama.generate that takes a prompt (and optional model) and calls Ollama’s HTTP API at http://127.0.0.1:11434.

You can:

• Use a community MCP server that targets Ollama, or

• Write your own tiny bridge in Node/Python using an MCP SDK that:

  • Reads JSON-RPC requests on stdin
  • Exposes a generate tool
  • Calls Ollama’s POST /api/generate
  • Streams the response back to the client

A minimal HTTP call your MCP server would make looks like:

• curl -s http://127.0.0.1:11434/api/generate -H 'Content-Type: application/json' -d '{"model":"llama3","prompt":"Explain MCP in one sentence."}'

Security tip: Only expose the tools you need, validate inputs, and consider adding model allowlists (e.g., ["llama3","mistral"]) and prompt length caps.

4) Wire it into an MCP-capable client

Most MCP clients discover servers via a simple JSON config. As an example, here’s what an entry might look like for a desktop assistant that supports MCP. The server is started via node and passed an environment variable pointing to Ollama:

• Create or edit your client’s MCP config (example path):

  • ~/.config/YourClient/your_client_mcp_config.json

• Add a server entry similar to:

  • {"mcpServers":{"ollama":{"command":"node","args":["/opt/mcp-ollama/server.js"],"env":{"OLLAMA_URL":"http://127.0.0.1:11434"},"stdio":"pipe"}}}

Adjust the path to your MCP server script or binary. After saving:

• Restart the client

• Approve the new MCP server when prompted

• Test with a simple request like: “Use the ollama.generate tool to summarize: MCP connects tools to AI.”

If your client includes a “tools” pane, you should see the ollama.generate tool show up. If not, check stdout/stderr from the server process and confirm OLLAMA_URL points to your local Ollama.

5) Real-world, bash-friendly workflows

• Private doc Q&A (filesystem + Ollama)

  • Use a filesystem MCP server to expose a project directory read-only.
  • Ask your assistant: “Read README.md and produce a concise release checklist.” The client fetches content via the filesystem tool; your Ollama tool does the summarizing—everything local.

• Local code helper (editor + Ollama)

  • Pull a code model: ollama pull codellama (or your preferred coding model).
  • In your IDE with MCP support, let the assistant read files via an MCP filesystem server and generate patches using ollama.generate.

• Air-gapped servers

  • Bind Ollama only to localhost: default is local. If you must expose on LAN, set export OLLAMA_HOST=0.0.0.0:11434 and firewall it. Use an MCP server on the same machine so tooling remains private and auditable.

• Scriptable batch tasks

  • Generate quick structured outputs from bash using a tiny wrapper:
  • printf '%s' '{"model":"llama3","prompt":"Return a CSV with 3 example servers and ports."}' | curl -s http://127.0.0.1:11434/api/generate -H 'Content-Type: application/json' -d @- | jq -r '.response'
  • In an MCP client, the same prompt runs through your ollama.generate tool with audit prompts and user consent.

Troubleshooting

• Port conflicts: If 11434 is taken, set a different host: export OLLAMA_HOST=127.0.0.1:11500 then sudo systemctl restart ollama.

• Model not found: Verify with ollama list and pull again with ollama pull <model>.

• Client doesn’t see the server: Double-check the MCP config entry, the command path, and that your MCP server prints a proper JSON-RPC “ready” handshake to stdout.

Conclusion and next steps

MCP gives you safe, permissioned access to tools and data; Ollama gives you fast, private local models. Together, they turn your Linux box into a programmable AI workstation.

Your next steps:

• Solidify your bridge: implement or adopt a lean MCP server that exposes generate and any other Ollama features you need (e.g., chat, embeddings).

• Add more capability: pair your Ollama server with filesystem, git, web search, or database MCP servers.

• Standardize your workflow: check in your MCP config, model allowlists, and prompts to your dotfiles for reproducibility.

If you try this setup, note which prompts, models, and tool combinations feel “snappy” on your hardware. Small tweaks—like switching models or adding a token limit—often deliver big wins when everything runs locally.