Posted on
• Artificial Intelligence

MCP with Local LLMs

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

MCP with Local LLMs: Private, Scriptable AI on Your Linux Box

If you’ve ever wished your shell scripts could think—contextually, securely, and offline—you’re going to like where this is going. The Model Context Protocol (MCP) lets an AI act on well-defined tools (like a shell, filesystem, databases, or web fetchers) under your control. Pair that with a local LLM, and you get a fast, private, automatable assistant that can write scripts, run them, and iterate—all from your own Linux machine.

This post explains what MCP is, why it pairs so well with local models, and how to get a practical setup working on Linux. You’ll walk away with a working local LLM runtime, a client that speaks MCP, and concrete examples you can try in Bash right now.

What is MCP and why it matters

  • MCP (Model Context Protocol) is an open protocol that standardizes how AI clients talk to “tools” (servers) like the shell, filesystem, HTTP fetchers, databases, and more.

  • It puts the client in charge: tools are exposed explicitly, permissions can be scoped, and every action is visible for review/approval.

  • It’s model-agnostic: you can use a cloud model today or a local LLM tomorrow without rewriting your tools.

  • With local LLMs, you add:

    • Privacy: keep your code and data on your machine.
    • Predictable cost: no API bills.
    • Low latency: GPU or CPU on-device.
    • Offline capability: work even without internet.

In short, MCP is the missing glue that turns “chat with a model” into “get work done safely,” and local LLMs make it your own.

What you’ll build

  • A local LLM runtime using Ollama.

  • An MCP-aware client (e.g., a VS Code assistant that can call shell/filesystem tools).

  • A workflow where the AI:

    • Writes a Bash script.
    • Saves it to your filesystem.
    • Runs it via an MCP shell tool (with your approval).
    • Iterates safely until the job is done.

Prerequisites: core tools

Install a few CLI essentials you’ll use for testing and scripting:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl git jq

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y curl git jq

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl git jq

Step 1 — Install a local LLM runtime (Ollama)

Ollama is a simple, popular way to run models locally on CPU or GPU.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Start and enable the service (so it runs on boot):

sudo systemctl enable --now ollama

Pull a model (examples below—pick one that fits your hardware):

# General-purpose instruction models
ollama pull llama3.1
ollama pull mistral:7b

# Coding-focused with strong reasoning
ollama pull qwen2.5-coder:7b

Quick smoke test:

ollama run llama3.1 "In one sentence, explain what MCP is."

Tip: If you have an NVIDIA GPU, ensure your drivers and CUDA runtime are installed for best performance. If not, Ollama will fall back to CPU just fine—just slower.

Step 2 — Pick an MCP client that supports local LLMs

You need a client that:

  • Can talk to a local model (Ollama).

  • Understands MCP tools and their permissions.

  • Lets you approve or deny tool calls (e.g., running shell commands).

Popular option: a VS Code assistant extension that supports MCP and local providers like Ollama (for example, Cline). In that setup you would:

  • Set the provider to “Ollama.”

  • Choose your local model, e.g., qwen2.5-coder:7b.

  • Enable or configure MCP servers (filesystem, shell, fetch, etc.).

  • Grant or deny tool calls as they happen.

Exact setup steps vary by client, but the pattern is the same:

  • Provider: Ollama with host http://127.0.0.1:11434.

  • Model: one you pulled above.

  • MCP servers: enable filesystem and shell at minimum so the model can create files and run commands under your supervision.

Security tip:

  • Run shell-capable MCP servers with least privilege. Consider a dedicated Unix user, constrained working directory, and no access to secrets. Always require explicit approval before executing commands.

Step 3 — A safe mental model for MCP on Linux

  • The model “asks” to use tools.

  • The client mediates and shows you the exact command/file change first.

  • You approve/deny. Nothing runs without you.

  • The tool returns output back to the model for follow-up steps.

That means you can keep everything in a project directory, see every command, and roll back easily with git.

Initialize a test workspace:

mkdir -p ~/mcp-lab && cd ~/mcp-lab
git init

Step 4 — Real-world examples you can try

Below are 4 practical, Bash-friendly tasks you can run once your client is wired to Ollama and your MCP servers (filesystem + shell) are enabled.

1) Parse Nginx logs into structured CSV

  • Prompt your assistant: “We have access to a shell and filesystem. Create a Bash script that reads Nginx access logs from ./logs/access.log and outputs a CSV with columns: ip, time, method, path, status, bytes. Save it as parse_nginx.sh, make it executable, then run it.”

  • Expect the following flow:

    • It writes parse_nginx.sh using the filesystem tool.
    • It asks to run chmod +x parse_nginx.sh via the shell tool.
    • It asks to run ./parse_nginx.sh ./logs/access.log > out.csv.

  • Verify results:

head -n 5 out.csv
  • Iterate: Ask it to add a header row, handle quoted paths, or include user-agent.

2) Summarize and visualize disk usage

  • Prompt: “Write a Bash script that prints the top 10 directories by size under the current folder, then outputs a JSON file and a simple Markdown report. Save as du_report.sh and run it.”

  • The assistant will likely call:

du -sh ./* | sort -hr | head -n 10
  • Then it may create a JSON snapshot for later tracking with jq. You can review diffs in git:
git add .
git commit -m "Add disk usage scripts and reports"

3) Postmortem helper for a failed service

  • Prompt: “We have an MCP shell and filesystem. Write a script that checks systemd status for a given service name, tails the last 200 journal lines, and extracts common failure patterns (OOM, permission denied, missing file). Save it as diagnose_service.sh. Run it for ‘ssh’.”

  • Expect tool calls like:

systemctl status ssh
journalctl -u ssh -n 200 --no-pager
  • Iterate: Ask it to detect repeated failures over time and output a Markdown checklist with proposed fixes.

4) Generate, run, refine: a CLI for log filtering

  • Prompt: “Create a Bash CLI ‘loggrep’ that supports flags: --since, --until, --contains, and --json. It should stream logs from stdin or a file, filter by time/substring, and optionally emit JSON lines. Save to bin/loggrep, mark executable, then test it on a sample log. Include usage in the README.”

  • The model will write files, execute tests, and refine after you give feedback.

Each example demonstrates the loop: propose code -> write file -> run commands -> analyze output -> refine.

Step 5 — Tips for model and tool quality

  • Choose capable local models:

    • General tasks: llama3.1, mistral:7b
    • Coding/ops-heavy: qwen2.5-coder:7b (often very good at CLI and code editing)

  • Keep iterations small:

    • Ask it to write a minimal script, test it, then expand.

  • Lock down the environment:

    • Point the shell tool at a working directory.
    • Avoid exposing dotfiles or secrets.
    • Use a non-privileged user for shell tasks.

  • Track changes:

    • Commit frequently with git so you can revert if an iteration goes awry.

Optional: install common tools it may request

Many scripts rely on ripgrep, fd, or GNU tools. Here are quick installs:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y ripgrep fd-find coreutils findutils
# Note: fd is often installed as fdfind on Debian/Ubuntu

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y ripgrep fd-find coreutils findutils

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y ripgrep fd coreutils findutils

Troubleshooting

  • The client won’t connect to Ollama:

    • Ensure the service is up:
    systemctl status ollama
curl -s http://127.0.0.1:11434/api/tags | jq .
    • Confirm your client is pointing to http://127.0.0.1:11434.

  • Tool calls fail or hang:

    • Check that your MCP client has the filesystem/shell tools enabled.
    • Verify permissions: the tool’s working directory exists and is writable.

  • Model gives weak Bash:

    • Try a coder-focused model (e.g., qwen2.5-coder:7b) or a larger variant if you have GPU headroom.
    • Be explicit in prompts: “Use POSIX sh,” “Assume GNU tools,” “Handle spaces in filenames,” etc.

Security notes

  • Treat shell-capable MCP servers as you would any automation:

    • Least privilege user.
    • Explicit allowlist of directories.
    • Mandatory approvals for every command.

  • Review scripts before running. The beauty of MCP is that it shows you exactly what will happen—use that.

Conclusion and next steps

You now have the building blocks for private, scriptable AI on Linux:

  • A local LLM runtime (Ollama).

  • An MCP-aware client that mediates safe tool use.

  • Practical Bash workflows where the model writes code, runs it with your approval, and iterates.

Your next step:

  • Wire up filesystem and shell tools in your preferred MCP client, open a project folder, and try the “Nginx-to-CSV” example above.

  • Once you’re comfortable, add more tools (HTTP fetchers, Git, or a database server) and let your local LLM orchestrate more complex tasks—still under your control.

If you build something cool with MCP and Bash, share it—these patterns get better as we trade real scripts, prompts, and guardrails.

Related posts