Posted on
• Artificial Intelligence

Open Source Artificial Intelligence for Businesses

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Open Source Artificial Intelligence for Businesses: A Linux + Bash Playbook

Modern businesses want AI that is cost-effective, private, and under their control. But the gap between “we should use AI” and “we run AI in production” is often filled with vendor lock-in, unpredictable pricing, and compliance risks. Open source AI closes that gap.

This article shows you how to stand up a private, business-ready open source AI stack on Linux using nothing but Bash, Docker, and Python. You’ll get actionable steps, copy-paste commands for apt, dnf, and zypper, and a practical example you can run today.

Why open source AI is winning in business

  • Cost and predictability: No per-token sticker shock. Optimize for your workload and hardware, not someone else’s margins.

  • Control and privacy: Keep data and models on-prem or in your VPC to align with compliance, data residency, and IP policies.

  • Customization: Fine-tune, swap models, or choose acceleration libraries without waiting for a vendor roadmap.

  • Interoperability: Use proven components (LLMs, vector DBs, pipelines) that compose via open APIs and standards.

  • Community and resilience: Large upstream communities, faster issue resolution, and no single-vendor dependency.

What you’ll build

  • A private LLM (using an open-weight model via Ollama or llama.cpp)

  • A retrieval-augmented generation (RAG) pipeline:

    • Vector database (Qdrant) to index your documents
    • Sentence-transformers for embeddings
    • Lightweight Python glue to answer questions with citations

  • Simple logging/guardrails to keep outputs auditable

Everything runs locally or on your server with Linux + Bash.

1) Prepare your Linux environment

Install core tools, build chain, OpenBLAS (for fast CPU math), Python, and Docker.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y \
  git curl jq \
  python3 python3-venv python3-pip \
  build-essential cmake pkg-config libopenblas-dev \
  docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Fedora/RHEL/CentOS (dnf):

sudo dnf -y groupinstall "Development Tools"
sudo dnf install -y \
  git curl jq \
  python3 python3-pip \
  cmake openblas-devel \
  docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y \
  git curl jq \
  python3 python3-pip \
  gcc gcc-c++ make cmake pkg-config libopenblas-devel \
  docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Then re-login so your user is in the docker group:

newgrp docker

Tip: If you prefer Podman to Docker, you can swap commands accordingly.

2) Run a private LLM locally

Option A: Ollama (easiest developer experience; open-source server, models vary by license)

Install (works on most distros):

curl -fsSL https://ollama.com/install.sh | sh

Start the server and pull an open-weight, business-friendly model (e.g., Mistral 7B Instruct, Apache-2.0):

ollama serve &
ollama pull mistral

Chat:

ollama run mistral "Write a two-sentence security policy for internal AI tools."

Programmatic call via REST:

curl -s http://localhost:11434/api/generate \
  -d '{"model":"mistral","prompt":"Summarize: Open source AI for business."}'

Option B: Ollama via Docker (good for servers)

docker run -d --name ollama \
  -p 11434:11434 \
  -v ollama:/root/.ollama \
  --restart unless-stopped \
  ollama/ollama:latest

# Pull and test
curl -s http://localhost:11434/api/pull -d '{"name":"mistral"}'
curl -s http://localhost:11434/api/generate -d '{"model":"mistral","prompt":"Hello from Docker!"}'

Option C: llama.cpp (fully local, lean; great on CPUs; compile from source)

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc)
# You’ll need to download a GGUF model file, e.g., Mistral 7B Instruct in GGUF format.
# Then:
./main -m /path/to/mistral-7b-instruct.Q4_K_M.gguf -p "Hello from llama.cpp!"

Notes:

  • Choose models with permissive licenses (e.g., Mistral, OpenHermes, or other Apache/MIT licensed variants) if you need maximum commercial flexibility.

  • Llama 3 is “open-weight” but not OSI open source; check license terms for your use case.

3) Add search over your documents (RAG) with Qdrant + sentence-transformers

We’ll:

  • Run Qdrant as a vector DB

  • Embed your documents with an open model

  • Retrieve context and ask the LLM for grounded answers

Start Qdrant:

docker run -d --name qdrant \
  -p 6333:6333 \
  -v qdrant_storage:/qdrant/storage \
  --restart unless-stopped \
  qdrant/qdrant:latest

Create a Python virtual environment and install deps:

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install sentence-transformers qdrant-client requests

Put a few .txt files in a local folder:

mkdir -p docs
# Add your company policies, SOPs, product info, etc. as .txt files in ./docs

Minimal RAG script (save as rag.py):

import os, glob, json, uuid
from typing import List, Tuple
import requests
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434")
MODEL = os.environ.get("OLLAMA_MODEL", "mistral")
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
COLLECTION = os.environ.get("QDRANT_COLLECTION", "docs")

# 1) Connect and ensure collection
qc = QdrantClient(url=QDRANT_URL)
if COLLECTION not in [c.name for c in qc.get_collections().collections]:
    qc.recreate_collection(
        collection_name=COLLECTION,
        vectors_config=VectorParams(size=384, distance=Distance.COSINE),
    )

# 2) Load and chunk docs
def load_docs(path="docs") -> List[Tuple[str,str]]:
    chunks = []
    for f in glob.glob(os.path.join(path, "*.txt")):
        with open(f, "r", encoding="utf-8", errors="ignore") as fh:
            text = fh.read()
        # Simple chunking
        words = text.split()
        for i in range(0, len(words), 150):
            chunk = " ".join(words[i:i+150])
            if chunk.strip():
                chunks.append((os.path.basename(f), chunk))
    return chunks

docs = load_docs("docs")

# 3) Embed and upsert
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")  # Apache-2.0
vectors = model.encode([c for _, c in docs], normalize_embeddings=True)

points = []
for (fname, chunk), vec in zip(docs, vectors):
    points.append(PointStruct(id=str(uuid.uuid4()), vector=vec.tolist(), payload={"file": fname, "text": chunk}))

if points:
    qc.upsert(collection_name=COLLECTION, points=points)

# 4) Query flow
def retrieve(query: str, top_k=4):
    qvec = model.encode([query], normalize_embeddings=True)[0].tolist()
    res = qc.search(collection_name=COLLECTION, query_vector=qvec, limit=top_k)
    return [{"text": r.payload["text"], "file": r.payload["file"], "score": r.score} for r in res]

def ask_llm(prompt: str) -> str:
    payload = {"model": MODEL, "prompt": prompt, "stream": False}
    r = requests.post(f"{OLLAMA_URL}/api/generate", json=payload, timeout=300)
    r.raise_for_status()
    return r.json().get("response", "")

def answer(query: str):
    passages = retrieve(query)
    context = "\n\n".join([f"[{i+1}] ({p['file']}) {p['text']}" for i, p in enumerate(passages)])
    prompt = f"You are a helpful assistant. Use the context to answer.\n\nContext:\n{context}\n\nQuestion: {query}\n\nCite sources by [number]."
    resp = ask_llm(prompt)
    print("Q:", query)
    print("A:", resp)
    print("\nSources:")
    for i, p in enumerate(passages):
        print(f"[{i+1}] {p['file']} (score={p['score']:.3f})")

if __name__ == "__main__":
    q = os.environ.get("QUESTION", "What is our password reset policy?")
    answer(q)

Run it:

export OLLAMA_MODEL=mistral
python rag.py
QUESTION="Summarize product warranty terms." python rag.py

You now have a private, local RAG system answering questions grounded in your own content.

4) Add simple governance: logs and guardrails

Start by logging every prompt/response pair to JSONL so you can audit, measure, and iterate.

Bash wrapper for curl calls to Ollama’s REST API:

export LLM_LOG=llm-audit.jsonl

ask() {
  local prompt="$1"
  local payload
  payload=$(printf '{"model":"%s","prompt":"%s","stream":false}' "${OLLAMA_MODEL:-mistral}" "$(printf "%s" "$prompt" | sed 's/"/\\"/g')")
  local resp
  resp=$(curl -s http://localhost:11434/api/generate -d "$payload")
  printf '{"ts":"%s","prompt":%s,"response":%s}\n' \
    "$(date -Is)" \
    "$(printf '%s' "$payload" | jq -c '.prompt')" \
    "$(printf '%s' "$resp" | jq -c '.response')" | tee -a "$LLM_LOG" >/dev/null
  echo "$resp" | jq -r '.response'
}

# Example
ask "List three risks of deploying AI without governance."

Filter and analyze logs:

jq -r '.prompt' llm-audit.jsonl | head
jq -r '.response' llm-audit.jsonl | wc -w

Practical guardrails you can add next:

  • Policy checks: Prepend a policy preamble into prompts (e.g., “Don’t reveal PII; cite sources.”).

  • Domain whitelists: Restrict context sources to approved repositories/buckets.

  • PII scrubbing: Run inputs/outputs through regex or an open-source PII detector before storage.

Install jq if you skipped earlier:

  • apt:
sudo apt update && sudo apt install -y jq
  • dnf:
sudo dnf install -y jq
  • zypper:
sudo zypper install -y jq

5) Rollout plan and real-world use cases

Start small, deliver value in weeks, then scale.

  • Week 1–2: Pilot a private LLM + RAG on one team’s documents. Success metric: time-to-answer reduced by 50% vs. manual search.

  • Week 3–4: Add logging and basic guardrails. Success metric: zero policy violations in a curated test set.

  • Month 2: Integrate SSO, add change management. Success metric: adoption by 2–3 teams, <200ms retrieval latency on your corpus.

Examples from the field:

  • Support copilot: Index product manuals and past tickets; reduce first-response times and boost case deflection.

  • Compliance Q&A: Ground answers in approved policies; show citations for auditability.

  • Engineering knowledge base: Replace tribal knowledge with searchable design docs and ADRs.

Troubleshooting tips

  • Slow CPU inference? Use quantized GGUF models (Q4_K_M etc.) with llama.cpp or Ollama; consider OpenBLAS/BLIS for better CPU perf.

  • GPU acceleration: Both Ollama and llama.cpp have CUDA/ROCm builds; ensure drivers are installed and matched to your distro.

  • Vector DB choice: Qdrant (AGPL) is great; if AGPL isn’t a fit, consider pgvector (PostgreSQL extension) or Milvus (Apache 2.0).

Conclusion and next steps (CTA)

Open source AI gives your business control, privacy, and predictable cost—without sacrificing capability. You just built a private LLM and a document-aware assistant entirely on Linux using Bash and a few containers.

Next steps:

  • Point the RAG pipeline at a real corpus (policy docs, KB, or wikis).

  • Add SSO and request/response logging in your app layer.

  • Pilot with one team, measure outcomes, then scale.

If you want a step-by-step hardening checklist (SSO, secrets, backups, monitoring) or help picking models for your domain, ask me for a tailored blueprint for your distro and hardware.

Related posts