Posted on
• Artificial Intelligence

Hybrid Search Explained

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Hybrid Search Explained (for Bash-first Linux Users)

If you’ve ever grepped for an error, only to learn later the system logged a “throttle” instead of your expected “rate limit,” you’ve met the limits of keyword search. Pure lexical tools (grep, ripgrep) are precise and fast, but they don’t understand meaning. Pure semantic tools (embeddings/vector search) understand meaning, but can miss exact filters and syntax. Hybrid search blends both: it keeps the precision of keywords and the intelligence of semantics—so you find the right thing faster.

This post explains hybrid search in plain terms, why it’s useful on the command line, and gives you a minimal, reproducible setup that combines ripgrep with a tiny local vector index built in Python. You’ll get actionable steps, examples, and scripts you can drop into your workflow.

What is hybrid search?

  • Lexical search (e.g., grep, ripgrep, BM25 in full‑text engines) matches exact tokens and patterns. It’s unbeatable for precision and filters (file globs, regex, line anchors).

  • Semantic search (vector/embedding-based) maps text to high-dimensional vectors. Similar meanings land close together. It excels at recall when wording varies (synonyms, paraphrases, typos).

  • Hybrid search scores documents with both approaches and fuses the scores. Typical strategies:

    • Weighted sum: score = α * lexical + (1-α) * semantic
    • Two-stage: lexical filter/top‑K → semantic re-rank
    • Reciprocal rank fusion: combine ranks from both lists

Result: you still get exact hits for “OOM” while also surfacing “out of memory” mentions—and you can sort or filter as you like.

Why it matters on the CLI

  • Logs and traces: “auth failed” vs “authentication error” vs “login denied”

  • Code search: “rate limit” vs “throttle”, “cache bust” vs “invalidate”

  • Docs/notes/Wikis: phrasing drift over time, across teams and vendors

Hybrid search tightens feedback loops: fewer missed hits, fewer yak-shaves, faster root cause and code spelunking.

Prereqs: install the basics

We’ll use ripgrep for lexical search and a small Python script for vector search.

  • Debian/Ubuntu (apt): sudo apt update && sudo apt install -y ripgrep jq curl python3 python3-pip python3-venv

  • Fedora/RHEL/CentOS (dnf): sudo dnf install -y ripgrep jq curl python3 python3-pip python3-virtualenv

  • openSUSE (zypper): sudo zypper install -y ripgrep jq curl python3 python3-pip python3-virtualenv

Note: On some distros, python3 -m venv works without an extra package. If python3-virtualenv or python3-venv isn’t found, skip it.

Create and activate a venv, then install Python deps:

  • Create venv: python3 -m venv .venv && . .venv/bin/activate

  • Upgrade pip: pip install --upgrade pip

  • Install libs: pip install sentence-transformers faiss-cpu numpy

Minimal hybrid search in 3 steps

We’ll index one vector per file (simple and good enough to get started). Then we’ll combine semantic scores with ripgrep hit counts.

1) Prepare a folder of text to search

  • Example: docs/ with .md, .txt, or code files. You can index any UTF‑8 file.

2) Build a tiny vector index (Python)

  • Save this as embed.py:
import sys, os, numpy as np
from sentence_transformers import SentenceTransformer

if len(sys.argv) < 2:
    print("Usage: python3 embed.py <root_dir>")
    sys.exit(1)

root = sys.argv[1]
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

paths, texts = [], []
for base, _, files in os.walk(root):
    for f in files:
        p = os.path.join(base, f)
        try:
            with open(p, "r", encoding="utf-8", errors="ignore") as h:
                txt = h.read().strip()
                if txt:
                    paths.append(p)
                    texts.append(txt)
        except Exception:
            pass

emb = model.encode(texts, normalize_embeddings=True, show_progress_bar=True)
np.save("emb.npy", emb)
with open("paths.txt", "w", encoding="utf-8") as o:
    o.write("\n".join(paths))
print(f"Indexed {len(paths)} files → emb.npy, paths.txt")
  • Run it: python3 embed.py docs/

3) Query semantically and fuse with ripgrep

  • Save this as search.py (semantic scoring; prints “path score” for all files):
import sys, numpy as np
from sentence_transformers import SentenceTransformer

if len(sys.argv) < 2:
    print("Usage: python3 search.py \"your query\" [topK]")
    sys.exit(1)

q = sys.argv[1]
topK = int(sys.argv[2]) if len(sys.argv) > 2 else 200

E = np.load("emb.npy")
paths = [p.strip() for p in open("paths.txt", encoding="utf-8")]
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
qv = model.encode([q], normalize_embeddings=True)[0]
scores = (E @ qv)  # cosine similarity because both are normalized
idx = np.argsort(-scores)[:min(topK, len(paths))]
for i in idx:
    print(f"{paths[i]} {scores[i]:.6f}")

  • Run semantic search: python3 search.py "rate limit burst throttle" 500 > vec.txt

  • Run lexical search over the same tree and normalize counts:

    • Get counts per file: rg -n -S -c "rate limit|throttle" docs/ > raw_lex.txt
    • Convert to “path count”: awk -F: '{print $1" "$2}' raw_lex.txt | sort > lex_counts.txt
    • Compute max and normalize to [0,1]:
    • Max: max=$(awk '{print $2}' lex_counts.txt | sort -nr | head -1)
    • Normalize: awk -v m=$max '{if(m>0){printf "%s %.6f\n",$1,$2/m}else{printf "%s %.6f\n",$1,0}}' lex_counts.txt > lex.txt

  • Fuse scores with a weighted sum (α = lexical weight):

    • Set weight: alpha=0.5
    • Join by path and compute fused score:
    • join -j 1 <(sort vec.txt) <(sort lex.txt) | awk -v a=$alpha '{printf "%s %.6f\n",$1, a*$3 + (1-a)*$2}' | sort -k2,2nr | head -20

Tip: join expects both files sorted on the join column (the path). The output prints the top 20 files by hybrid score.

Real-world examples

  • Logs: Query oom|out of memory|killed process. Lexical finds exact OOM lines; semantic also bubbles up lines saying “memory pressure” or “allocator failed” that lacked your exact tokens.

  • Code search: Query throttle rate limit backoff. Lexical catches exact “rate_limit” identifiers; semantic surfaces code/comments that say “debounce,” “cooldown,” or “quota” even without the exact words.

  • Docs/Runbooks: Query rotate credentials expired token. Results include exact “rotate keys” plus semantically similar guidance (“refresh secret,” “renew auth”).

4 actionable tips for better results

1) Start with two-stage retrieval for speed: use ripgrep to get the top 1,000 candidate files, then run search.py only on those. It’s cheap and often outperforms either method alone.
Example filter: rg -l -S "keyword1|keyword2" docs/ | head -1000 > cands.txt then adapt embed.py/search.py to load only cands.txt.

2) Tune α: if you’re missing exact token hits, increase alpha (e.g., 0.7). If you’re missing paraphrases, decrease it (e.g., 0.3).

3) Chunk long files: very large files benefit from paragraph- or section-level embeddings. Modify embed.py to split on blank lines so search.py returns more pinpointed hits.

4) Keep indexes fresh: rerun embed.py in CI or a cron job after content changes. Store emb.npy/paths.txt alongside your docs to speed bootstrapping.

Troubleshooting

  • No faiss-cpu on your arch: try pip install faiss-cpu==1.7.4 or skip Faiss entirely (we used plain NumPy dot products above, so Faiss isn’t strictly required in this minimal example).

  • Model download slow: prefetch with python3 -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')" on a box with better bandwidth and cache ~/.cache as needed.

  • Encoding errors: the scripts open files with errors="ignore". For binary files, they’re skipped; consider filtering with find globs if you have many binaries.

Conclusion and next steps

Hybrid search combines the best of both worlds: exactness from ripgrep, understanding from embeddings. You now have a minimal, composable Bash-friendly workflow you can run anywhere:

  • Build the vector index once (python3 embed.py docs/)

  • Query semantically (python3 search.py "your query")

  • Fuse with lexical hits (rg ... + the join/awk line)

From here, you can:

  • Add chunking to improve granularity.

  • Cache per-repo indexes and wire this into a shell function.

  • Scale up to a server with a native hybrid engine (e.g., systems that support BM25+vector and RRF), once you outgrow local scripts.

Your move: point this setup at your logs or docs, pick a query that’s historically painful, and see what hybrid search surfaces that pure grepping missed.

Related posts