Vector Database Benchmarks on Linux: Measure What Matters, Reproduce What You Trust

Ever shipped a Retrieval-Augmented Generation (RAG) system that “works on my laptop” but crawls in production? Or swapped vector databases because “someone’s benchmark” looked good—only to find your workload performs differently? Vector search performance is brutally context-dependent. The only benchmark that truly matters is the one that mirrors your data, your queries, and your SLOs.

This guide shows you how to design and run meaningful vector database benchmarks on Linux using Bash-friendly tooling. You’ll learn what to measure, how to stand up a reproducible benchmarking environment, and how to collect recall/latency numbers for both an in-process FAISS baseline and a server-side engine (Qdrant) you can run locally with Podman. We’ll include shell-first install commands for apt, dnf, and zypper so you can follow along on almost any distro.

Why benchmark vector databases at all?

• Vector search is not one-size-fits-all. HNSW/IVF/flat, RAM vs. disk, quantization, and filter selectivity each change behavior.

• Latency is not a single number. You must track p50/p95/p99 under concurrency and account for warm vs. cold caches.

• Recall at k is a target, not a side effect. Comparing latency without matching recall is apples-to-oranges.

• Real workloads have payload filters, varied dimensions, and mixed query distributions. Synthetic top-1 nearest-neighbor isn’t enough.

• Costs matter. Index time, RAM/VRAM footprint, and disk IO all affect total cost of ownership.

The upshot: benchmark the way you actually search, and do it reproducibly.

---

What you’ll build

• A Linux-native, reproducible harness to:
  • Generate a synthetic dataset (or plug in your own embeddings)
  • Build/search FAISS (in-process) as a “golden” baseline
  • Stand up Qdrant via Podman and measure server-side performance
  • Report recall@k, median and p95 latency, and QPS

You can extend the same pattern to compare other vector databases and index settings later.

---

Prerequisites: Install the basics

You’ll need Python, a container runtime (we’ll use Podman), and a couple of CLI helpers. Choose the commands for your distro.

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git curl jq numactl podman

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y python3 python3-venv python3-pip git curl jq numactl podman

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-venv python3-pip git curl jq numactl podman

Tip: If you prefer Docker, install it instead of Podman. For most local cases, Podman is simpler and rootless by default.

Create and activate a Python virtual environment:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip

Install Python packages we’ll use:

pip install numpy tqdm faiss-cpu qdrant-client

---

Start a local vector DB (Qdrant) with Podman

Qdrant is a popular open-source vector database that uses HNSW. Launch it rootlessly:

podman run -d --name qdrant -p 6333:6333 qdrant/qdrant:latest
podman logs -f qdrant | sed -n '1,20p'

You should see it listening on 0.0.0.0:6333. If a previous container exists, remove it first:

podman rm -f qdrant

---

A minimal, reproducible benchmark script

The script below will:

• Generate synthetic vectors and queries

• Compute exact neighbors with FAISS IndexFlatL2 (ground truth)

• Compare FAISS HNSW and Qdrant HNSW against that ground truth

• Report recall@k, median/p95 latency, and QPS for each engine

Save as bench.py:

#!/usr/bin/env python3
import os
import time
import json
import math
import numpy as np
from tqdm import tqdm

import faiss
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, SearchParams

# Config (override via env vars)
DIM = int(os.getenv("DIM", "128"))
N = int(os.getenv("N", "200000"))          # number of base vectors
Q = int(os.getenv("Q", "1000"))            # number of queries
K = int(os.getenv("K", "10"))
SEED = int(os.getenv("SEED", "42"))
HNSW_M = int(os.getenv("HNSW_M", "32"))
HNSW_EF = int(os.getenv("HNSW_EF", "128"))
HNSW_EF_CONSTRUCT = int(os.getenv("HNSW_EF_CONSTRUCT", "128"))
BATCH = int(os.getenv("BATCH", "1000"))
QDRANT_HOST = os.getenv("QDRANT_HOST", "127.0.0.1")
QDRANT_PORT = int(os.getenv("QDRANT_PORT", "6333"))
COLLECTION = os.getenv("COLLECTION", "bench")

rng = np.random.default_rng(SEED)
base = rng.standard_normal((N, DIM), dtype=np.float32)
queries = rng.standard_normal((Q, DIM), dtype=np.float32)

def recall_at_k(truth, preds):
    # truth/preds: (Q, K) int64 arrays of ids
    intersection = 0
    for i in range(truth.shape[0]):
        intersection += len(set(truth[i]).intersection(preds[i]))
    return intersection / (truth.shape[0] * truth.shape[1])

def measure_latencies(fn, iters=Q):
    latencies = []
    start = time.perf_counter()
    for _ in range(iters):
        t0 = time.perf_counter()
        fn()
        latencies.append((time.perf_counter() - t0) * 1000.0)  # ms
    total = time.perf_counter() - start
    latencies_sorted = sorted(latencies)
    median = latencies_sorted[int(0.5 * iters)]
    p95 = latencies_sorted[int(0.95 * iters)]
    qps = iters / total if total > 0 else 0.0
    return median, p95, qps, latencies

# Ground truth with exact search
gt_index = faiss.IndexFlatL2(DIM)
gt_index.add(base)
Dgt, Igt = gt_index.search(queries, K)

# FAISS HNSW
faiss_index = faiss.IndexHNSWFlat(DIM, HNSW_M)
faiss_index.hnsw.efConstruction = HNSW_EF_CONSTRUCT
faiss_index.hnsw.efSearch = HNSW_EF
faiss_index.add(base)

def faiss_searcher():
    _D, I = faiss_index.search(queries, K)
    return I

# Warm-up + measure FAISS
_ = faiss_searcher()
t0 = time.perf_counter()
I = faiss_searcher()
faiss_time = time.perf_counter() - t0
faiss_recall = recall_at_k(Igt, I)

# per-query FAISS latencies (simulate individual calls)
def faiss_one(i):
    _D, I = faiss_index.search(queries[i:i+1], K)
    return I

faiss_median, faiss_p95, faiss_qps, _lat = measure_latencies(lambda idx=[0]: faiss_one(rng.integers(0, Q)), iters=Q)

print(json.dumps({
    "engine": "faiss_hnsw",
    "dim": DIM, "n": N, "q": Q, "k": K,
    "hnsw_m": HNSW_M, "hnsw_ef": HNSW_EF, "hnsw_ef_construct": HNSW_EF_CONSTRUCT,
    "recall_at_k": round(faiss_recall, 4),
    "median_ms": round(faiss_median, 3),
    "p95_ms": round(faiss_p95, 3),
    "qps": round(faiss_qps, 2)
}, ensure_ascii=False))

# Qdrant setup
client = QdrantClient(host=QDRANT_HOST, port=QDRANT_PORT)
client.recreate_collection(
    collection_name=COLLECTION,
    vectors_config=VectorParams(size=DIM, distance=Distance.EUCLID)
)

# Upsert in batches
ids = np.arange(N, dtype=np.int64)
for i in tqdm(range(0, N, BATCH), desc="Qdrant upsert"):
    pts = [
        PointStruct(id=int(ids[j]), vector=base[j].tolist())
        for j in range(i, min(i + BATCH, N))
    ]
    client.upsert(collection_name=COLLECTION, points=pts, wait=True)

# Qdrant batched search (single-shot for recall)
def qdrant_search_all():
    # For efficiency, we’ll issue queries one-by-one; you can batch using search_batch
    preds = np.empty((Q, K), dtype=np.int64)
    for i in range(Q):
        res = client.search(
            collection_name=COLLECTION,
            query_vector=queries[i].tolist(),
            limit=K,
            search_params=SearchParams(hnsw_ef=HNSW_EF)
        )
        preds[i] = [int(p.id) for p in res]
    return preds

# Warm-up + measure Qdrant
_ = qdrant_search_all()
t0 = time.perf_counter()
Iq = qdrant_search_all()
qdrant_time = time.perf_counter() - t0
qdrant_recall = recall_at_k(Igt, Iq)

# per-query Qdrant latencies
def qdrant_one(i):
    res = client.search(
        collection_name=COLLECTION,
        query_vector=queries[i].tolist(),
        limit=K,
        search_params=SearchParams(hnsw_ef=HNSW_EF)
    )
    return [int(p.id) for p in res]

qdrant_median, qdrant_p95, qdrant_qps, _lat = measure_latencies(lambda idx=[0]: qdrant_one(rng.integers(0, Q)), iters=Q)

print(json.dumps({
    "engine": "qdrant_hnsw",
    "dim": DIM, "n": N, "q": Q, "k": K,
    "hnsw_m": HNSW_M, "hnsw_ef": HNSW_EF,
    "recall_at_k": round(qdrant_recall, 4),
    "median_ms": round(qdrant_median, 3),
    "p95_ms": round(qdrant_p95, 3),
    "qps": round(qdrant_qps, 2)
}, ensure_ascii=False))

Run it with a fair, CPU-stable setup. For single-thread baselines, pin to cores and avoid OpenMP overcommit:

export OMP_NUM_THREADS=1
taskset -c 0-3 python bench.py | tee results.jsonl

Tweak parameters (e.g., larger ef for higher recall, at the cost of latency):

HNSW_EF=256 K=20 python bench.py

Replace the synthetic dataset with your own embeddings by loading arrays into base and queries. Keep IndexFlatL2 for ground truth unless you switch metrics.

---

4 actionable steps for trustworthy vector DB benchmarks

1) Define your SLOs and match recall before comparing latency

• Pick a target recall@k (e.g., 0.95 at k=10).

• Tune index parameters (e.g., HNSW ef) per engine until both hit the same recall.

• Only then compare tail latency (p95/p99) and QPS. Otherwise, you’ll benchmark “faster but wrong.”

2) Reproduce the environment

• Fix seeds, pin CPU cores, and freeze versions:

pip freeze > requirements.txt
uname -a
cat /etc/os-release
lscpu | sed -n '1,10p'

• Disable noisy neighbors (browser tabs, cron jobs) and keep thermals steady. Consider:

sudo sysctl vm.swappiness=1
numactl --interleave=all taskset -c 0-3 python bench.py

3) Benchmark realistic queries and filters

• Real systems filter by metadata (tenant, time window, type). Test with filters and without.

• Batch queries if your service does; engines behave differently under batching and concurrency.

• Vary vector dimension and payload size; memory and cache behavior can change dramatically.

4) Measure cost, not just speed

• Track index time, index size, and resident memory. For containers:

podman stats --no-stream qdrant

• If you’ll run on disks, test on disks (cold cache, SSD vs. HDD). If you’ll run in RAM, drop caches between runs to test cold start:

sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

---

Real-world extensions

• Compare more engines the same way: Weaviate, Milvus, Elasticsearch/OpenSearch kNN, pgvector. Keep the “match recall first” rule.

• Try canonical datasets (e.g., SIFT1M, GIST1M) or your production embeddings. Synthetic vectors are fine for measuring relative trends but won’t capture your distribution quirks.

• Test concurrency:

GNU_TIME="command time -f 'elapsed=%E maxrss=%M KB'"
$GNU_TIME xargs -I{} -P 8 -n 1 bash -c 'python bench.py >/dev/null' <<< "$(yes | head -n 16)"

• Automate runs and parameter sweeps (e.g., ef in [64, 128, 256], M in [16, 32]) and export JSONL results to a plotting tool.

---

Troubleshooting

• faiss-cpu build errors: upgrade pip and wheel, or install a newer Python. Manylinux wheels exist for common platforms.

• Qdrant port conflicts: stop old containers:

podman ps
podman rm -f qdrant

• Low recall: increase HNSW_EF (search) and consider a larger HNSW_EF_CONSTRUCT or HNSW_M (index). For FAISS HNSW:

HNSW_M=32 HNSW_EF=256 HNSW_EF_CONSTRUCT=200 python bench.py

---

Conclusion and next steps

Benchmarking vector databases isn’t about finding the universally “fastest” engine—it’s about proving the best setup for your workload at your target recall and SLOs. You now have a Linux-first, Bash-friendly harness to measure recall, latency, and throughput for both an in-process baseline and a server engine.

Your next step:

• Swap in your real embeddings and filters.

• Match recall across two engines you’re considering.

• Run a parameter sweep and compare p95 latency vs. memory footprint.

• Capture your environment and publish a short benchmark note so others (and future you) can reproduce it.

If you want a follow-up post, tell me which engines you’d like to add (Weaviate, Milvus, pgvector, OpenSearch), and I’ll extend this harness with turnkey install/run snippets for each.