Chunking Strategies for RAG: A Bash-Friendly, Practical Guide

If your Retrieval-Augmented Generation (RAG) pipeline is feeding entire PDFs or randomly sliced 500-token blocks into a vector store, you’re likely paying more, retrieving worse, and hallucinating more than you need to. Good chunking is one of the highest-leverage improvements you can make: it directly affects retrieval recall/precision, latency, and token cost.

This guide explains why chunking matters and gives you 3–5 actionable strategies you can run from a Linux terminal. You’ll get ready-to-use Bash-friendly scripts, plus installation instructions for apt, dnf, and zypper.

Why chunking matters in RAG

• Relevance: Retrieval works best when chunks are self-contained, coherent units. Sentence boundaries, headings, and topic shifts matter.

• Recall vs precision: Oversized chunks reduce precision (too much unrelated text), undersized chunks reduce recall (insufficient context to answer).

• Cost and latency: Smaller, smarter chunks reduce tokens sent to the LLM and improve response times.

• Model compatibility: Tokenization, not characters, determines how much context a chunk really consumes. Always chunk by tokens when possible.

Common anti-patterns:

• Splitting purely by character count.

• No overlap (boundary loss).

• Ignoring document structure (headings, lists, code blocks).

• Not evaluating chunk coverage or retrieval quality.

Prerequisites (Linux)

We’ll use a few command-line tools and Python packages for tokenization and sentence splitting.

Install system packages:

• Debian/Ubuntu (apt):

  sudo apt update
  sudo apt install -y python3 python3-pip jq ripgrep
  

• Fedora/RHEL/CentOS Stream (dnf):

  sudo dnf install -y python3 python3-pip jq ripgrep
  

• openSUSE/SLE (zypper):

  sudo zypper refresh
  sudo zypper install -y python3 python3-pip jq ripgrep
  

Install Python libraries (user scope):

python3 -m pip install --user tiktoken blingfire sentence-transformers

Tip: If you prefer isolation, create a virtual environment and install the same pip packages there.

---

Strategy 1: Fixed-size token chunks with overlap (robust baseline)

When to use:

• Baseline for most corpora.

• You want predictable token sizes and straightforward indexing.

What it does:

• Splits text by tokens using a model-compatible tokenizer.

• Adds overlap to avoid boundary loss.

Script: chunk_by_tokens.py

#!/usr/bin/env python3
import os, sys, json
import tiktoken

ENCODING = os.getenv("ENCODING", "cl100k_base")  # good default for many OpenAI-like models
CHUNK_TOKENS = int(os.getenv("CHUNK_TOKENS", "512"))
OVERLAP = int(os.getenv("OVERLAP", "64"))

def read_text(path):
    if path and path != "-":
        with open(path, "r", encoding="utf-8", errors="ignore") as f:
            return f.read()
    return sys.stdin.read()

def main():
    path = sys.argv[1] if len(sys.argv) > 1 else "-"
    text = read_text(path)
    enc = tiktoken.get_encoding(ENCODING)
    toks = enc.encode(text)
    step = CHUNK_TOKENS - OVERLAP if CHUNK_TOKENS > OVERLAP else CHUNK_TOKENS
    i = 0
    chunk_id = 0
    while i < len(toks):
        j = min(i + CHUNK_TOKENS, len(toks))
        chunk_tokens = toks[i:j]
        chunk_text = enc.decode(chunk_tokens)
        out = {
            "id": f"{os.path.basename(path)}::{chunk_id}",
            "source": path,
            "start_token": i,
            "end_token": j,
            "n_tokens": len(chunk_tokens),
            "text": chunk_text
        }
        print(json.dumps(out, ensure_ascii=False))
        chunk_id += 1
        if j == len(toks): break
        i += step

if __name__ == "__main__":
    main()

Usage:

export CHUNK_TOKENS=512 OVERLAP=64
python3 chunk_by_tokens.py README.md > chunks_token.jsonl

Notes:

• Adjust CHUNK_TOKENS and OVERLAP to your model and retrieval settings.

• This yields durable, cost-predictable chunks.

---

Strategy 2: Sentence-aware packing up to a token budget

When to use:

• You want coherent chunks with natural boundaries.

• You have irregular sentence lengths or mixed prose/code.

What it does:

• Splits into sentences, then packs as many sentences as will fit into your token budget.

Script: chunk_sentence_pack.py

#!/usr/bin/env python3
import os, sys, json
import tiktoken
from blingfire import text_to_sentences

ENCODING = os.getenv("ENCODING", "cl100k_base")
CHUNK_TOKENS = int(os.getenv("CHUNK_TOKENS", "512"))
OVERLAP_SENTENCES = int(os.getenv("OVERLAP_SENTENCES", "1"))

def read_text(path):
    if path and path != "-":
        with open(path, "r", encoding="utf-8", errors="ignore") as f:
            return f.read()
    return sys.stdin.read()

def tokenize_len(enc, s): return len(enc.encode(s))

def main():
    path = sys.argv[1] if len(sys.argv) > 1 else "-"
    text = read_text(path)
    enc = tiktoken.get_encoding(ENCODING)
    sents = [s.strip() for s in text_to_sentences(text).splitlines() if s.strip()]
    chunks, cur, cur_tokens = [], [], 0
    cid = 0

    i = 0
    while i < len(sents):
        s = sents[i]
        s_tokens = tokenize_len(enc, s)
        # If single sentence is too big, hard-split by tokens
        if s_tokens > CHUNK_TOKENS:
            toks = enc.encode(s)
            start = 0
            while start < len(toks):
                end = min(start + CHUNK_TOKENS, len(toks))
                chunk_text = enc.decode(toks[start:end])
                print(json.dumps({
                    "id": f"{os.path.basename(path)}::big::{cid}",
                    "source": path,
                    "n_tokens": end - start,
                    "text": chunk_text
                }, ensure_ascii=False))
                cid += 1
                start = end
            i += 1
            continue

        if cur_tokens + s_tokens <= CHUNK_TOKENS:
            cur.append(s)
            cur_tokens += s_tokens
            i += 1
        else:
            chunk_text = " ".join(cur).strip()
            print(json.dumps({
                "id": f"{os.path.basename(path)}::{cid}",
                "source": path,
                "n_tokens": cur_tokens,
                "text": chunk_text
            }, ensure_ascii=False))
            cid += 1
            # apply sentence overlap
            cur = cur[-OVERLAP_SENTENCES:] if OVERLAP_SENTENCES > 0 else []
            cur_tokens = tokenize_len(enc, " ".join(cur)) if cur else 0

    if cur:
        print(json.dumps({
            "id": f"{os.path.basename(path)}::{cid}",
            "source": path,
            "n_tokens": cur_tokens,
            "text": " ".join(cur).strip()
        }, ensure_ascii=False))

if __name__ == "__main__":
    main()

Usage:

export CHUNK_TOKENS=512 OVERLAP_SENTENCES=1
python3 chunk_sentence_pack.py docs/guide.md > chunks_sentence.jsonl

---

Strategy 3: Structure-aware chunking for Markdown and docs

When to use:

• Technical docs with headings, lists, and code blocks.

• You want chunks aligned to logical sections.

What it does:

• Splits by Markdown headings (H1–H3), respecting code fences to avoid splitting inside code blocks.

Script: chunk_md_heading.sh

#!/usr/bin/env bash
# Split a Markdown file by headings into JSONL chunks. Keeps headings with text.
# Usage: ./chunk_md_heading.sh README.md > chunks_md.jsonl

file="${1:-/dev/stdin}"
awk -v SRC="$file" '
BEGIN {
  code=0; hdr=""; buf="";
}
function flush() {
  if (length(buf) > 0) {
    gsub(/\r/,"",buf)
    gsub(/\r/,"",hdr)
    # escape JSON
    t=buf; gsub(/\\/,"\\\\",t); gsub(/"/,"\\\"",t)
    h=hdr; gsub(/\\/,"\\\\",h); gsub(/"/,"\\\"",h)
    print "{ \"source\": \"" SRC "\", \"heading\": \"" h "\", \"text\": \"" t "\" }"
    buf=""
  }
}
{
  if ($0 ~ /^```/) { code = 1 - code; buf = buf $0 "\n"; next }
  if (!code && $0 ~ /^#{1,3} /) {
    flush()
    hdr=$0
    buf=$0 "\n"
  } else {
    buf = buf $0 "\n"
  }
}
END { flush() }
' "$file"

Usage:

bash chunk_md_heading.sh README.md > chunks_md.jsonl

Tip:

• You can feed each emitted section into Strategy 1 or 2 for token-aware sub-chunking. For example:

bash chunk_md_heading.sh README.md \
| jq -r .text \
| python3 chunk_by_tokens.py - > chunks_md_token.jsonl

---

Strategy 4: Semantic chunking using embeddings (when topic shifts)

When to use:

• Long narrative docs where topics shift mid-section.

• You want to start new chunks when semantics drift, not just on headings.

What it does:

• Splits into sentences, embeds them, and starts a new chunk when semantic similarity drops or the token budget would be exceeded.

Script: chunk_semantic.py

#!/usr/bin/env python3
import os, sys, json, math
import tiktoken
from blingfire import text_to_sentences
from sentence_transformers import SentenceTransformer
import numpy as np

ENCODING = os.getenv("ENCODING", "cl100k_base")
CHUNK_TOKENS = int(os.getenv("CHUNK_TOKENS", "512"))
SIM_THRESHOLD = float(os.getenv("SIM_THRESHOLD", "0.65"))  # lower => larger chunks
OVERLAP_SENTENCES = int(os.getenv("OVERLAP_SENTENCES", "1"))

def read_text(path):
    if path and path != "-":
        with open(path, "r", encoding="utf-8", errors="ignore") as f:
            return f.read()
    return sys.stdin.read()

def cosine(a, b):
    na = np.linalg.norm(a); nb = np.linalg.norm(b)
    if na == 0 or nb == 0: return 0.0
    return float(np.dot(a, b) / (na * nb))

def main():
    path = sys.argv[1] if len(sys.argv) > 1 else "-"
    text = read_text(path)
    enc = tiktoken.get_encoding(ENCODING)
    sents = [s.strip() for s in text_to_sentences(text).splitlines() if s.strip()]
    if not sents:
        return
    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    embs = model.encode(sents, convert_to_numpy=True, normalize_embeddings=False)

    cid = 0
    cur_sents, cur_tokens = [], 0
    ref_vec = None

    for i, s in enumerate(sents):
        s_tok = len(enc.encode(s))
        if s_tok > CHUNK_TOKENS:
            # flush current
            if cur_sents:
                print(json.dumps({
                    "id": f"{os.path.basename(path)}::{cid}",
                    "source": path,
                    "n_tokens": cur_tokens,
                    "text": " ".join(cur_sents)
                }, ensure_ascii=False))
                cid += 1
                cur_sents, cur_tokens, ref_vec = [], 0, None
            # hard-split big sentence by tokens
            toks = enc.encode(s)
            start = 0
            while start < len(toks):
                end = min(start + CHUNK_TOKENS, len(toks))
                chunk_text = enc.decode(toks[start:end])
                print(json.dumps({
                    "id": f"{os.path.basename(path)}::big::{cid}",
                    "source": path,
                    "n_tokens": end - start,
                    "text": chunk_text
                }, ensure_ascii=False))
                cid += 1
                start = end
            continue

        sim_ok = True
        if ref_vec is not None:
            sim = cosine(ref_vec, embs[i])
            sim_ok = sim >= SIM_THRESHOLD

        would_fit = (cur_tokens + s_tok) <= CHUNK_TOKENS
        if cur_sents and (not would_fit or not sim_ok):
            print(json.dumps({
                "id": f"{os.path.basename(path)}::{cid}",
                "source": path,
                "n_tokens": cur_tokens,
                "text": " ".join(cur_sents)
            }, ensure_ascii=False))
            cid += 1
            # overlap
            cur_sents = cur_sents[-OVERLAP_SENTENCES:] if OVERLAP_SENTENCES > 0 else []
            cur_tokens = len(enc.encode(" ".join(cur_sents))) if cur_sents else 0
            ref_vec = embs[i] if cur_sents == [] else embs[i-1]  # reset or keep near last

        if not cur_sents:
            ref_vec = embs[i]
        cur_sents.append(s)
        cur_tokens += s_tok

    if cur_sents:
        print(json.dumps({
            "id": f"{os.path.basename(path)}::{cid}",
            "source": path,
            "n_tokens": cur_tokens,
            "text": " ".join(cur_sents)
        }, ensure_ascii=False))

if __name__ == "__main__":
    main()

Usage:

export CHUNK_TOKENS=512 SIM_THRESHOLD=0.65 OVERLAP_SENTENCES=1
python3 chunk_semantic.py docs/whitepaper.md > chunks_semantic.jsonl

Notes:

• Lower SIM_THRESHOLD yields fewer, larger chunks; higher splits more aggressively.

• This is slower (embeddings) but can boost retrieval quality on narrative text.

---

Real-world pipeline: From docs to JSONL chunks

Example: Chunk a Markdown repo README using structure-aware + token packing, then preview a few chunks.

# 1) Structure-aware split
bash chunk_md_heading.sh README.md > sections.jsonl

# 2) Token pack each section to ~512 tokens with overlap
jq -r .text sections.jsonl \
| CHUNK_TOKENS=512 OVERLAP=64 python3 chunk_by_tokens.py - \
> chunks.jsonl

# 3) Quick peek
head -n 3 chunks.jsonl | jq .

Want to sanity-check coverage for a known keyword?

rg -n "kernel parameters|NUMA|cgroup" -e "$(jq -r .text chunks.jsonl)"  # illustrative pattern usage

Or count tokens distribution quickly:

jq -r .n_tokens chunks.jsonl | awk '{sum+=$1; cnt+=1} END{print "avg:",sum/cnt,"n=",cnt}'

---

How to choose a strategy

• If you need a fast, reliable baseline: Strategy 1 (fixed tokens + overlap).

• If your content is prose-heavy or FAQs: Strategy 2 (sentence-aware packing).

• If docs are technical with clear sections: Strategy 3 (heading-aware), optionally followed by 1 or 2 inside each section.

• If long narratives with topic drift: Strategy 4 (semantic).

Default starting point for many teams:

• Split by headings (3), then sentence-pack to 400–700 tokens with 5–10% overlap (2). Evaluate, then iterate.

---

Quick evaluation ideas (before you index)

• Keyword coverage: For a sample of questions, check whether the gold answer phrase exists in the top-k retrieved chunks. Quick and crude but useful.

• Chunk quality spot-check: Randomly sample 20 chunks; verify each reads coherently and stands on its own.

• Token cost modeling: Average tokens per chunk × expected top-k × request rate.

---

Conclusion and next steps

Chunking isn’t busywork; it’s a force multiplier for RAG accuracy and cost. Start simple with fixed-size tokens plus overlap, then graduate to sentence- or structure-aware chunking. When your docs are narrative and complex, add semantic chunking.

Call to action:

• Install the prerequisites using apt/dnf/zypper above.

• Drop the scripts into your repo.

• Generate chunks for one representative document with Strategy 1 or 2.

• Inspect the JSONL, adjust budgets/overlaps, and only then wire your chunks into your vector index.

Strong chunks in, strong retrieval out.