Posted on
• Artificial Intelligence

Artificial Intelligence Document Processing

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

From Paper Piles to Queryable Knowledge: AI Document Processing on Linux (with Bash)

Buried under PDFs, scans, and forms? Imagine turning that mess into searchable, structured data you can query from the command line. In this post, you’ll learn how to build a practical AI-driven document processing pipeline using Linux tools and a local LLM. We’ll OCR scanned PDFs, normalize and extract text, enrich with AI-based classification/extraction, and index everything for lightning-fast search — all from Bash.

Why it matters:

  • Time: Reduce hours of manual data entry to minutes.

  • Accuracy: De-skewed, de-noised OCR improves search and extraction.

  • Control: Keep documents local and private; no mandatory cloud.

What you’ll get:

  • A reproducible CLI workflow.

  • 3–5 actionable steps with real commands.

  • Package install instructions for apt, dnf, and zypper.

The Big Picture: A Simple, Valid Pipeline

Classic enterprise document processing needs five stages: 1) Normalize: Standardize inputs (deskew, OCR). 2) Extract: Text, images, metadata. 3) Structure: Convert to Markdown/JSON, chunk intelligently. 4) Enrich with AI: Classify documents or extract fields (e.g., invoice totals). 5) Index & Query: Make it discoverable with FTS.

We’ll do each of these locally with open tools and an optional local LLM.

1) Install Your Linux Toolbox

These are the utilities we’ll use:

  • OCR and cleanup: tesseract-ocr, ocrmypdf

  • PDF tools: poppler (pdftotext, pdfimages)

  • Conversions: pandoc, ImageMagick, ghostscript

  • Data plumbing: jq, sqlite3

  • Python (optional for extra scripting): python3, pip/venv

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y \
  tesseract-ocr tesseract-ocr-eng ocrmypdf poppler-utils \
  imagemagick ghostscript jq sqlite3 pandoc \
  python3 python3-venv python3-pip

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y \
  tesseract tesseract-langpack-eng ocrmypdf poppler-utils \
  ImageMagick ghostscript jq sqlite pandoc \
  python3 python3-pip python3-virtualenv

openSUSE (zypper):

sudo zypper install -y \
  tesseract tesseract-ocr-traineddata-english ocrmypdf \
  poppler-tools ImageMagick ghostscript jq sqlite3 \
  pandoc python3 python3-pip python3-virtualenv

Notes:

  • You can add more Tesseract languages:
    • apt: apt search tesseract-ocr- (e.g., tesseract-ocr-deu)
    • dnf: dnf list tesseract-langpack\*
    • zypper: zypper search tesseract-ocr-traineddata

Optional local LLM (Ollama) for on-device AI:

curl -fsSL https://ollama.com/install.sh | sh
# Then pull a model (examples):
ollama pull llama3
# or
ollama pull mistral

2) OCR and Normalize Your PDFs

Turn scanned PDFs into searchable, deskewed PDFs. This makes downstream extraction more accurate.

Single file:

ocrmypdf --force-ocr --deskew --rotate-pages --optimize 3 \
  input.pdf ocred/input.ocr.pdf

Batch a folder:

mkdir -p ocred
find scans -type f -iname '*.pdf' -print0 | while IFS= read -r -d '' f; do
  base="$(basename "$f")"
  ocrmypdf --force-ocr --deskew --rotate-pages --optimize 3 \
    "$f" "ocred/${base%.pdf}.ocr.pdf"
done

Tips:

  • If files are already text PDFs, ocrmypdf will preserve text and add a clean text layer.

  • Use --language eng (or others) if mixed languages hurt OCR accuracy.

3) Extract Text, Images, and Convert to Markdown

Turn your normalized PDFs into machine-friendly text and Markdown for chunking.

Extract text with layout preserved:

pdftotext -layout ocred/sample.ocr.pdf - | awk 'NF' > extracted/sample.txt

Extract images for downstream vision models (optional):

mkdir -p extracted/images
pdfimages -png ocred/sample.ocr.pdf extracted/images/page

Convert to Markdown (often easier for LLMs due to headings/tables):

pandoc -s ocred/sample.ocr.pdf -t markdown -o extracted/sample.md

Chunk your Markdown into manageable pieces (simple word-based chunker to JSONL):

in="extracted/sample.md"
out="extracted/sample.jsonl"
max_words=220

awk -v max="$max_words" -v file="$in" '
function flush() {
  if (buf != "") {
    gsub(/"/, "\\\"", buf)
    print "{ \"source\": \"" file "\", \"text\": \"" buf "\" }"
    buf = ""
    w = 0
  }
}
{
  n = split($0, a, /[ \t]+/)
  for (i=1; i<=n; i++) {
    if (a[i] == "") continue
    if (w >= max) { flush() }
    if (buf == "") { buf = a[i]; w = 1 }
    else { buf = buf " " a[i]; w++ }
  }
  # Keep paragraph boundaries meaningful
  if ($0 == "") { if (w > 0) flush() }
}
END { flush() }
' "$in" > "$out"

Now you have chunked JSONL suitable for prompting an LLM or indexing.

4) Add AI: Classify or Extract Structured Fields (Locally)

Run a local model with Ollama to classify documents or extract key fields like totals, dates, or addresses.

Example: Extract invoice fields (vendor, invoice_number, total, date) from a chunk.

Prepare a prompt template:

read -r -d '' SYS << 'EOF'
You are a precise information extraction engine. Return strict JSON only.
Fields: vendor, invoice_number, total, date, currency, confidence (0..1).
If a field is not present, return null.
EOF

Process chunks from JSONL and produce JSON lines:

model="llama3"   # or "mistral" / another pulled model
input="extracted/sample.jsonl"

jq -c '.text' "$input" | while IFS= read -r TEXT; do
  PROMPT=$(cat <<EOF
System:
$SYS

User:
Extract these fields from the following document fragment and return only minified JSON:
$TEXT
EOF
)
  # Generate with Ollama
  out=$(printf "%s" "$PROMPT" | ollama generate -m "$model" 2>/dev/null)
  # Best-effort parse in case the model adds prose
  echo "$out" | jq -c 'try fromjson catch {"_raw": .}'
done > ai_extractions.jsonl

Classify document type (invoice, receipt, contract, report):

model="llama3"
text="$(pdftotext -layout ocred/sample.ocr.pdf -)"
prompt=$(cat << 'EOF'
System:
Classify the document type. Choose one of: ["invoice","receipt","contract","report","other"].
Return only {"type": "..."}.

User:
<<DOC>>
EOF
)
printf "%s\n%s\n" "$prompt" "$text" | ollama generate -m "$model"

Real-world examples:

  • Invoices: Extract line items and totals for accounting.

  • Contracts: Flag renewal dates or confidentiality clauses.

  • Lab reports: Extract test names and result values.

Privacy note: Local models keep your docs on your machine. For sensitive data, avoid sending to external APIs.

5) Index and Search with SQLite FTS5

Create a lightweight searchable index for all chunks (plus AI output if you like).

Create the DB and FTS table:

db="docs.db"
sqlite3 "$db" << 'SQL'
PRAGMA journal_mode=WAL;
CREATE TABLE IF NOT EXISTS docs (
  id INTEGER PRIMARY KEY,
  source TEXT,
  text TEXT
);
CREATE VIRTUAL TABLE IF NOT EXISTS docs_fts USING fts5(
  text, source, content='docs', content_rowid='id'
);
SQL

Ingest chunked JSONL:

jq -c '. as $o | {source: $o.source, text: $o.text}' extracted/sample.jsonl | \
while IFS= read -r row; do
  src=$(echo "$row" | jq -r '.source')
  txt=$(echo "$row" | jq -r '.text' | sed "s/'/''/g")
  sqlite3 "$db" "INSERT INTO docs (source, text) VALUES ('$src', '$txt');"
done

sqlite3 "$db" "INSERT INTO docs_fts (rowid, text, source) SELECT id, text, source FROM docs WHERE rowid NOT IN (SELECT rowid FROM docs_fts);"

Query it:

sqlite3 -cmd ".mode box" "$db" \
  "SELECT source, snippet(docs_fts, 0, '[', ']', '…', 8) AS hit
   FROM docs_fts WHERE docs_fts MATCH 'renewal NEAR/5 2025' LIMIT 5;"

Optionally, join AI-extracted JSON to the index:

  • Keep ai_extractions.jsonl alongside your chunks (same source).

  • Store extractions in a separate table and join by source.

Putting It All Together (One-Liner-ish)

Process a folder end-to-end:

set -euo pipefail

in_dir="scans"
ocr_dir="ocred"
work_dir="extracted"
db="docs.db"
model="llama3"

mkdir -p "$ocr_dir" "$work_dir"

# 1) OCR
find "$in_dir" -type f -iname '*.pdf' -print0 | while IFS= read -r -d '' f; do
  base="$(basename "$f" .pdf)"
  ocrmypdf --force-ocr --deskew --rotate-pages --optimize 3 \
    "$f" "$ocr_dir/$base.ocr.pdf"
done

# 2) Convert to Markdown
find "$ocr_dir" -type f -iname '*.pdf' -print0 | while IFS= read -r -d '' f; do
  md="$work_dir/$(basename "$f" .pdf).md"
  pandoc -s "$f" -t markdown -o "$md"
done

# 3) Chunk to JSONL (simple word-based)
> "$work_dir/chunks.jsonl"
for md in "$work_dir"/*.md; do
  awk -v max=220 -v file="$md" '
  function flush() { if (buf!=""){ gsub(/"/,"\\\"",buf); print "{ \"source\":\"" file "\", \"text\":\"" buf "\" }"; buf=""; w=0 } }
  {
    n=split($0,a,/[ \t]+/);
    for(i=1;i<=n;i++){ if(a[i]=="")continue; if(w>=max){flush()} if(buf==""){buf=a[i];w=1}else{buf=buf" "a[i];w++} }
    if($0==""){ if(w>0)flush() }
  } END{ flush() }' "$md" >> "$work_dir/chunks.jsonl"
done

# 4) AI extraction (optional)
jq -c '.text' "$work_dir/chunks.jsonl" | while IFS= read -r TEXT; do
  PROMPT=$(cat <<EOF
System:
You are a precise extraction engine. Return strict minified JSON.
Fields: vendor, invoice_number, total, date, currency, confidence (0..1).

User:
$TEXT
EOF
)
  printf "%s" "$PROMPT" | ollama generate -m "$model" || true
done > "$work_dir/ai_extractions.jsonl"

# 5) Index
sqlite3 "$db" << 'SQL'
PRAGMA journal_mode=WAL;
CREATE TABLE IF NOT EXISTS docs (id INTEGER PRIMARY KEY, source TEXT, text TEXT);
CREATE VIRTUAL TABLE IF NOT EXISTS docs_fts USING fts5(text, source, content='docs', content_rowid='id');
SQL

jq -c '. as $o | {source: $o.source, text: $o.text}' "$work_dir/chunks.jsonl" | \
while IFS= read -r row; do
  src=$(echo "$row" | jq -r '.source')
  txt=$(echo "$row" | jq -r '.text' | sed "s/'/''/g")
  sqlite3 "$db" "INSERT INTO docs (source, text) VALUES ('$src', '$txt');"
done

sqlite3 "$db" "INSERT INTO docs_fts (rowid, text, source) SELECT id, text, source FROM docs WHERE rowid NOT IN (SELECT rowid FROM docs_fts);"

Troubleshooting and Tips

  • OCR quality: Try different languages: --language eng+deu. Clean scans improve accuracy.

  • Speed: Use xargs -P "$(nproc)" to parallelize long-running steps like OCR and pandoc.

  • LLM output: Always validate and parse with jq -c 'try fromjson catch {"_raw": .}' to guard against formatting drift.

  • Storage: Use PRAGMA optimize; and VACUUM; in SQLite for large corpora.

  • Security: If documents are sensitive, stay fully local. If you must use a cloud API, anonymize/redact first.

Conclusion and Next Steps (CTA)

You now have a practical, local-first AI document pipeline on Linux:

  • OCR and normalize with ocrmypdf

  • Extract and structure with poppler + pandoc

  • Enrich with a local LLM via Ollama

  • Index and query with SQLite FTS5

Next steps:

  • Point it at a big directory and schedule via cron/systemd.

  • Add domain-specific prompts (invoices, contracts, reports).

  • Persist AI-extracted fields to tables and build dashboards or CLI reports.

  • Containerize the workflow for reproducibility across machines.

If you’d like, paste your best one-liners or improvements into a gist and share back — let’s make Linux + Bash the fastest way to turn documents into knowledge.

Related posts