AI-Assisted Digital Forensics on Linux: Bash-First Workflows that Scale

If you’ve ever stared at a multi-terabyte image thinking “where do I even start?”, you’re not alone. Modern investigations can involve millions of artifacts across endpoints, servers, and cloud buckets. The problem: exhaustive, manual review doesn’t scale and can bury the signal in the noise. The value: combine Unix-y, auditable Bash pipelines with lightweight AI to prioritize what matters—fast—while preserving forensic rigor.

This post shows how to:

Stand up a reproducible, Linux-first toolchain
Extract features from evidence with Bash
Triage at scale with an Isolation Forest anomaly detector
Add YARA and network triage to focus your time where it counts

Everything runs locally, is scriptable, and keeps you in control.

Why AI + Bash is a valid forensic strategy

Reproducibility and chain-of-custody: Bash pipelines are transparent. Every command can be logged, hashed, and rerun. You can cite the exact version and parameters in your report.
Speed-to-triage: Simple ML models don’t replace expert judgment; they help prioritize files, processes, and flows that look statistically unusual.
Open tooling: Linux CLI + Python gives you cross-distro flexibility, works offline, and avoids black-box dependencies.
Auditability: Output artifacts (CSV/JSON, model hashes, command logs) support peer review and courtroom scrutiny.

Install the toolchain

The commands below install common forensic and ML tooling. Use the block for your distro.

Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y git binutils jq ripgrep parallel sleuthkit yara tshark python3 python3-venv python3-pip pipx

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y git binutils jq ripgrep parallel sleuthkit yara wireshark-cli python3 python3-pip pipx

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y git binutils jq ripgrep parallel sleuthkit yara wireshark-cli python3 python3-pip pipx

Then make sure pipx is on your PATH and install Volatility 3 (memory forensics):

pipx ensurepath
# Restart your shell if needed, then:
pipx install volatility3

Create a Python virtual environment for the ML bits:

python3 -m venv ~/.venvs/df-ai
source ~/.venvs/df-ai/bin/activate
pip install --upgrade pip
pip install scikit-learn pandas joblib

Tip: Keep a small requirements.txt with exact versions to lock your environment for court-ready reproducibility.

Actionable 1: Extract triage features from files with Bash

Before AI can help, you need features. The script below walks a directory or mounted image, then records size, MIME type, SHA-256, and a basic “strings density” indicator (strings per KB), all as line-delimited JSON (JSONL) for easy parsing.

Save as extract_features.sh and make it executable:

chmod +x extract_features.sh

Script:

#!/usr/bin/env bash
set -euo pipefail

if [[ $# -lt 2 ]]; then
  echo "Usage: $0 <evidence_path> <out.jsonl>" >&2
  exit 1
fi

EVID="$1"
OUT="$2"
: > "$OUT"

# Walk files safely (handles spaces/newlines)
while IFS= read -r -d '' f; do
  # Skip sockets/too-large-for-quick-triage files if desired
  if [[ ! -f "$f" ]]; then continue; fi

  size=$(stat -c %s -- "$f" 2>/dev/null || echo 0)
  # Avoid reading >100MB in quick triage, adjust to your risk appetite
  if (( size > 100*1024*1024 )); then continue; fi

  mime=$(file -b --mime-type -- "$f" 2>/dev/null || echo "application/octet-stream")
  hash=$(sha256sum -- "$f" 2>/dev/null | awk '{print $1}')
  # Count printable strings of length >=6
  scount=$(strings -n 6 -- "$f" 2>/dev/null | wc -l | tr -d ' ')
  # Normalize by size in KB to get a density-like signal
  kb=$(awk -v s="$size" 'BEGIN{printf("%.6f", (s/1024.0)+0.0001)}')
  spk=$(awk -v n="$scount" -v k="$kb" 'BEGIN{printf("%.6f", n/k)}')

  # Emit JSONL. We rely on jq for safe JSON construction.
  jq -nc \
    --arg path "$f" \
    --arg mime "$mime" \
    --arg sha256 "$hash" \
    --argjson size "$size" \
    --argjson strings_per_kb "$spk" \
    '{path:$path, mime:$mime, sha256:$sha256, size:$size, strings_per_kb:$strings_per_kb}' \
    >> "$OUT"
done < <(find "$EVID" -type f -print0)

Run it:

./extract_features.sh /mnt/evidence features.jsonl

Now you have machine- and human-friendly metadata to drive both rules and ML.

Actionable 2: Fast anomaly detection with scikit-learn

We’ll train a lightweight Isolation Forest on two numeric features: file size and strings_per_kb. This flags items that look unlike the bulk of the set. In practice, you can add more features (entropy, PE header flags, byte n-grams) later.

Save as ai_df_detect.py:

#!/usr/bin/env python3
import argparse, json, math, sys
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler

def load_jsonl(path):
    return pd.read_json(path, lines=True)

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--in", dest="inp", required=True, help="features.jsonl")
    ap.add_argument("--out", dest="out", required=True, help="ranked_findings.jsonl")
    ap.add_argument("--contamination", type=float, default=0.03, help="Expected anomaly rate")
    ap.add_argument("--top", type=int, default=200, help="Top N to output")
    args = ap.parse_args()

    df = load_jsonl(args.inp).fillna(0)
    # Numeric features; log-scale size to reduce skew
    df["log_size"] = (df["size"] + 1).apply(lambda x: math.log10(x))
    X = df[["log_size", "strings_per_kb"]].values

    scaler = StandardScaler()
    Xs = scaler.fit_transform(X)

    iso = IsolationForest(
        n_estimators=200,
        contamination=args.contamination,
        random_state=42,
        n_jobs=-1,
    )
    iso.fit(Xs)

    # Lower scores = more anomalous. Convert to positive "anomaly_score".
    scores = -iso.score_samples(Xs)
    df["anomaly_score"] = scores

    out_df = df.sort_values("anomaly_score", ascending=False).head(args.top)
    with open(args.out, "w") as w:
      for _, row in out_df.iterrows():
        rec = {
          "path": row["path"],
          "mime": row["mime"],
          "sha256": row["sha256"],
          "size": int(row["size"]),
          "strings_per_kb": float(row["strings_per_kb"]),
          "anomaly_score": float(row["anomaly_score"]),
        }
        w.write(json.dumps(rec) + "\n")

if __name__ == "__main__":
    sys.exit(main())

Run it in your venv:

source ~/.venvs/df-ai/bin/activate
python ai_df_detect.py --in features.jsonl --out ranked_findings.jsonl --contamination 0.02 --top 150

Review the output:

jq -r '.anomaly_score, .path, .mime, .size | @tsv' ranked_findings.jsonl | paste - - - - | sort -nr

This gives you a prioritized review list that often floats:

Oddly small/large executables with unusual string density
Packed or obfuscated binaries
Non-binary types with binary-like density (potential masquerading)

Reminder: These are leads, not verdicts. Anything elevated should go through your normal validation (static/dynamic analysis, sandboxing, timeline correlation).

Actionable 3: Hybrid triage with YARA + ML

Rules catch known-bad. ML surfaces unknown-weird. Use both.

1) Run your YARA set across the evidence to pull in high-confidence hits:

# Recursively scan with YARA; adjust rules path to yours
yara -r /path/to/rules.yar /mnt/evidence > yara_hits.txt
cut -d' ' -f2- yara_hits.txt | sort -u > matched_files.txt

2) Focus ML on what rules didn’t catch:

# Produce a file list of non-YARA-matched artifacts
comm -23 <(find /mnt/evidence -type f -print0 | tr '\0' '\n' | sort -u) <(sort -u matched_files.txt) > to_score.txt

# Extract features only for the non-matched set
while IFS= read -r f; do printf '%q\0' "$f"; done < to_score.txt \
| xargs -0 -I{} bash -c './extract_features.sh "{}" features_nomatch.jsonl'

3) Score and review:

python ai_df_detect.py --in features_nomatch.jsonl --out ranked_nomatch.jsonl --contamination 0.02 --top 200

This hybrid approach keeps your review volume low while preserving coverage.

Actionable 4: Quick network triage from PCAP with tshark

Extract simple flow features to identify spikes, rare talkers, or odd protocols.

1) Export flow-like records to CSV:

tshark -r capture.pcap \
  -T fields \
  -e frame.time_epoch -e ip.src -e ip.dst -e tcp.srcport -e tcp.dstport \
  -e udp.srcport -e udp.dstport -e frame.len -e _ws.col.Protocol \
  -E header=y -E separator=, \
  > flows.csv

2) Surface outliers with shell:

Find rare destinations:

cut -d, -f3 flows.csv | sort | uniq -c | sort -n | tail -50

Flag jumbo frames or size anomalies:

awk -F, 'NR>1 && $8>2000 {print $0}' flows.csv | head

Investigate uncommon protocols:

cut -d, -f9 flows.csv | sort | uniq -c | sort -n | tail -50

3) Feed sizes/ports into the same ai_df_detect.py pattern (use pandas to compute per-5s aggregates and run an Isolation Forest per host).

Real-world notes and good practice

Don’t overfit: Start with a small, general feature set and conservative anomaly rates. Validate against known-good corpora to reduce false positives.
Log everything: Save exact commands (use script -a session.log), versions (--version of tools), and model hashes (sha256sum ai_df_detect.py) in your case notes.
Keep artifacts immutable: Always work on verified copies or read-only mounts. Hash inputs and outputs.
Be explicit in reports: “AI-assisted triage” should be described as a prioritization aid. Final determinations rely on standard forensic techniques.

Conclusion and next steps

AI won’t replace your investigative judgment, but it can compress hours of needle-in-haystack triage into minutes—directly from your terminal. You now have:

A Linux-native, reproducible toolchain
A feature extractor for files
A quick anomaly detector to prioritize review
A hybrid YARA+ML and PCAP triage pattern

Next steps:

Extend features (e.g., PE header fields, simple entropy, byte histograms)
Add model validation sets and CI checks to your lab repo
Wrap these steps into a Makefile or single Bash driver script per case
Share sanitized pipelines internally for peer review

Have a favorite twist on this workflow or a dataset you’d like to see benchmarked? Drop a comment or open an issue in your team’s lab repo and iterate—like any good Bash script, these pipelines get sharper with every case.