Posted on
• Artificial Intelligence

Artificial Intelligence for JSON Processing in Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence for JSON Processing in Linux: Turn Plain Bash Into a Smart Data Toolbelt

Ever stared at a 50 MB JSON log and thought, “I wish I could just ask for what I need”? JSON is everywhere on Linux—APIs, logs, Kubernetes, CI artifacts—but the path from “what I want” to a correct jq filter can be slow and error-prone. AI can bridge that gap: describe your intent in natural language and let a model propose jq filters, summarize large JSON blobs, infer schemas for validation, and more—while you stay in control and run the final commands locally.

This post shows how to combine battle-tested Linux tools with AI assistance for faster, safer JSON processing at the command line.

Why AI + JSON on Linux makes sense

  • jq is powerful but verbose: LLMs are excellent at code synthesis and can generate jq filters from intent, saving trial-and-error.

  • Complex JSON is hard to “see”: AI can summarize and explain nested structures quickly.

  • Validation matters: inferring a JSON Schema with AI and using local validators adds guardrails to your automation.

  • Privacy and control: you can run models locally (via Ollama) or use a cloud model if you prefer—either way, you review and run the final commands.

What we’ll build

  • A minimal toolchain: jq, Python with pip, and an AI CLI

  • A Bash helper to convert natural language into a jq filter (and then run it)

  • JSON summarization for quick insights

  • Schema inference + local validation for safer pipelines

1) Install the basics

We’ll need jq, curl, Python 3, and pip. Install them with your package manager:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y jq curl python3 python3-pip
  • Fedora/RHEL (dnf):
sudo dnf install -y jq curl python3 python3-pip
  • openSUSE (zypper):
sudo zypper install -y jq curl python3 python3-pip

Next, install an AI CLI. We’ll use Simon Willison’s llm, which supports both cloud models and local models via plugins.

  • Install llm via pip (user install keeps it in your home directory):
python3 -m pip install --user llm

Optional: local models with Ollama (runs on your machine):

  • Install Ollama (official script):
curl -fsSL https://ollama.com/install.sh | sh
  • Start Ollama and pull a small model (example: Llama 3 8B):
ollama serve &
ollama pull llama3
  • Add the llm plugin for Ollama:
python3 -m pip install --user llm-ollama

Cloud option (OpenAI, etc.): Set your key for llm if you prefer a hosted model:

llm keys set openai

Tip: Add ~/.local/bin to your PATH if needed so llm is found:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

2) Natural language to jq: ask, review, run

Here’s a Bash function that:

  • Reads JSON from stdin

  • Asks an AI model for an appropriate jq filter based on your prompt

  • Shows you the filter

  • Runs it on your full input only after your confirmation

Add this to your ~/.bashrc (then source ~/.bashrc):

# ai2jq: turn natural-language intent into a jq filter, then run it
ai2jq() {
  if [ -z "$1" ]; then
    echo "Usage: ai2jq \"your instruction (what to extract/transform)\" [model]"
    echo "Example: ... | ai2jq \"list issue numbers and titles for open issues\""
    return 1
  fi

  local INSTRUCTION="$1"
  local MODEL="${2:-ollama:llama3}"  # use 'openai:gpt-4o-mini' or another llm model if you prefer

  # Capture full input to a temp file and create a small sample for the model
  local TMP_IN
  TMP_IN="$(mktemp /tmp/ai2jq.in.XXXXXX.json)"
  cat > "$TMP_IN"

  # If input is JSON Lines, sample the first 200 lines; otherwise slice arrays/objects
  local SAMPLE
  if head -n 1 "$TMP_IN" | grep -q '^[[:space:]]*{'; then
    SAMPLE="$(head -n 200 "$TMP_IN")"
  else
    # Try to make a compact array/object sample
    SAMPLE="$(jq 'try (.[0:50]) catch . | . as $x | (type == "array") or (type == "object") | if . then $x else input end' "$TMP_IN" 2>/dev/null || head -c 200000 "$TMP_IN")"
  fi

  # Ask the model for a jq filter. We ask for "only the jq program" to avoid extra prose.
  local PROMPT
  PROMPT="$(cat <<'EOF'
You are a helpful assistant that writes jq programs.
Given:
1) A natural-language instruction
2) A JSON sample

Write ONLY the jq program (no comments, no backticks, no extra text) that fulfills the instruction.
Prefer robust filters that won't break if some fields are missing.
EOF
)"

  local FILTER_RAW
  FILTER_RAW="$(printf "%s\n\nInstruction:\n%s\n\nJSON sample:\n%s\n" "$PROMPT" "$INSTRUCTION" "$SAMPLE" | llm -m "$MODEL")" || {
    echo "Model call failed." >&2
    rm -f "$TMP_IN"
    return 2
  }

  # Strip code fences or accidental prefixes
  local FILTER
  FILTER="$(printf "%s" "$FILTER_RAW" | sed -e 's/^```.*$//g' -e 's/^jq \+//g' -e 's/^#.*$//g')"

  echo "Proposed jq filter:"
  echo "--------------------------------"
  echo "$FILTER"
  echo "--------------------------------"
  read -r -p "Run this on your JSON? [y/N] " OK
  if [ "$OK" != "y" ] && [ "$OK" != "Y" ]; then
    echo "Aborted."
    rm -f "$TMP_IN"
    return 0
  fi

  jq -r "$FILTER" "$TMP_IN"
  local STATUS=$?
  rm -f "$TMP_IN"
  return $STATUS
}

Usage example (GitHub issues):

curl -s "https://api.github.com/repos/tensorflow/tensorflow/issues?per_page=50" \
| ai2jq "Show open issues as JSON lines with fields: number, title, label_names (array of names)."

Usage example (Kubernetes pods):

kubectl get pods -A -o json \
| ai2jq "List pods where phase != 'Running', output: {ns: namespace, pod: name, phase, restarts: containerStatuses[].restartCount | add}"

Notes:

  • Always review the suggested filter; you remain in control.

  • For privacy, use a local model like ollama:llama3. For better accuracy or complex cases, a hosted model may perform better.

3) Summarize and explain JSON fast

When you just need a quick sense of what’s inside a large JSON file or JSON Lines log, summarization helps. Here’s a function that:

  • Samples your JSON/JSONL

  • Asks the model to return a machine-readable summary (JSON)

  • Prints that summary

Add to ~/.bashrc:

json_summarize() {
  local MODEL="${1:-ollama:llama3}"
  local TMP_IN
  TMP_IN="$(mktemp /tmp/jsonsum.in.XXXXXX.json)"
  cat > "$TMP_IN"

  # Build a compact sample
  local SAMPLE
  if head -n 1 "$TMP_IN" | grep -q '^[[:space:]]*{'; then
    SAMPLE="$(head -n 200 "$TMP_IN")"
  else
    SAMPLE="$(jq -s '.[0:200]' "$TMP_IN" 2>/dev/null || head -n 200 "$TMP_IN")"
  fi

  local PROMPT
  PROMPT="$(cat <<'EOF'
You are a JSON analyst. Given a JSON or JSON Lines sample, return a compact machine-readable summary as pure JSON with keys:

- type: "array" | "object" | "jsonl"

- approx_items: integer (best guess)

- top_keys: array of strings (most common top-level keys)

- sample_paths: array of example JSONPath-like paths to notable fields

- quick_stats: object with a few counts or distributions (e.g., by level/severity/type)
Return only valid JSON, no commentary.
EOF
)"
  printf "%s\n\nSAMPLE:\n%s\n" "$PROMPT" "$SAMPLE" | llm -m "$MODEL"
  rm -f "$TMP_IN"
}

Examples:

  • Summarize GitHub issues:
curl -s "https://api.github.com/repos/tensorflow/tensorflow/issues?per_page=100" | json_summarize
  • Summarize a JSON Lines log:
head -n 500 /var/log/myapp.jsonl | json_summarize

Tip: You can then use the summary to decide exactly what jq filter you want the model to produce in the previous step.

4) Infer a JSON Schema and validate locally

AI can propose a JSON Schema from examples. Then, you can validate real data with a local validator for safety and CI.

Install the Python validator:

python3 -m pip install --user jsonschema

Infer a schema (adds a helper function):

infer_schema() {
  local MODEL="${1:-ollama:llama3}"
  local TMP_IN
  TMP_IN="$(mktemp /tmp/infer.in.XXXXXX.json)"
  cat > "$TMP_IN"

  # Small representative sample
  local SAMPLE
  SAMPLE="$(jq 'try (.[0:100]) catch .' "$TMP_IN" 2>/dev/null || head -c 200000 "$TMP_IN")"

  local PROMPT
  PROMPT="$(cat <<'EOF'
Given the following JSON sample, produce a JSON Schema (Draft 2020-12 or Draft-07) that:

- captures field types and required vs optional fields conservatively

- allows additionalProperties unless obviously wrong

- uses enum only when clearly discrete
Return ONLY the schema JSON.
EOF
)"
  printf "%s\n\nSAMPLE:\n%s\n" "$PROMPT" "$SAMPLE" | llm -m "$MODEL"
  rm -f "$TMP_IN"
}

Usage:

# Infer a schema from a sample and save it
curl -s "https://api.github.com/repos/tensorflow/tensorflow/issues?per_page=50" \
| infer_schema > schema.json

# Validate live data against the schema locally
curl -s "https://api.github.com/repos/tensorflow/tensorflow/issues?per_page=50" \
> data.json

python3 -m jsonschema -i data.json schema.json

This gives you reproducible, local checks—great for CI pipelines.

5) Real-world workflow: alert on slow requests in JSONL logs

Suppose access.jsonl contains web request logs with fields like latency_ms, route, and status.

  • Start with a quick summary:
head -n 1000 access.jsonl | json_summarize
  • Ask for a jq filter to extract slow requests:
head -n 5000 access.jsonl \
| ai2jq "From JSON Lines, select records with latency_ms > 1000, output minimal JSON lines with {t: time, route, latency_ms, status}. If fields missing, skip."
  • Ask for a grouped view by route with basic stats:
head -n 5000 access.jsonl \
| ai2jq "Group by route and produce an array of {route, count, p95_latency_ms, max_latency_ms, error_rate}, best-effort if fields missing."
  • Infer a schema to codify expectations (e.g., numeric latency, integer status) and validate the latest logs in CI.

This augments, not replaces, your standard Bash/jq practice: AI proposes; you confirm; local tools enforce correctness.

Tips for reliable use

  • Sample wisely: Feed the model a representative sample (not the entire file) to stay fast and private.

  • Review the filter: Always inspect the proposed jq before running.

  • Keep it local when needed: Use Ollama-backed models for sensitive data; use cloud only when appropriate.

  • Cache prompts: For recurring tasks, save the best jq filters to scripts and skip the AI step next time.

Conclusion and next steps

You don’t have to choose between raw power and convenience. With a few small Bash helpers, AI becomes a practical companion to jq—helping you get from “what I want” to working filters, summaries, and schemas faster, while you remain in control and run everything locally.

Your next step:

  • Install the prerequisites with apt/dnf/zypper and set up llm.

  • Paste the ai2jq, json_summarize, and infer_schema functions into your ~/.bashrc.

  • Try them on a real JSON source you use daily (API output, Kubernetes resources, or JSONL logs).

If you build a neat ai2jq recipe, share it with your team—future you will thank you.

Related posts