Posted on
• Artificial Intelligence

Combining grep, awk, sed and Artificial Intelligence

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Combining grep, awk, sed and Artificial Intelligence

If your terminal is where real work gets done, this post is for you. We’ll combine the speed and precision of grep/awk/sed with the judgment and summarization power of AI to slash time spent on log triage, incident response, data cleanup, and commit messaging—without abandoning your Bash muscle memory.

Problem/value in one line: Unix text tools are unbeatable for filtering and reshaping data; AI is unbeatable for fuzzy tasks like summarization, clustering, and naming. Together, they turn mountains of text into decisions in minutes.

Why this combo works

  • Deterministic + probabilistic: grep/awk/sed do exactly what you tell them; AI infers what you mean. Use the former to narrow and structure; use the latter to summarize, classify, or explain.

  • Speed and cost: Pre-filter with grep/sed/awk to reduce tokens before hitting an AI model (local or cloud). Faster, cheaper, safer.

  • Privacy by design: Scrub secrets with sed and awk before sending anything to AI. Keep raw data on your box; send only what’s necessary.

Install the essentials

You likely have these already, but here are one-liners to ensure the basics are installed. We’ll use grep, awk, sed, curl, and jq (for handling JSON). Optionally install ripgrep for speed.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y grep gawk sed curl jq ripgrep
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y grep gawk sed curl jq ripgrep
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y grep gawk sed curl jq ripgrep

Optional local AI runtime (Ollama, cross-distro install script):

curl -fsSL https://ollama.com/install.sh | sh
# Then pull a model, e.g.:
ollama pull llama3

Note: You can use a local LLM via Ollama or a cloud AI provider. Local models keep data on your machine; cloud models can be stronger but require an API key and careful redaction.

Minimal shell helpers for AI

Pick one of these approaches.

1) Local model via Ollama:

ai() {
  # Usage: echo "Summarize this" | ai
  # Or: ai "Write a one-line summary" < input.txt
  if ! command -v ollama >/dev/null 2>&1; then
    echo "ollama not found. Install it first." >&2
    return 1
  fi
  local model="${AI_MODEL:-llama3}"    # set AI_MODEL to override
  cat | ollama run "$model"
}

2) Cloud-style API (OpenAI-compatible; adapt to your provider):

# Requires: export AI_API_KEY="sk-..." and optionally AI_MODEL and AI_API_URL
# Defaults point to a common OpenAI-compatible path; change for your provider.
ai() {
  : "${AI_API_KEY:?Set AI_API_KEY first}"
  local model="${AI_MODEL:-gpt-4o-mini}"
  local url="${AI_API_URL:-https://api.openai.com/v1/chat/completions}"
  local prompt
  prompt="$(cat)"
  curl -sS "$url" \
    -H "Authorization: Bearer $AI_API_KEY" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg p "$prompt" --arg m "$model" \
      '{model:$m, messages:[{role:"user", content:$p}], temperature:0.2}')" \
  | jq -r '.choices[0].message.content // .error.message'
}

Add your chosen function to ~/.bashrc or ~/.bash_profile and reload your shell:

source ~/.bashrc

5 actionable workflows you can use today

Each example shows classic text tools shaping the data, then AI doing meaning-making. Feel free to tweak prompts to your style.

1) Triage incidents: Compress noisy logs, then summarize root causes

  • Narrow to relevant lines:
journalctl -u myservice --since -30min \
| grep -Ei 'error|failed|timeout|exception' \
| sed -E 's/\[[0-9]{2}:[0-9]{2}:[0-9]{2}\]//g; s/([A-Fa-f0-9]{8,})/[HASH]/g' \
| head -n 500 \
| ai <<'EOF'
Summarize the top recurring issues in these logs.

- Group similar messages together.

- Suggest likely root causes and next diagnostics.

- Output concise bullet points.
EOF

Tips:

  • head -n 500 controls token usage.

  • sed removes volatile timestamps and hashes so AI can group messages better.

2) Turn semi-structured logs into CSV, then ask AI to find hotspots

  • Example: NGINX access logs to CSV:
# Common NGINX combined format: extract status, method, path, and request_time (if logged)
awk '{
  # naive parse: method at $6 (quoted), path at $7, status at $9, bytes at $10
  gsub(/"/,"",$6);
  method=$6; path=$7; status=$9;
  rt=$NF; # if $request_time is last; adapt for your log format
  if (status ~ /^[0-9]+$/) printf "%s,%s,%s,%s\n", status, method, path, rt;
}' /var/log/nginx/access.log > access.csv
  • Ask AI to cluster slow endpoints:
{
  echo "status,method,path,rt"
  head -n 500 access.csv
} | ai <<'EOF'
You are a performance analyst. From this CSV (status,method,path,rt seconds):

- Identify the slowest 10 endpoints by median rt.

- Group endpoints by path patterns (e.g., /api/v1/users/:id).

- Output a table with pattern, count, median rt, 95th percentile rt, and a short recommendation.
EOF

Pre-shaping with awk makes the AI’s job trivial and reliable.

3) Draft precise commit messages from diffs

  • Generate a Conventional Commit message proposal:
git diff --staged \
| sed -E '/^(diff --git|index |@@ |\+\+\+ |--- )/d' \
| head -n 600 \
| ai <<'EOF'
Write a Conventional Commit message (type(scope): subject) with a short body.

- Summarize intent and key changes.

- Mention any breaking changes.

- Keep subject under 72 chars.
EOF
  • Sanity-check, then edit as needed:
git commit -e -m "$(git diff --staged \
  | sed -E '/^(diff --git|index |@@ |\+\+\+ |--- )/d' \
  | head -n 600 \
  | ai \
  | sed -n '1,12p')"

4) Use AI to draft a regex, then harden it with grep/awk tests

  • Ask AI for a PCRE that matches IPv4 addresses in private ranges:
cat <<'EOF' | ai
Write a single-line Perl-compatible regex that matches only private IPv4 addresses (RFC1918).
No anchors; no whitespace; no comments.
EOF
  • Test the candidate:
regex='YOUR_REGEX_FROM_AI'

# Positive cases
printf "%s\n" 10.0.0.1 172.16.5.9 192.168.0.254 \
| grep -P "^$regex$" || echo "Positive test failed"

# Negative cases
printf "%s\n" 8.8.8.8 172.15.1.1 192.169.1.1 \
| grep -Pv "^$regex$" || echo "Negative test failed"

AI gives you a starting point; grep and tests make it production-grade.

5) Redact secrets before sending anything to AI

Never send raw secrets, tokens, or PII upstream. Scrub first, then summarize.

  • Basic redaction filters:
# Emails
sed -E 's/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/[EMAIL]/g' \
# IPv4
| sed -E 's/\b([0-9]{1,3}\.){3}[0-9]{1,3}\b/[IP]/g' \
# UUIDs
| sed -E 's/\b[0-9a-fA-F]{8}-([0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}\b/[UUID]/g' \
# Bearer/API tokens (very rough)
| sed -E 's/(Bearer|Token|Api-?Key)[[:space:]]+[A-Za-z0-9._-]+/\1 [REDACTED]/g'
  • Example pipeline for support tickets:
cat ticket.txt \
| sed -E 's/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/[EMAIL]/g' \
| sed -E 's/\b([0-9]{1,3}\.){3}[0-9]{1,3}\b/[IP]/g' \
| sed -E 's/\b[0-9a-fA-F]{8}-([0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}\b/[UUID]/g' \
| head -n 400 \
| ai <<'EOF'
Summarize the user issue and list 3 likely root causes with one diagnostic step for each.
Keep it under 150 words.
EOF

Practical prompting tips for the shell

  • Be directive: “Output 5 bullet points with action items.” Ambiguity wastes tokens.

  • Constrain format: “Return valid JSON with fields: cause, evidence, next_step.” Then parse with jq.

  • Control size: Use head/tail or awk to pre-aggregate before AI. For example:

cut -d' ' -f5 error.log | sort | uniq -c | sort -nr | head -n 50
  • Keep a library of prompts in files and reuse:
ai < prompts/incident-summary.prompt <(filtered_logs)

Conclusion and next step (CTA)

The terminal already gives you superpowers; pairing grep/awk/sed with AI turns them into time multipliers. Start small:

1) Copy one ai() function into your ~/.bashrc.
2) Try Workflow #1 on your service logs today.
3) Add redaction filters before any cloud calls.

When you see the first 10-minute incident triage shrink to 90 seconds, you’ll never go back. If you want a follow-up post with a reusable Bash toolkit (functions and prompts) for common SRE/data tasks, let me know what you need most.

Related posts