Build Your Own AI Voice Assistant on Linux (with Bash)

Voice assistants are everywhere—but most are black boxes that phone home, lock you to a vendor, and don’t play nicely with your shell. What if you could keep the convenience, cut the creepiness, and wire your assistant straight into Bash?

In this guide, you’ll learn how to stand up a simple, private, offline voice assistant on Linux using open tools. You’ll record audio, transcribe speech to text, run commands or generate answers, and speak responses back out—all from your terminal.

Why this matters:

Privacy and control: Keep audio and transcripts on your machine.
Automation: Drive real system tasks via Bash and CLI tools you already use.
Hackability: Swap parts, add “skills,” and script behaviors in minutes.
Portability: Works across distros with minimal dependencies.

What you’ll build:

A minimal offline pipeline using whisper.cpp (speech-to-text) and espeak-ng or Piper (text-to-speech).
A Bash script that listens, interprets, runs local commands, and replies out loud.
Optional: Plug in a local LLM or custom skills without cloud services.

1) Install prerequisites

You’ll need build tools, audio utilities, and common CLI helpers. Pick the commands for your distro.

Apt (Debian/Ubuntu and derivatives):

sudo apt update
sudo apt install -y git build-essential cmake ffmpeg sox espeak-ng alsa-utils curl unzip

Dnf (Fedora/RHEL family):

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git cmake ffmpeg sox espeak-ng alsa-utils curl unzip

Zypper (openSUSE):

sudo zypper refresh
sudo zypper install -y -t pattern devel_C_C++ git cmake ffmpeg sox espeak-ng alsa-utils curl unzip

Notes:

If your distro’s ffmpeg is in an optional repo (e.g., RPM Fusion on Fedora, Packman on openSUSE), enable that first.
espeak-ng gives you a fast, no-fuss TTS baseline. We’ll also show Piper for higher-quality voices.

2) Set up speech-to-text with whisper.cpp

We’ll compile whisper.cpp once and download a small English model.

# Get the source and build
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
make -j

# Fetch a compact English model (good trade-off of size/speed)
mkdir -p models
curl -L -o models/ggml-base.en.bin \
  "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin?download=true"

Quick test:

# Transcribe a sample file (provided in repo)
./main -m models/ggml-base.en.bin -f samples/jfk.wav -otxt
cat samples/jfk.wav.txt

If you see readable text, your ASR (automatic speech recognition) is ready.

3) Optional: Install higher-quality TTS with Piper

espeak-ng works out of the box, but Piper gives more natural speech.

Install Piper binary and a voice:

# Create a location for Piper and voices
mkdir -p ~/.local/bin ~/.local/share/piper
cd ~/.local/bin

# Download the latest Piper binary for your CPU (example: x86_64)
# Check https://github.com/rhasspy/piper/releases for other architectures (aarch64, etc.)
curl -L -o piper.tar.gz \
  https://github.com/rhasspy/piper/releases/latest/download/piper_linux_x86_64.tar.gz
tar -xzf piper.tar.gz
rm piper.tar.gz
chmod +x piper

# Grab a voice (example: en_US-amy-medium)
cd ~/.local/share/piper
curl -L -o en_US-amy-medium.onnx \
  https://github.com/rhasspy/piper/releases/download/2024.04.15/en_US-amy-medium.onnx
curl -L -o en_US-amy-medium.onnx.json \
  https://github.com/rhasspy/piper/releases/download/2024.04.15/en_US-amy-medium.onnx.json

Test Piper:

echo "Piper is ready to speak" | ~/.local/bin/piper \
  --model ~/.local/share/piper/en_US-amy-medium.onnx \
  --output-raw | aplay -r 22050 -f S16_LE -t raw -

If you prefer to stick with espeak-ng, you can skip Piper and the script will fall back automatically.

4) A minimal offline assistant in one Bash script

This script:

Records 5 seconds of audio from your default ALSA device
Transcribes it with whisper.cpp
Maps text to “skills” (run a command or answer)
Speaks the response via Piper if present, else espeak-ng

Save as voice.sh:

#!/usr/bin/env bash
set -euo pipefail

# Config
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WHISPER_DIR="${ROOT}/whisper.cpp"
MODEL="${WHISPER_DIR}/models/ggml-base.en.bin"
REC_WAV="$(mktemp --suffix=.wav)"
OUT_TXT=""
PIPER_BIN="${HOME}/.local/bin/piper"
PIPER_VOICE="${HOME}/.local/share/piper/en_US-amy-medium.onnx"

cleanup() { rm -f "$REC_WAV" "$OUT_TXT" 2>/dev/null || true; }
trap cleanup EXIT

require() {
  command -v "$1" >/dev/null 2>&1 || { echo "Missing: $1"; exit 1; }
}

# Check deps
[ -x "${WHISPER_DIR}/main" ] || { echo "Build whisper.cpp first (see guide)"; exit 1; }
[ -f "$MODEL" ] || { echo "Missing model at $MODEL"; exit 1; }
require arecord
if ! command -v aplay >/dev/null 2>&1; then echo "Tip: install aplay (alsa-utils) for TTS playback"; fi

# Record 5 seconds of audio at 16 kHz, mono, signed 16-bit little-endian
echo "Speak now (recording 5 seconds)..."
arecord -q -f S16_LE -r 16000 -c 1 -d 5 "$REC_WAV" || {
  echo "Recording failed. Check your mic (arecord -l)."; exit 1;
}

# Transcribe with whisper.cpp (outputs .txt next to file)
echo "Transcribing..."
"${WHISPER_DIR}/main" -m "$MODEL" -f "$REC_WAV" -otxt >/dev/null
OUT_TXT="${REC_WAV}.txt"

# Read and normalize transcript
TRANSCRIPT="$(tr -d '\r' < "$OUT_TXT" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' )"
echo "You said: $TRANSCRIPT"

# Intent/Skill routing (simple examples)
respond() {
  local text="$1" reply=""
  shopt -s nocasematch

  if [[ "$text" =~ (cpu|temperature) ]]; then
    if command -v sensors >/dev/null 2>&1; then
      reply="$(sensors | sed -n '1,10p')"
    else
      reply="Install lm-sensors to read CPU temperature."
    fi

  elif [[ "$text" =~ (ip address|my ip|ip) ]]; then
    reply="$(ip -4 addr | awk '/inet / && $2 !~ /^127/ {print $2}' | paste -sd ', ' -)"
    reply="Your IPv4 addresses are: ${reply:-unknown}."

  elif [[ "$text" =~ (open.*firefox|launch.*firefox) ]]; then
    ( nohup firefox >/dev/null 2>&1 & )
    reply="Opening Firefox."

  elif [[ "$text" =~ (play music|start music) ]]; then
    if command -v vlc >/dev/null 2>&1; then
      ( nohup vlc ~/Music >/dev/null 2>&1 & )
      reply="Playing your music folder in VLC."
    else
      reply="Install VLC or point me to a player."
    fi

  elif [[ "$text" =~ (time|what.*time) ]]; then
    reply="It is $(date '+%H:%M on %A, %B %d')."

  else
    # Optional: use a local LLM if available (e.g., ollama). Otherwise, echo back.
    if command -v ollama >/dev/null 2>&1; then
      reply="$(printf "%s" "$text" | ollama run mistral:latest 2>/dev/null | sed '/^>/d')"
      reply="${reply:-I heard you.}"
    else
      reply="You said: $text"
    fi
  fi

  printf "%s" "$reply"
}

REPLY="$(respond "$TRANSCRIPT")"
echo "Assistant: $REPLY"

# Speak it
if [[ -x "$PIPER_BIN" && -f "$PIPER_VOICE" ]]; then
  printf "%s" "$REPLY" | "$PIPER_BIN" --model "$PIPER_VOICE" --output-raw \
    | aplay -q -r 22050 -f S16_LE -t raw -
else
  espeak-ng -v en "$REPLY" >/dev/null 2>&1 || true
fi

Make it executable and run:

chmod +x voice.sh
./voice.sh

Tips:

If arecord can’t find your mic, list devices with arecord -l and set -D plughw:1,0 (for example).
Reduce background noise using a close mic or add a “silence” gate with sox in a preprocessing step.

5) Add 3–5 practical improvements

Push-to-talk or hotkeys
- Map a keyboard shortcut that runs ./voice.sh so you talk only on demand. This avoids wake-word false triggers and unwanted commands.
Safer command execution
- For any action that changes state (install, delete, power), ask for confirmation. Example: require the phrase “confirm” after the intent is recognized, or run in a “dry-run” mode and read the planned command back first.
Better models, faster inference
- On modern CPUs/GPUs you can use larger Whisper GGML/GGUF models for accuracy, or a GPU build for speed. See whisper.cpp docs for OpenCL/CUDA/Vulkan builds.
Natural speech with Piper
- Try different Piper voices for clarity and speed. Medium models are a good balance. Cache frequent phrases to WAV to cut latency.
Extend skills with Bash
- Add cases for “connect VPN” (nmcli), “what’s my disk space” (df -h), “toggle night light” (gsettings), or “turn on lights” (send MQTT with mosquitto_pub).

Example: quick MQTT skill addition:

elif [[ "$text" =~ (turn on.*office light) ]]; then
  mosquitto_pub -h 192.168.1.10 -t home/office/light -m "on"
  reply="Turning on the office light."

Install mosquitto_pub if needed:

apt: sudo apt install -y mosquitto-clients
dnf: sudo dnf install -y mosquitto-clients
zypper: sudo zypper install -y mosquitto-clients

Real-world examples you can try today

“What’s my IP?” → Reads your current IPs without touching a browser.
“Open Firefox” → Launches your browser hands-free.
“Play music” → Starts VLC on your Music directory.
“What time is it?” → Speaks the current time and date.
Dictation helper
- Replace the reply block with wl-copy or xclip to put recognized text on your clipboard for quick paste into editors or IM:
```
echo -n "$TRANSCRIPT" | wl-copy  # Wayland
# or
echo -n "$TRANSCRIPT" | xclip -selection clipboard  # X11
```

Install clipboard tools:

apt: sudo apt install -y wl-clipboard xclip
dnf: sudo dnf install -y wl-clipboard xclip
zypper: sudo zypper install -y wl-clipboard xclip

Why this approach works

Composable Unix philosophy: Each stage (record → transcribe → decide → speak) is swappable.
Local-first: You control data flow; nothing leaves your box unless you choose.
Distro-friendly: Everything here can be installed with apt, dnf, or zypper plus a couple of standalone binaries.
Extensible: Add skills as shell functions or call out to Python, Node, or Go tools as needed.

Conclusion and next steps

You now have a working offline voice loop that’s private, hackable, and fully scriptable. From here:

Add two new skills you’d actually use daily. Start simple: system status, app launchers, or clipboard dictation.
Improve quality: try a larger Whisper model or a faster build; test multiple Piper voices.
Productionize: wrap it in a systemd user service, add a hotkey, or tie it to a Stream Deck button.
Explore ecosystems: OVOS/Mycroft, Rhasspy, and Home Assistant all integrate well with Linux and can reuse the same ASR/TTS pieces.

If this helped, share your tweaks or skills as a gist and link them in your README. The best assistants are the ones you shape to your workflow—right from your shell.