How I Built Transcriber - Local Whisper Voice Backend for Browser AI Tools

Voice input in the browser is a study in inconsistency.

Chrome, Edge, and most Chromium-based browsers have the Web Speech API with continuous recognition that works well for dictation. Firefox ships a stub that throws a not-supported error at runtime. Safari’s implementation works but requires a permission gesture every few sentences. And all of them send your audio to a cloud service.

When I built voice dictation for PromptBoard, I needed something that worked everywhere and ran locally. Transcriber is the answer: a Node.js server that runs Whisper in-process and exposes a single HTTP endpoint. Any browser tool that wants voice input sends audio there and gets text back.

Why a separate server

The first instinct was to embed voice logic directly into each tool. But that means duplicating the Whisper setup, the model download, the ffmpeg conversion, and the CORS configuration in every tool that wants dictation.

A shared local server solves this cleanly. Transcriber runs once. PromptBoard, BugCapture Web, and LearnBoard Web all point to the same http://127.0.0.1:4324/transcribe endpoint. If you upgrade the model or change the language, it takes effect everywhere at once.

The tradeoff is that Transcriber needs to be running for voice to work in non-Chromium browsers. The tools handle this gracefully — if /health returns a non-200 (or times out), they fall back to a manual textarea with audio playback. You can always replay what you said and type it.

The cascade

Every Machina tool that supports voice uses the same three-level cascade:

Level 1 — Web Speech API (Chromium)
If window.SpeechRecognition or window.webkitSpeechRecognition is available, use it. continuous: true, interimResults: false. The model runs in the browser, zero latency, zero network. This is the happy path for Chrome users.

Level 2 — Transcriber (port 4324)
For Firefox and other browsers, MediaRecorder captures audio as a webm or ogg blob. On stop, the blob is POSTed to /transcribe. Transcriber converts it to WAV with ffmpeg, runs it through Whisper, and returns the text. Round-trip is typically 1–3 seconds depending on clip length.

Level 3 — Manual fallback
If Transcriber isn’t running and the Web Speech API isn’t available, a dialog opens with the recorded audio and a textarea. You can play back what you recorded and type the transcription yourself. Not ideal, but nothing is silently lost.

The tool checks Transcriber availability on load with a GET /health call. If it responds { ok: true, ready: true }, Level 2 is available. If the model is still loading, ready is false and the tool waits a few seconds before trying again.

How Transcriber works

The server is ~115 lines of Node.js. No framework, one dependency for Whisper.

Model loading

At startup, Transcriber loads the Whisper model using @xenova/transformers, a JavaScript port of the Hugging Face transformers library:

whisperPipe = await pipeline(
  'automatic-speech-recognition',
  'Xenova/whisper-base',
  { quantized: true }
);

The model downloads on first run (~150MB for whisper-base) and caches to tools/transcriber/models/. Every subsequent start loads from disk in a few seconds. The model is pre-loaded at server startup so the first transcription request doesn’t pay the loading cost.

You can swap the model via env var:

TRANSCRIBER_MODEL=Xenova/whisper-small  # better accuracy, ~460MB
TRANSCRIBER_MODEL=Xenova/whisper-tiny   # faster, ~75MB

Audio conversion

Browsers record audio as WebM (Chrome) or OGG (Firefox). Whisper wants 16kHz mono WAV. Transcriber uses ffmpeg for the conversion:

function toWav(inPath, outPath) {
  return new Promise((resolve, reject) =>
    exec(
      `ffmpeg -y -i "${inPath}" -ar 16000 -ac 1 -f wav "${outPath}" 2>/dev/null`,
      err => err ? reject(new Error('ffmpeg: ' + err.message)) : resolve()
    )
  );
}

ffmpeg is a system dependency — it needs to be installed separately. On most Linux systems it’s sudo apt install ffmpeg. The setup.sh in the Machina root checks for it.

PCM parsing

AudioContext isn’t available in Node.js, so you can’t use the browser’s audio decoding pipeline. Transcriber parses the WAV file directly, walking the RIFF chunk structure to find the data chunk, then converting 16-bit little-endian PCM samples to Float32:

function readWavPCM(wavPath) {
  const buf = readFileSync(wavPath);
  let offset = 12;
  while (offset + 8 <= buf.length) {
    const id   = buf.slice(offset, offset + 4).toString('ascii');
    const size = buf.readUInt32LE(offset + 4);
    if (id === 'data') {
      const nSamples = Math.floor(size / 2);
      const float32  = new Float32Array(nSamples);
      for (let i = 0; i < nSamples; i++)
        float32[i] = buf.readInt16LE(offset + 8 + i * 2) / 32768.0;
      return float32;
    }
    offset += 8 + size + (size & 1);
  }
  throw new Error('"data" chunk not found in WAV file');
}

The resulting Float32Array is what @xenova/transformers expects as input. The pipeline handles tokenization, inference, and decoding internally.

The transcribe endpoint

POST /transcribe
Content-Type: audio/webm (or audio/ogg)
Body: raw audio bytes

The server writes the bytes to a temp file, converts to WAV, runs Whisper, cleans up, and returns:

{ "ok": true, "text": "Fix the checkout form after the last refactor" }

Temp files are always cleaned up in the finally block — even if the pipeline throws, you don’t accumulate .webm and .wav files in /tmp.

Language and model configuration

Three env vars control behavior:

TRANSCRIBER_PORT=4324        # default
TRANSCRIBER_MODEL=Xenova/whisper-base    # default
TRANSCRIBER_LANGUAGE=english # default

For Italian projects (or any other language), set TRANSCRIBER_LANGUAGE=italian. Whisper supports 99 languages. The language setting locks the model to a specific transcription language and slightly improves accuracy compared to auto-detection.

What I learned

The @xenova/transformers library was the right call. The alternative was shelling out to a Python whisper CLI, which would mean another dependency, subprocess management, and slower startup. The JS port runs in-process, and the quantized models are small enough that whisper-base is accurate for technical vocabulary without requiring a GPU.

The hardest part was the WAV parsing. I spent an hour on a bug where transcriptions were consistently garbled before realizing I had the wrong assumption about WAV file structure — specifically that the data chunk always starts at a fixed offset. It doesn’t. You have to walk the chunks. The while (offset + 8 <= buf.length) loop above is the fix.

The three-level cascade turned out to be the right abstraction. Tools don’t need to know which level they’re using — they call the same startVoice() function, which internally picks the best available method. Adding a new input method in the future would mean updating one place, not every tool.

What’s running on port 4324

GET  /health      → { ok, ready, model, language }
POST /transcribe  → { ok, text }  (body: raw audio bytes)
POST /shutdown    → graceful stop

That’s the entire API. Small surface, single responsibility. Transcriber doesn’t manage sessions, doesn’t store audio, doesn’t know which tool is calling it.

---

Transcriber is part of Machina — a free, open-source suite of AI developer tools.
→ View on GitHub