fix: voice tts format settings (#919)

* feat(voice): add optional speech-to-text input and read-aloud TTS

Adds a push-to-talk mic button in the composer and a read-aloud button on
assistant messages. Both are opt-in and hidden unless a voice backend is
configured via VOICE_SIDECAR_URL.

The auth-gated /api/voice proxy forwards to a configurable backend exposing
/transcribe and /tts (provider-agnostic); the frontend probes /api/voice/health
and hides the controls when disabled. Adds i18n keys and docs/voice.md.

Includes a local, no-API-key reference backend in voice-sidecar/ (faster-whisper
for STT, Kokoro-82M for TTS, both CPU-capable).

* refactor(voice): provider-agnostic backend and in-app config

Switches the voice proxy to the OpenAI audio API (/v1/audio/transcriptions and
/v1/audio/speech) so it works with OpenAI, Groq, or a local server. Adds a
Settings -> Voice tab (base URL, API key, models, voice) plus a Quick Settings
toggle, and removes the bundled Python sidecar.

Review fixes: stop mic tracks on unmount, clear the global TTS stop handler and
revoke leaked blob URLs, add fetch timeouts in the proxy, surface mic errors in
the button, trim before appending transcripts, and drop the repo-wide wav ignore.

* fix(voice): relax backend timeout and surface timeout errors

Bumps the proxy timeout to 5 minutes (VOICE_TIMEOUT_MS) since local TTS can
synthesize long messages at roughly real-time, and returns a clear timed-out
message (504) instead of failing silently. The read-aloud button now shows
backend errors.

* fix(voice): play read-aloud through an app-level player to stop cutoffs

Read-aloud now runs in a single module-level player outside the React tree instead
of per-message component state. Switching chats or re-rendering a message no longer
revokes the blob URL mid-play (the 'Invalid URI' cutoff). Adds content-keyed caching so
re-listening doesn't regenerate, and reuses one audio element (also unlocks iOS once).

* fix(voice): address review (SSRF guard, auth mapping, client timeout)

Validates the user-supplied backend URL (http/https only, blocks the link-local
metadata range) to prevent SSRF; remaps upstream 401/403 so a bad voice API key
isn't read as the app's own auth failing; adds a client-side AbortController timeout
on the read-aloud request so the button can't sit in loading if a request stalls.

* docs(voice): provider-agnostic wording and jsdoc on proxy functions

drop leftover sidecar/faster-whisper references now that the backend is any
openai-compatible voice api, and add jsdoc to the voice-proxy functions so the
docstring coverage check passes.

* fix(voice): harden timeout parsing, tts input check, and player abort

- fall back to the default when VOICE_TIMEOUT_MS is non-numeric or <= 0, so a
  bad override can't make the abort fire immediately
- type-check the tts `text` before calling .trim() so a non-string body returns
  400 instead of throwing
- abort the in-flight TTS fetch on stop() and on a superseding play, so tapping
  read-aloud repeatedly doesn't leave orphaned requests generating audio

* feat(voice): send transcript with the main send button while recording

while dictating, the main send button stops recording, transcribes, and sends
in one tap, matching the codex-style flow. the mic button still stops and drops
the transcript into the input box to edit before sending. voice recording state
is lifted into the composer so both buttons share it, and the send button is
enabled (not grayed) while recording. also fix a pre-existing type error: the
quick-settings preferences map was missing voiceEnabled.

* fix(voice): make stop() idempotent so a double tap can't throw

guard on the recorder's own state instead of react state, so a double tap or
the mic and send buttons both firing won't call stop() on an already-inactive
MediaRecorder.

* fix(voice): expose TTS format in user settings

* fix(voice): harden recording and backend behavior

Redirects could bypass the backend URL guard, and TTS playback waited for full buffering.

Recording could overlap or finish after teardown. Controls also ignored backend readiness.

Explicit formats and config-aware cache keys prevent stale audio after settings change.

* fix(voice): validate config and request boundaries

Malformed stored settings could break voice requests instead of using safe defaults.

Health results could outlive auth changes. URL checks also did not guard the fetch sink.

Remove constant recorder branches so lifecycle cancellation stays clear.

* fix(voice): separate client and server backends

User-selected backend URLs must remain usable without letting clients control server requests.

Call custom providers from the browser while keeping the server proxy bound to its configured host.

This restores voice controls for frontend settings without reopening the SSRF path.

* fix: hide voice options until enabled

---------

Co-authored-by: newsbubbles <nathaniel.gibson@gmail.com>
Co-authored-by: Simos Mikelatos <simosmik@gmail.com>

src/lib/voiceApi.ts

@@ -0,0 +1,60 @@
+import { authenticatedFetch } from '../utils/api';
+import { readVoiceConfig, voiceConfigHeaders } from '../hooks/useVoiceConfig';
+
+function directUrl(baseUrl: string, path: string): string {
+  return `${baseUrl.replace(/\/$/, '')}${path}`;
+}
+
+export function voiceConfigSignature(): string {
+  return JSON.stringify(readVoiceConfig());
+}
+
+export function transcribeVoice(blob: Blob, filename: string): Promise<Response> {
+  const config = readVoiceConfig();
+  const body = new FormData();
+
+  if (config.baseUrl.trim()) {
+    body.append('file', blob, filename);
+    body.append('model', config.sttModel || 'whisper-1');
+    return fetch(directUrl(config.baseUrl.trim(), '/audio/transcriptions'), {
+      method: 'POST',
+      headers: config.apiKey ? { Authorization: `Bearer ${config.apiKey}` } : {},
+      body,
+    });
+  }
+
+  body.append('audio', blob, filename);
+  return authenticatedFetch('/api/voice/transcribe', {
+    method: 'POST',
+    headers: voiceConfigHeaders(),
+    body,
+  });
+}
+
+export function synthesizeVoice(text: string, signal: AbortSignal): Promise<Response> {
+  const config = readVoiceConfig();
+
+  if (config.baseUrl.trim()) {
+    return fetch(directUrl(config.baseUrl.trim(), '/audio/speech'), {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json',
+        ...(config.apiKey ? { Authorization: `Bearer ${config.apiKey}` } : {}),
+      },
+      body: JSON.stringify({
+        model: config.ttsModel || 'tts-1',
+        voice: config.ttsVoice || 'alloy',
+        input: text,
+        ...(config.ttsFormat.trim() ? { response_format: config.ttsFormat.trim() } : {}),
+      }),
+      signal,
+    });
+  }
+
+  return authenticatedFetch('/api/voice/tts', {
+    method: 'POST',
+    body: JSON.stringify({ text }),
+    headers: voiceConfigHeaders(),
+    signal,
+  });
+}

src/lib/voicePlayer.ts

@@ -0,0 +1,196 @@
+import { synthesizeVoice, voiceConfigSignature } from './voiceApi';
+
+// A single app-level audio player for read-aloud. It owns one <audio> element, lives
+// outside the React tree, and caches generated audio by content. Because playback is not
+// tied to a component, switching chats or re-rendering a message can't revoke the blob URL
+// out from under it (the cause of mid-play cutoffs). v1 plays one message at a time
+// (a new play replaces the current one); the design leaves room for a queue later.
+
+export type VoicePlayState = 'idle' | 'loading' | 'playing';
+
+export type VoiceSnapshot = { state: VoicePlayState; error: string | null };
+
+const IDLE: VoiceSnapshot = { state: 'idle', error: null };
+const CACHE_MAX = 24;
+const CLIENT_TIMEOUT_MS = 330000; // backstop; the server proxy already times out at 5 min
+
+// Stable id / cache key from the text and voice settings that affect its audio (djb2).
+export function voiceId(content: string, signature = voiceConfigSignature()): string {
+  const input = JSON.stringify([content, signature]);
+  let h = 5381;
+  for (let i = 0; i < input.length; i++) h = (((h << 5) + h) + input.charCodeAt(i)) | 0;
+  return (h >>> 0).toString(36);
+}
+
+class VoicePlayer {
+  private audio: HTMLAudioElement | null = null;
+  private unlocked = false;
+  private cache = new Map<string, string>(); // id -> blob URL (insertion order = LRU)
+  private currentId: string | null = null;
+  private state: VoicePlayState = 'idle';
+  private errorId: string | null = null;
+  private errorMsg: string | null = null;
+  private token = 0; // bumps to ignore stale in-flight results
+  private activeController: AbortController | null = null; // aborts the in-flight TTS fetch
+  private errorTimer: ReturnType<typeof setTimeout> | null = null;
+  private listeners = new Set<() => void>();
+
+  subscribe(listener: () => void): () => void {
+    this.listeners.add(listener);
+    return () => {
+      this.listeners.delete(listener);
+    };
+  }
+
+  private emit() {
+    this.listeners.forEach((l) => l());
+  }
+
+  getSnapshot(id: string): VoiceSnapshot {
+    const state = this.currentId === id ? this.state : 'idle';
+    const error = this.errorId === id ? this.errorMsg : null;
+    if (state === 'idle' && error === null) return IDLE;
+    return { state, error };
+  }
+
+  private ensureAudio(): HTMLAudioElement {
+    if (!this.audio) {
+      const audio = new Audio();
+      audio.addEventListener('ended', () => this.onEnded());
+      audio.addEventListener('error', () => {
+        // Only meaningful while we believe we're playing.
+        if (this.state === 'playing') this.onEnded();
+      });
+      this.audio = audio;
+    }
+    return this.audio;
+  }
+
+  // Call synchronously from the click handler so iOS grants the (reused) element playback.
+  unlock() {
+    if (this.unlocked) return;
+    const audio = this.ensureAudio();
+    try {
+      const p = audio.play();
+      if (p && typeof p.catch === 'function') p.catch(() => {});
+      audio.pause();
+    } catch {
+      /* priming attempt; ignore */
+    }
+    this.unlocked = true;
+  }
+
+  toggle(content: string) {
+    const id = voiceId(content);
+    if (this.currentId === id && (this.state === 'playing' || this.state === 'loading')) {
+      this.stop();
+      return;
+    }
+    void this.play(id, content);
+  }
+
+  stop() {
+    this.token++; // ignore any stale in-flight result
+    this.abortActive(); // and actually cancel the network request
+    if (this.audio) this.audio.pause();
+    this.state = 'idle';
+    this.currentId = null;
+    this.emit();
+  }
+
+  private abortActive() {
+    if (this.activeController) {
+      this.activeController.abort();
+      this.activeController = null;
+    }
+  }
+
+  private onEnded() {
+    this.state = 'idle';
+    this.currentId = null;
+    this.emit();
+    // (queue auto-advance would hook in here)
+  }
+
+  private setError(id: string, msg: string) {
+    this.state = 'idle';
+    this.currentId = id;
+    this.errorId = id;
+    this.errorMsg = msg;
+    this.emit();
+    if (this.errorTimer) clearTimeout(this.errorTimer);
+    this.errorTimer = setTimeout(() => {
+      if (this.errorId === id) {
+        this.errorId = null;
+        this.errorMsg = null;
+        if (this.currentId === id) this.currentId = null;
+        this.emit();
+      }
+    }, 6000);
+  }
+
+  private async play(id: string, content: string) {
+    const audio = this.ensureAudio();
+    audio.pause();
+    this.currentId = id;
+    this.errorId = null;
+    this.errorMsg = null;
+    this.state = 'loading';
+    this.emit();
+
+    const myToken = ++this.token;
+    this.abortActive(); // cancel any request this play supersedes
+
+    try {
+      let url = this.cache.get(id);
+      if (!url) {
+        const controller = new AbortController();
+        this.activeController = controller;
+        const timer = setTimeout(() => controller.abort(), CLIENT_TIMEOUT_MS);
+        const res = await synthesizeVoice(content, controller.signal).finally(() => {
+          clearTimeout(timer);
+          if (this.activeController === controller) this.activeController = null;
+        });
+        if (myToken !== this.token) return; // superseded by another play/stop
+        if (!res.ok) {
+          let msg = `Read-aloud failed (${res.status})`;
+          try {
+            const j = await res.json();
+            if (j?.error) msg = String(j.error);
+          } catch {
+            /* non-JSON error body */
+          }
+          throw new Error(msg);
+        }
+        const blob = await res.blob();
+        if (myToken !== this.token) return;
+        url = URL.createObjectURL(blob);
+        this.cacheSet(id, url);
+      }
+      if (myToken !== this.token) return;
+      audio.src = url;
+      audio.load();
+      await audio.play();
+      if (myToken !== this.token) return;
+      this.state = 'playing';
+      this.emit();
+    } catch (e) {
+      if (myToken !== this.token) return;
+      const aborted = e instanceof Error && e.name === 'AbortError';
+      this.setError(id, aborted ? 'Read-aloud timed out.' : e instanceof Error ? e.message : 'Read-aloud failed');
+    }
+  }
+
+  private cacheSet(id: string, url: string) {
+    this.cache.set(id, url);
+    while (this.cache.size > CACHE_MAX) {
+      const oldest = this.cache.keys().next().value as string | undefined;
+      if (oldest === undefined) break;
+      const oldUrl = this.cache.get(oldest);
+      this.cache.delete(oldest);
+      if (oldUrl && oldUrl !== this.audio?.src) URL.revokeObjectURL(oldUrl);
+    }
+  }
+}
+
+export const voicePlayer = new VoicePlayer();