Compare commits

..

14 Commits

Author SHA1 Message Date
Haileyesus
a187a21ef2 feat(shell): add Ctrl+C button to mobile terminal shortcuts
Add a Ctrl+C key to the right end of the mobile shortcuts bar so users
can interrupt the foreground process without a physical keyboard. Sends
\x03 (ETX), which is the same interrupt sequence on Windows and Linux.
2026-06-27 17:41:22 +03:00
Haileyesus
7d925e95ab fix(shell): add inertial scrolling to mobile terminal
xterm scrolls the viewport 1:1 with the finger and prevents native
scrolling, so the terminal stopped the instant the finger lifted with
no momentum — making it tedious to scroll long output on mobile.

Track finger velocity during a one-finger drag and, on release, keep
scrolling the viewport with friction until it slows to a stop or hits
the buffer bounds. Inertia is cancelled on a new touch, selection,
pinch, or dispose so it never fights the user. Coasting behaviour is
tunable via the SCROLL_INERTIA_* constants.
2026-06-27 17:31:54 +03:00
Haileyesus
fb06a0ce20 fix(shell): resize terminal when mobile keyboard opens
On Chrome/Android the on-screen keyboard overlays content by default
(interactive-widget=resizes-visual), so the layout viewport kept its
full height when the keyboard appeared. The terminal stayed at its
pre-keyboard size, leaving dead space above the keyboard and a stale
scroll geometry.

Set interactive-widget=resizes-content so Android shrinks the layout
viewport on keyboard open. The fixed inve
the keyboard and the existing ResizeObserver re-fits xterm to the new
height. iOS is unaffected (interactive-widget is ignored there; the
VisualViewport --keyboard-height path still applies).
2026-06-27 17:07:22 +03:00
Haileyesus
da27dd93db fix(app): prevent mobile document overscroll 2026-06-26 19:33:05 +03:00
Haileyesus
5459d7c60e fix(shell): add copy/select-all buttons and zoom in and out 2026-06-26 19:06:09 +03:00
Simos Mikelatos
645914f2c8 Merge branch 'main' into fix/mobile-shell-selection-auth-flow 2026-06-26 16:06:59 +02:00
Haile
591e8e7642 fix: voice tts format settings (#919)
* feat(voice): add optional speech-to-text input and read-aloud TTS

Adds a push-to-talk mic button in the composer and a read-aloud button on
assistant messages. Both are opt-in and hidden unless a voice backend is
configured via VOICE_SIDECAR_URL.

The auth-gated /api/voice proxy forwards to a configurable backend exposing
/transcribe and /tts (provider-agnostic); the frontend probes /api/voice/health
and hides the controls when disabled. Adds i18n keys and docs/voice.md.

Includes a local, no-API-key reference backend in voice-sidecar/ (faster-whisper
for STT, Kokoro-82M for TTS, both CPU-capable).

* refactor(voice): provider-agnostic backend and in-app config

Switches the voice proxy to the OpenAI audio API (/v1/audio/transcriptions and
/v1/audio/speech) so it works with OpenAI, Groq, or a local server. Adds a
Settings -> Voice tab (base URL, API key, models, voice) plus a Quick Settings
toggle, and removes the bundled Python sidecar.

Review fixes: stop mic tracks on unmount, clear the global TTS stop handler and
revoke leaked blob URLs, add fetch timeouts in the proxy, surface mic errors in
the button, trim before appending transcripts, and drop the repo-wide wav ignore.

* fix(voice): relax backend timeout and surface timeout errors

Bumps the proxy timeout to 5 minutes (VOICE_TIMEOUT_MS) since local TTS can
synthesize long messages at roughly real-time, and returns a clear timed-out
message (504) instead of failing silently. The read-aloud button now shows
backend errors.

* fix(voice): play read-aloud through an app-level player to stop cutoffs

Read-aloud now runs in a single module-level player outside the React tree instead
of per-message component state. Switching chats or re-rendering a message no longer
revokes the blob URL mid-play (the 'Invalid URI' cutoff). Adds content-keyed caching so
re-listening doesn't regenerate, and reuses one audio element (also unlocks iOS once).

* fix(voice): address review (SSRF guard, auth mapping, client timeout)

Validates the user-supplied backend URL (http/https only, blocks the link-local
metadata range) to prevent SSRF; remaps upstream 401/403 so a bad voice API key
isn't read as the app's own auth failing; adds a client-side AbortController timeout
on the read-aloud request so the button can't sit in loading if a request stalls.

* docs(voice): provider-agnostic wording and jsdoc on proxy functions

drop leftover sidecar/faster-whisper references now that the backend is any
openai-compatible voice api, and add jsdoc to the voice-proxy functions so the
docstring coverage check passes.

* fix(voice): harden timeout parsing, tts input check, and player abort

- fall back to the default when VOICE_TIMEOUT_MS is non-numeric or <= 0, so a
  bad override can't make the abort fire immediately
- type-check the tts `text` before calling .trim() so a non-string body returns
  400 instead of throwing
- abort the in-flight TTS fetch on stop() and on a superseding play, so tapping
  read-aloud repeatedly doesn't leave orphaned requests generating audio

* feat(voice): send transcript with the main send button while recording

while dictating, the main send button stops recording, transcribes, and sends
in one tap, matching the codex-style flow. the mic button still stops and drops
the transcript into the input box to edit before sending. voice recording state
is lifted into the composer so both buttons share it, and the send button is
enabled (not grayed) while recording. also fix a pre-existing type error: the
quick-settings preferences map was missing voiceEnabled.

* fix(voice): make stop() idempotent so a double tap can't throw

guard on the recorder's own state instead of react state, so a double tap or
the mic and send buttons both firing won't call stop() on an already-inactive
MediaRecorder.

* fix(voice): expose TTS format in user settings

* fix(voice): harden recording and backend behavior

Redirects could bypass the backend URL guard, and TTS playback waited for full buffering.

Recording could overlap or finish after teardown. Controls also ignored backend readiness.

Explicit formats and config-aware cache keys prevent stale audio after settings change.

* fix(voice): validate config and request boundaries

Malformed stored settings could break voice requests instead of using safe defaults.

Health results could outlive auth changes. URL checks also did not guard the fetch sink.

Remove constant recorder branches so lifecycle cancellation stays clear.

* fix(voice): separate client and server backends

User-selected backend URLs must remain usable without letting clients control server requests.

Call custom providers from the browser while keeping the server proxy bound to its configured host.

This restores voice controls for frontend settings without reopening the SSRF path.

* fix: hide voice options until enabled

---------

Co-authored-by: newsbubbles <nathaniel.gibson@gmail.com>
Co-authored-by: Simos Mikelatos <simosmik@gmail.com>
2026-06-26 16:06:40 +02:00
Haileyesus
c7a0891f56 fix(shell): remove mobile auth url controls 2026-06-26 15:51:10 +03:00
Haileyesus
241ed1da54 fix(shell): support mobile terminal text selection 2026-06-26 15:38:11 +03:00
Haileyesus
ee002fc3f7 fix(shell): keep c input during auth URL flow 2026-06-26 14:25:55 +03:00
Haile
c947eaaee5 feat: play sound for pending tool requests (#918) 2026-06-25 14:57:10 +02:00
Haile
4a503b1dc8 fix(shell): prioritize user npm binaries (#913)
Interactive shells could resolve bundled or system CLIs before user-installed npm binaries.

Move existing user npm global directories to the front of PATH while preserving all other entries.

Co-authored-by: Simos Mikelatos <simosmik@gmail.com>
2026-06-24 20:15:52 +02:00
Koya Kikuchi
f6326c8082 feat(version): warn when the server was updated but not restarted (#898)
When the package is updated on disk but the long-lived server process is
not restarted, the new frontend bundle (served from disk) talks to the
old running backend. New DB-backed features then fail silently — e.g.
deleting/archiving a session appears to do nothing — because the new
schema/routes only take effect on restart.

Nothing currently detects this skew: useVersionCheck only compares the
frontend's build-time version against the latest GitHub release.

This exposes the running server's version (captured once at startup) via
/health, compares it to the frontend's build-time version in
useVersionCheck, and shows a "restart required" banner in the sidebar
(and a small indicator in the collapsed sidebar) when they differ.

- server: add `version` (RUNNING_VERSION, read once at startup) to /health
- useVersionCheck: return `restartRequired` / `runningVersion`
- SidebarFooter / SidebarCollapsed: surface a restart-required banner
- i18n: add `version.restartRequired` to all 10 sidebar locales

Verified with `tsc --noEmit` (client + server) and eslint.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Simos Mikelatos <simosmik@gmail.com>
2026-06-22 22:49:57 +02:00
Haile
c5fe127958 feat(skills): add provider skill management (#909)
* feat(skills): add provider skill management

Users need one settings surface to discover and install skills without manually navigating provider-specific directories.

Add provider-backed global skill installation for Claude, Codex, Gemini, and Cursor, while keeping OpenCode read-only because it reuses other providers' skill locations.

Add a responsive Skills settings tab with scoped discovery, search, refresh controls, markdown and folder uploads, upload feedback, and overflow-safe layouts.

Validate bundled skill files and paths before writing them, preserve scripts and assets, and cover provider discovery and installation behavior with tests.

* fix(skills): preserve uploaded skill folders

Folder drops discarded supporting scripts and assets.

Keep relative paths and upload every file from the selected skill folder.

Use the selected folder name for installation and cover it in provider tests.

* fix(skills): restrict standalone skill uploads

Only show Markdown files when selecting standalone skills.

Normalize browser file paths so SKILL.md is not mistaken for a folder named dot.

* fix(skills): validate installs before writing

Preserve bundled files and normalize fallback names across skill installation paths.

Validate complete batches before writing and reject existing targets to avoid partial installs.

Keep project metadata and make folder selection tolerant of casing and cancelled dialogs.

* fix(skills): overwrite existing installations

Replace an existing skill directory instead of rejecting a duplicate installation.

Remove stale supporting files so the installed directory exactly matches the new upload.
2026-06-22 22:45:27 +02:00
56 changed files with 2424 additions and 277 deletions

View File

@@ -4,7 +4,7 @@
<meta charset="UTF-8" /> <meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" /> <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<link rel="icon" type="image/png" href="/favicon.png" /> <link rel="icon" type="image/png" href="/favicon.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0, user-scalable=no, viewport-fit=cover" /> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0, user-scalable=no, viewport-fit=cover, interactive-widget=resizes-content" />
<title>CloudCLI UI</title> <title>CloudCLI UI</title>
<!-- PWA Manifest --> <!-- PWA Manifest -->

View File

@@ -61,6 +61,7 @@ import userRoutes from './routes/user.js';
import geminiRoutes from './routes/gemini.js'; import geminiRoutes from './routes/gemini.js';
import pluginsRoutes from './routes/plugins.js'; import pluginsRoutes from './routes/plugins.js';
import providerRoutes from './modules/providers/provider.routes.js'; import providerRoutes from './modules/providers/provider.routes.js';
import voiceRoutes from './voice-proxy.js';
import browserUseRoutes from './modules/browser-use/browser-use.routes.js'; import browserUseRoutes from './modules/browser-use/browser-use.routes.js';
import browserUseMcpRoutes from './modules/browser-use/browser-use-mcp.routes.js'; import browserUseMcpRoutes from './modules/browser-use/browser-use-mcp.routes.js';
import { browserUseService } from './modules/browser-use/browser-use.service.js'; import { browserUseService } from './modules/browser-use/browser-use.service.js';
@@ -76,6 +77,19 @@ const __dirname = getModuleDir(import.meta.url);
// Resolving the app root once keeps every repo-level lookup below aligned across both layouts. // Resolving the app root once keeps every repo-level lookup below aligned across both layouts.
const APP_ROOT = findAppRoot(__dirname); const APP_ROOT = findAppRoot(__dirname);
const installMode = fs.existsSync(path.join(APP_ROOT, '.git')) ? 'git' : 'npm'; const installMode = fs.existsSync(path.join(APP_ROOT, '.git')) ? 'git' : 'npm';
// Version of the code that is actually running, captured once at process
// startup. This intentionally does NOT re-read package.json per request: after
// an update replaces the files on disk, package.json reflects the NEW version
// while this long-lived process still runs the OLD code. The frontend bundle is
// rebuilt on update, so a mismatch between this value and the frontend's
// build-time version means the server was updated but not restarted.
const RUNNING_VERSION = (() => {
try {
return JSON.parse(fs.readFileSync(path.join(APP_ROOT, 'package.json'), 'utf8')).version || null;
} catch {
return null;
}
})();
const MAX_FILE_UPLOAD_SIZE_MB = 200; const MAX_FILE_UPLOAD_SIZE_MB = 200;
const MAX_FILE_UPLOAD_SIZE_BYTES = MAX_FILE_UPLOAD_SIZE_MB * 1024 * 1024; const MAX_FILE_UPLOAD_SIZE_BYTES = MAX_FILE_UPLOAD_SIZE_MB * 1024 * 1024;
const MAX_FILE_UPLOAD_COUNT = 20; const MAX_FILE_UPLOAD_COUNT = 20;
@@ -156,7 +170,8 @@ app.get('/health', (req, res) => {
res.json({ res.json({
status: 'ok', status: 'ok',
timestamp: new Date().toISOString(), timestamp: new Date().toISOString(),
installMode installMode,
version: RUNNING_VERSION
}); });
}); });
@@ -208,6 +223,8 @@ app.use('/api/providers', authenticateToken, providerRoutes);
// Agent API Routes (uses API key authentication) // Agent API Routes (uses API key authentication)
app.use('/api/agent', agentRoutes); app.use('/api/agent', agentRoutes);
app.use('/api/voice', authenticateToken, voiceRoutes);
// Serve public files (like api-docs.html) // Serve public files (like api-docs.html)
app.use(express.static(path.join(APP_ROOT, 'public'))); app.use(express.static(path.join(APP_ROOT, 'public')));

View File

@@ -171,6 +171,62 @@ function buildShellCommand(
return command; return command;
} }
function readEnvValue(env: NodeJS.ProcessEnv, key: string): string | undefined {
const resolvedKey = Object.keys(env).find((envKey) => envKey.toLowerCase() === key.toLowerCase());
return resolvedKey ? env[resolvedKey] : undefined;
}
function getPathEnvKey(env: NodeJS.ProcessEnv): string {
return Object.keys(env).find((key) => key.toLowerCase() === 'path') || 'PATH';
}
function prioritizeUserNpmGlobalBin(env: NodeJS.ProcessEnv): { key: string; value: string | undefined } {
const pathKey = getPathEnvKey(env);
const currentPath = env[pathKey];
if (!currentPath) {
return { key: pathKey, value: currentPath };
}
const delimiter = path.delimiter;
const pathEntries = currentPath.split(delimiter).filter(Boolean);
const npmPrefix = readEnvValue(env, 'npm_config_prefix');
const appData = readEnvValue(env, 'APPDATA');
const candidates = [
npmPrefix || '',
npmPrefix ? path.join(npmPrefix, 'bin') : '',
appData ? path.join(appData, 'npm') : '',
path.join(os.homedir(), 'AppData', 'Roaming', 'npm'),
path.join(os.homedir(), '.npm-global', 'bin'),
].filter(Boolean);
const normalizedPathEntries = pathEntries.map((entry) => os.platform() === 'win32' ? entry.toLowerCase() : entry);
const preferredEntries = candidates.filter((candidate, index) => {
const normalizedCandidate = os.platform() === 'win32' ? candidate.toLowerCase() : candidate;
return (
candidates.indexOf(candidate) === index &&
normalizedPathEntries.includes(normalizedCandidate)
);
});
if (preferredEntries.length === 0) {
return { key: pathKey, value: currentPath };
}
const normalizedPreferredEntries = preferredEntries.map((entry) =>
os.platform() === 'win32' ? entry.toLowerCase() : entry
);
const value = [
...preferredEntries,
...pathEntries.filter((entry) => {
const normalizedEntry = os.platform() === 'win32' ? entry.toLowerCase() : entry;
return !normalizedPreferredEntries.includes(normalizedEntry);
}),
].join(delimiter);
return { key: pathKey, value };
}
/** /**
* Handles websocket connections used by the standalone shell terminal UI. * Handles websocket connections used by the standalone shell terminal UI.
*/ */
@@ -284,6 +340,7 @@ export function handleShellConnection(
os.platform() === 'win32' ? ['-Command', shellCommand] : ['-c', shellCommand]; os.platform() === 'win32' ? ['-Command', shellCommand] : ['-c', shellCommand];
const termCols = readNumber(data.cols, 80); const termCols = readNumber(data.cols, 80);
const termRows = readNumber(data.rows, 24); const termRows = readNumber(data.rows, 24);
const prioritizedPath = prioritizeUserNpmGlobalBin(process.env);
shellProcess = pty.spawn(shell, shellArgs, { shellProcess = pty.spawn(shell, shellArgs, {
name: 'xterm-256color', name: 'xterm-256color',
@@ -292,6 +349,7 @@ export function handleShellConnection(
cwd: resolvedProjectPath, cwd: resolvedProjectPath,
env: { env: {
...process.env, ...process.env,
[prioritizedPath.key]: prioritizedPath.value,
TERM: 'xterm-256color', TERM: 'xterm-256color',
COLORTERM: 'truecolor', COLORTERM: 'truecolor',
FORCE_COLOR: '3', FORCE_COLOR: '3',

224
server/voice-proxy.js Normal file
View File

@@ -0,0 +1,224 @@
// Optional voice proxy — forwards STT/TTS to an OpenAI-compatible audio backend.
//
// The backend is whatever the user points at: OpenAI, Groq, or a local server
// (LocalAI / Speaches / Kokoro-FastAPI / openedai-speech / etc.). It must expose the
// standard OpenAI audio endpoints:
// POST {base}/audio/transcriptions (multipart 'file' + 'model') -> { text }
// POST {base}/audio/speech ({ model, voice, input }) -> audio bytes
//
// Config is resolved per-request from headers (set by the client's voice settings),
// falling back to server env defaults. Mounted at /api/voice behind authenticateToken.
import { Readable } from 'node:stream';
import express from 'express';
const ENV = {
baseUrl: (process.env.VOICE_API_BASE_URL || '').replace(/\/$/, ''),
apiKey: process.env.VOICE_API_KEY || '',
sttModel: process.env.VOICE_STT_MODEL || 'whisper-1',
ttsModel: process.env.VOICE_TTS_MODEL || 'tts-1',
ttsVoice: process.env.VOICE_TTS_VOICE || 'alloy',
};
/**
* Resolve the voice backend config for a request. Client headers (set from the
* user's in-app voice settings) take precedence over the server env defaults.
* @param {import('express').Request} req
* @returns {{baseUrl: string, apiKey: string, sttModel: string, ttsModel: string, ttsVoice: string, ttsFormat: string}}
*/
function resolveConfig(req) {
const h = req.headers;
return {
// Security: do not allow clients to control the outbound backend host.
// Always use the server-side configured base URL.
baseUrl: ENV.baseUrl,
apiKey: String(h['x-voice-api-key'] || '') || ENV.apiKey,
sttModel: String(h['x-voice-stt-model'] || '') || ENV.sttModel,
ttsModel: String(h['x-voice-tts-model'] || '') || ENV.ttsModel,
ttsVoice: String(h['x-voice-tts-voice'] || '') || ENV.ttsVoice,
ttsFormat: String(h['x-voice-tts-format'] || '').trim(),
};
}
const router = express.Router();
// Generous by default — local TTS can synthesize long messages at ~real-time on CPU.
// Guard against a non-numeric/zero override that would make setTimeout fire immediately.
const DEFAULT_VOICE_TIMEOUT_MS = 300000;
const _parsedTimeout = Number(process.env.VOICE_TIMEOUT_MS);
const VOICE_TIMEOUT_MS = Number.isFinite(_parsedTimeout) && _parsedTimeout > 0
? _parsedTimeout
: DEFAULT_VOICE_TIMEOUT_MS;
/**
* fetch() with an AbortController timeout so a stalled backend can't hold the
* request open indefinitely. Aborts after VOICE_TIMEOUT_MS.
* @param {string} url
* @param {RequestInit} [options]
* @returns {Promise<Response>}
*/
async function fetchWithTimeout(url, options = {}) {
const parsed = new URL(url);
if (!['http:', 'https:'].includes(parsed.protocol) || !isAllowedBackendUrl(parsed.origin)) {
throw new Error('Blocked outbound voice backend URL');
}
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), VOICE_TIMEOUT_MS);
try {
return await fetch(parsed.toString(), { redirect: 'manual', ...options, signal: controller.signal });
} finally {
clearTimeout(timer);
}
}
/**
* Turn a backend fetch failure into a clear, actionable client response:
* 504 on timeout (AbortError), 502 otherwise.
* @param {import('express').Response} res
* @param {Error} e
*/
function backendError(res, e) {
if (e && e.name === 'AbortError') {
return res.status(504).json({
error: `Voice backend timed out after ${Math.round(VOICE_TIMEOUT_MS / 1000)}s. Check your voice backend.`,
});
}
return res.status(502).json({ error: `Voice backend unreachable: ${e.message}` });
}
/**
* SSRF guard for the user-configurable backend URL: allow http/https only and
* block the link-local / cloud-metadata range (169.254.x). localhost and private
* ranges are allowed on purpose so users can point at a local voice server
* (LocalAI, Speaches, Kokoro-FastAPI, etc.).
* @param {string} raw
* @returns {boolean}
*/
function isAllowedBackendUrl(raw) {
let u;
try {
u = new URL(raw);
} catch {
return false;
}
if (u.protocol !== 'http:' && u.protocol !== 'https:') return false;
if (u.hostname === '169.254.169.254' || u.hostname.startsWith('169.254.')) return false;
return true;
}
/**
* Relay an upstream (backend) error to the client without making an upstream
* 401/403 look like the user's own app login failed.
* @param {import('express').Response} res
* @param {number} status
* @param {string} [text]
*/
function upstreamError(res, status, text) {
if (status === 401 || status === 403) {
return res.status(502).json({ error: 'Voice backend rejected the request (check the API key).' });
}
return res.status(status).json({ error: text || 'voice backend error' });
}
let _upload = null;
/**
* Lazily build a memory-storage multer instance (25 MB cap) for audio uploads,
* so multer is only imported when the voice feature is actually used.
* @returns {Promise<import('multer').Multer>}
*/
async function getUpload() {
if (!_upload) {
const multer = (await import('multer')).default;
_upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: 25 * 1024 * 1024 } });
}
return _upload;
}
/**
* Build the Authorization header for the backend, or an empty object when no
* key is configured (e.g. a local server that needs none).
* @param {string} apiKey
* @returns {Record<string, string>}
*/
function authHeader(apiKey) {
return apiKey ? { Authorization: `Bearer ${apiKey}` } : {};
}
/**
* GET /api/voice/health -> { configured } (true when a backend base URL is set).
*/
router.get('/health', (req, res) => {
res.json({ configured: Boolean(resolveConfig(req).baseUrl) });
});
/**
* POST /api/voice/transcribe (multipart 'audio') -> { text }.
* Forwards the uploaded audio to the backend's /audio/transcriptions endpoint.
*/
router.post('/transcribe', async (req, res) => {
const cfg = resolveConfig(req);
if (!cfg.baseUrl) return res.status(503).json({ error: 'No voice backend configured' });
if (!isAllowedBackendUrl(cfg.baseUrl)) return res.status(400).json({ error: 'Invalid voice backend URL.' });
const upload = await getUpload();
upload.single('audio')(req, res, async (err) => {
if (err) return res.status(400).json({ error: err.message });
if (!req.file) return res.status(400).json({ error: 'No audio uploaded' });
try {
const fd = new FormData();
fd.append(
'file',
new Blob([req.file.buffer], { type: req.file.mimetype || 'audio/webm' }),
req.file.originalname || 'recording.webm',
);
fd.append('model', cfg.sttModel);
const r = await fetchWithTimeout(`${cfg.baseUrl}/audio/transcriptions`, {
method: 'POST',
headers: authHeader(cfg.apiKey),
body: fd,
});
const text = await r.text();
if (!r.ok) return upstreamError(res, r.status, text);
let data;
try { data = JSON.parse(text); } catch { data = { text }; }
res.json({ text: data.text ?? '' });
} catch (e) {
backendError(res, e);
}
});
});
/**
* POST /api/voice/tts { text } -> audio bytes.
* Forwards the text to the backend's /audio/speech endpoint and streams the audio back.
*/
router.post('/tts', async (req, res) => {
const cfg = resolveConfig(req);
if (!cfg.baseUrl) return res.status(503).json({ error: 'No voice backend configured' });
if (!isAllowedBackendUrl(cfg.baseUrl)) return res.status(400).json({ error: 'Invalid voice backend URL.' });
const text = req.body?.text;
if (typeof text !== 'string' || !text.trim()) return res.status(400).json({ error: 'text required' });
try {
const r = await fetchWithTimeout(`${cfg.baseUrl}/audio/speech`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', ...authHeader(cfg.apiKey) },
body: JSON.stringify({
model: cfg.ttsModel,
voice: cfg.ttsVoice,
input: text,
...(cfg.ttsFormat ? { response_format: cfg.ttsFormat } : {}),
}),
});
if (!r.ok) {
const errText = await r.text().catch(() => 'tts failed');
return upstreamError(res, r.status, errText);
}
res.setHeader('Content-Type', r.headers.get('content-type') || 'audio/mpeg');
res.setHeader('Cache-Control', 'no-store');
if (!r.body) return res.end();
Readable.fromWeb(r.body).on('error', (error) => res.destroy(error)).pipe(res);
} catch (e) {
backendError(res, e);
}
});
export default router;

View File

@@ -775,6 +775,17 @@ export function useChatComposerState({
handleSubmitRef.current = handleSubmit; handleSubmitRef.current = handleSubmit;
}, [handleSubmit]); }, [handleSubmit]);
// A voice transcript either fills the input (to edit before sending) or, when the
// user tapped "stop and send", is submitted straight away. Mirror the value into
// inputValueRef synchronously so handleSubmit reads the new text, not the stale state.
const handleVoiceTranscript = useCallback((text: string, send?: boolean) => {
const base = inputValueRef.current.trim();
const next = base ? `${base} ${text}` : text;
setInput(next);
inputValueRef.current = next;
if (send) handleSubmitRef.current?.(createFakeSubmitEvent());
}, [setInput]);
useEffect(() => { useEffect(() => {
inputValueRef.current = input; inputValueRef.current = input;
}, [input]); }, [input]);
@@ -1013,6 +1024,7 @@ export function useChatComposerState({
isDragActive, isDragActive,
openImagePicker: open, openImagePicker: open,
handleSubmit, handleSubmit,
handleVoiceTranscript,
handleInputChange, handleInputChange,
handleKeyDown, handleKeyDown,
handlePaste, handlePaste,

View File

@@ -114,7 +114,6 @@ export function useChatProviderState({ selectedSession, selectedProject }: UseCh
const [providerModelsLoading, setProviderModelsLoading] = useState(true); const [providerModelsLoading, setProviderModelsLoading] = useState(true);
const [providerModelsRefreshing, setProviderModelsRefreshing] = useState(false); const [providerModelsRefreshing, setProviderModelsRefreshing] = useState(false);
const lastProviderRef = useRef(provider);
const providerModelsRequestIdRef = useRef(0); const providerModelsRequestIdRef = useRef(0);
const setStoredProviderModel = useCallback((targetProvider: LLMProvider, model: string) => { const setStoredProviderModel = useCallback((targetProvider: LLMProvider, model: string) => {
@@ -344,14 +343,8 @@ export function useChatProviderState({ selectedSession, selectedProject }: UseCh
localStorage.setItem('selected-provider', selectedSession.__provider); localStorage.setItem('selected-provider', selectedSession.__provider);
}, [provider, selectedSession]); }, [provider, selectedSession]);
useEffect(() => { // Permission prompts belong to a session, not to the transient provider
if (lastProviderRef.current === provider) { // selection that is synchronized after navigation.
return;
}
setPendingPermissionRequests([]);
lastProviderRef.current = provider;
}, [provider]);
useEffect(() => { useEffect(() => {
setPendingPermissionRequests((previous) => setPendingPermissionRequests((previous) =>
previous.filter((request) => !request.sessionId || request.sessionId === selectedSession?.id), previous.filter((request) => !request.sessionId || request.sessionId === selectedSession?.id),

View File

@@ -1,20 +1,29 @@
import { useEffect } from 'react'; import { useEffect, useRef } from 'react';
import type { Dispatch, MutableRefObject, SetStateAction } from 'react'; import type { Dispatch, MutableRefObject, SetStateAction } from 'react';
import type { ServerEvent } from '../../../contexts/WebSocketContext'; import type { ServerEvent } from '../../../contexts/WebSocketContext';
import { showCompletionTitleIndicator } from '../../../utils/pageTitleNotification'; import { showCompletionTitleIndicator } from '../../../utils/pageTitleNotification';
import { playChatCompletionSound } from '../../../utils/notificationSound'; import { playChatCompletionSound, playNotificationSound } from '../../../utils/notificationSound';
import type { MarkSessionIdle, MarkSessionProcessing } from '../../../hooks/useSessionProtection'; import type { MarkSessionIdle, MarkSessionProcessing } from '../../../hooks/useSessionProtection';
import type { PendingPermissionRequest } from '../types/types'; import type { PendingPermissionRequest } from '../types/types';
import type { ProjectSession, LLMProvider } from '../../../types/app'; import type { ProjectSession, LLMProvider } from '../../../types/app';
import type { SessionStore, NormalizedMessage } from '../../../stores/useSessionStore'; import type { SessionStore, NormalizedMessage } from '../../../stores/useSessionStore';
const isActionablePermissionRequest = (request: { toolName?: unknown } | null | undefined): boolean => {
return request?.toolName !== 'ExitPlanMode' && request?.toolName !== 'exit_plan_mode';
};
const hasActionablePermissionRequests = (requests: Array<{ toolName?: unknown }> | null | undefined): boolean => {
return Array.isArray(requests) && requests.some((request) => isActionablePermissionRequest(request));
};
interface UseChatRealtimeHandlersArgs { interface UseChatRealtimeHandlersArgs {
subscribe: (listener: (event: ServerEvent) => void) => () => void; subscribe: (listener: (event: ServerEvent) => void) => () => void;
provider: LLMProvider; provider: LLMProvider;
selectedSession: ProjectSession | null; selectedSession: ProjectSession | null;
currentSessionId: string | null; currentSessionId: string | null;
setTokenBudget: (budget: Record<string, unknown> | null) => void; setTokenBudget: (budget: Record<string, unknown> | null) => void;
pendingPermissionRequests: PendingPermissionRequest[];
setPendingPermissionRequests: Dispatch<SetStateAction<PendingPermissionRequest[]>>; setPendingPermissionRequests: Dispatch<SetStateAction<PendingPermissionRequest[]>>;
streamTimerRef: MutableRefObject<number | null>; streamTimerRef: MutableRefObject<number | null>;
accumulatedStreamRef: MutableRefObject<string>; accumulatedStreamRef: MutableRefObject<string>;
@@ -52,6 +61,7 @@ export function useChatRealtimeHandlers({
selectedSession, selectedSession,
currentSessionId, currentSessionId,
setTokenBudget, setTokenBudget,
pendingPermissionRequests,
setPendingPermissionRequests, setPendingPermissionRequests,
streamTimerRef, streamTimerRef,
accumulatedStreamRef, accumulatedStreamRef,
@@ -62,13 +72,29 @@ export function useChatRealtimeHandlers({
onWebSocketReconnect, onWebSocketReconnect,
sessionStore, sessionStore,
}: UseChatRealtimeHandlersArgs) { }: UseChatRealtimeHandlersArgs) {
// Session switches can send `chat.subscribe` before this effect has a chance
// to rebind the websocket listener. Read the visible session id from a ref
// so a fast `chat_subscribed` ack is matched against the current view, not
// the previous render's closed-over selection.
const activeViewSessionIdRef = useRef<string | null>(selectedSession?.id || currentSessionId || null);
activeViewSessionIdRef.current = selectedSession?.id || currentSessionId || null;
// Keep the latest pending-permission snapshot available to the websocket
// listener so back-to-back permission events can dedupe and re-arm the
// notification sound before React finishes a rerender.
const pendingPermissionRequestsRef = useRef(pendingPermissionRequests);
useEffect(() => {
pendingPermissionRequestsRef.current = pendingPermissionRequests;
}, [pendingPermissionRequests]);
useEffect(() => { useEffect(() => {
const handleEvent = (msg: ServerEvent) => { const handleEvent = (msg: ServerEvent) => {
if (!msg.kind) { if (!msg.kind) {
return; return;
} }
const activeViewSessionId = selectedSession?.id || currentSessionId || null; const activeViewSessionId = activeViewSessionIdRef.current;
const sid = (typeof msg.sessionId === 'string' && msg.sessionId) || activeViewSessionId; const sid = (typeof msg.sessionId === 'string' && msg.sessionId) || activeViewSessionId;
// Record replay progress for every sequenced live event. // Record replay progress for every sequenced live event.
@@ -101,7 +127,16 @@ export function useChatRealtimeHandlers({
const isViewedSession = sid === activeViewSessionId; const isViewedSession = sid === activeViewSessionId;
if (isViewedSession && Array.isArray(msg.pendingPermissions)) { if (isViewedSession && Array.isArray(msg.pendingPermissions)) {
setPendingPermissionRequests(msg.pendingPermissions as PendingPermissionRequest[]); const nextPendingPermissionRequests = msg.pendingPermissions as PendingPermissionRequest[];
const hadActionablePermissionRequests = hasActionablePermissionRequests(pendingPermissionRequestsRef.current);
const hasPendingActionablePermissionRequests = hasActionablePermissionRequests(nextPendingPermissionRequests);
pendingPermissionRequestsRef.current = nextPendingPermissionRequests;
setPendingPermissionRequests(nextPendingPermissionRequests);
if (hasPendingActionablePermissionRequests && !hadActionablePermissionRequests) {
void playNotificationSound();
}
} }
return; return;
} }
@@ -203,6 +238,7 @@ export function useChatRealtimeHandlers({
// hides it immediately and atomically. // hides it immediately and atomically.
onSessionIdle?.(sid); onSessionIdle?.(sid);
if (sid === activeViewSessionId) { if (sid === activeViewSessionId) {
pendingPermissionRequestsRef.current = [];
setPendingPermissionRequests([]); setPendingPermissionRequests([]);
} }
@@ -234,10 +270,14 @@ export function useChatRealtimeHandlers({
case 'permission_request': { case 'permission_request': {
if (!msg.requestId) break; if (!msg.requestId) break;
if (isActionablePermissionRequest({ toolName: msg.toolName })) {
void playNotificationSound();
}
if (sid === activeViewSessionId) { if (sid === activeViewSessionId) {
setPendingPermissionRequests((prev) => { const previousPendingPermissionRequests = pendingPermissionRequestsRef.current;
if (prev.some((r: PendingPermissionRequest) => r.requestId === msg.requestId)) return prev; if (!previousPendingPermissionRequests.some((request) => request.requestId === msg.requestId)) {
return [...prev, { const nextPendingPermissionRequests = [...previousPendingPermissionRequests, {
requestId: msg.requestId as string, requestId: msg.requestId as string,
toolName: (msg.toolName as string) || 'UnknownTool', toolName: (msg.toolName as string) || 'UnknownTool',
input: msg.input, input: msg.input,
@@ -245,7 +285,10 @@ export function useChatRealtimeHandlers({
sessionId: sid || null, sessionId: sid || null,
receivedAt: new Date(), receivedAt: new Date(),
}]; }];
});
pendingPermissionRequestsRef.current = nextPendingPermissionRequests;
setPendingPermissionRequests(nextPendingPermissionRequests);
}
} }
if (sid) { if (sid) {
onSessionProcessing?.(sid); onSessionProcessing?.(sid);
@@ -255,7 +298,12 @@ export function useChatRealtimeHandlers({
case 'permission_cancelled': { case 'permission_cancelled': {
if (msg.requestId && sid === activeViewSessionId) { if (msg.requestId && sid === activeViewSessionId) {
setPendingPermissionRequests((prev) => prev.filter((r: PendingPermissionRequest) => r.requestId !== msg.requestId)); const nextPendingPermissionRequests = pendingPermissionRequestsRef.current.filter(
(request: PendingPermissionRequest) => request.requestId !== msg.requestId,
);
pendingPermissionRequestsRef.current = nextPendingPermissionRequests;
setPendingPermissionRequests(nextPendingPermissionRequests);
} }
break; break;
} }
@@ -286,6 +334,7 @@ export function useChatRealtimeHandlers({
selectedSession, selectedSession,
currentSessionId, currentSessionId,
setTokenBudget, setTokenBudget,
pendingPermissionRequests,
setPendingPermissionRequests, setPendingPermissionRequests,
streamTimerRef, streamTimerRef,
accumulatedStreamRef, accumulatedStreamRef,

View File

@@ -0,0 +1,33 @@
import { useCallback, useEffect, useState } from 'react';
import { voicePlayer, voiceId, type VoiceSnapshot } from '../../../lib/voicePlayer';
export type TtsState = VoiceSnapshot['state'];
/**
* Thin adapter over the app-level voicePlayer. Playback lives outside React (see
* lib/voicePlayer), so switching chats or re-rendering a message no longer cuts the
* audio off. This hook just reflects the player's state for one message and forwards taps.
*/
export function useTts(getText: () => string) {
const content = getText();
const id = voiceId(content);
const [snap, setSnap] = useState<VoiceSnapshot>(() => voicePlayer.getSnapshot(id));
useEffect(() => {
const update = () =>
setSnap((prev) => {
const next = voicePlayer.getSnapshot(id);
return prev.state === next.state && prev.error === next.error ? prev : next;
});
update();
return voicePlayer.subscribe(update);
}, [id]);
const toggle = useCallback(() => {
voicePlayer.unlock(); // synchronous, within the click gesture (iOS)
voicePlayer.toggle(content);
}, [content]);
return { state: snap.state, toggle, error: snap.error };
}

View File

@@ -0,0 +1,85 @@
import { useEffect, useState } from 'react';
import { authenticatedFetch } from '../../../utils/api';
import { readVoiceConfig, VOICE_CONFIG_SYNC_EVENT } from '../../../hooks/useVoiceConfig';
// Voice UI is gated on the `voiceEnabled` UI preference (toggled in Quick Settings /
// the Settings modal) and a configured voice backend.
const STORAGE_KEY = 'uiPreferences';
const SYNC_EVENT = 'ui-preferences:sync';
let healthRequest: Promise<boolean> | null = null;
function checkVoiceHealth(): Promise<boolean> {
if (healthRequest) return healthRequest;
const request = authenticatedFetch('/api/voice/health')
.then(async (response) => {
if (!response.ok) throw new Error(`Voice health check failed (${response.status})`);
const data = await response.json();
return data?.configured === true;
})
.finally(() => {
healthRequest = null;
});
healthRequest = request;
return request;
}
function readVoiceEnabled(): boolean {
try {
const raw = localStorage.getItem(STORAGE_KEY);
if (!raw) return false;
const parsed = JSON.parse(raw);
return parsed?.voiceEnabled === true || parsed?.voiceEnabled === 'true';
} catch {
return false;
}
}
export function useVoiceAvailable(): boolean {
const [enabled, setEnabled] = useState<boolean>(() =>
typeof window === 'undefined' ? false : readVoiceEnabled(),
);
const [available, setAvailable] = useState(false);
useEffect(() => {
const update = () => setEnabled(readVoiceEnabled());
window.addEventListener('storage', update);
window.addEventListener(SYNC_EVENT, update as EventListener);
return () => {
window.removeEventListener('storage', update);
window.removeEventListener(SYNC_EVENT, update as EventListener);
};
}, []);
useEffect(() => {
let active = true;
let requestId = 0;
const check = async () => {
if (!enabled) {
setAvailable(false);
return;
}
if (readVoiceConfig().baseUrl.trim()) {
setAvailable(true);
return;
}
const id = ++requestId;
try {
const result = await checkVoiceHealth();
if (active && id === requestId) setAvailable(result);
} catch {
if (active && id === requestId) setAvailable(false);
}
};
void check();
window.addEventListener(VOICE_CONFIG_SYNC_EVENT, check);
return () => {
active = false;
window.removeEventListener(VOICE_CONFIG_SYNC_EVENT, check);
};
}, [enabled]);
return enabled && available;
}

View File

@@ -0,0 +1,149 @@
import { useCallback, useEffect, useRef, useState } from 'react';
import { transcribeVoice } from '../../../lib/voiceApi';
// Mobile-safe recording: iOS Safari 18.4+ supports webm/opus; older iOS needs mp4.
const MIME_CANDIDATES = [
'audio/webm;codecs=opus',
'audio/webm',
'audio/mp4',
'audio/ogg;codecs=opus',
'audio/ogg',
];
function pickMime(): string {
for (const t of MIME_CANDIDATES) {
try {
if (typeof MediaRecorder !== 'undefined' && MediaRecorder.isTypeSupported(t)) return t;
} catch {
/* isTypeSupported can throw on some iOS versions */
}
}
return '';
}
export type VoiceInputState = 'idle' | 'recording' | 'transcribing';
/**
* Push-to-talk dictation. Records the mic, uploads to /api/voice/transcribe
* (an OpenAI-compatible speech-to-text backend via the Express proxy), and
* returns the transcript through onTranscript.
*/
export function useVoiceInput(
onTranscript: (text: string, send?: boolean) => void,
onError?: (msg: string) => void,
) {
const [state, setState] = useState<VoiceInputState>('idle');
const recorderRef = useRef<MediaRecorder | null>(null);
const chunksRef = useRef<Blob[]>([]);
const streamRef = useRef<MediaStream | null>(null);
const cancelledRef = useRef(false);
const startingRef = useRef(false);
// Whether the in-progress stop should auto-send the transcript (vs just fill the box).
const sendRef = useRef(false);
const stopTracks = () => {
streamRef.current?.getTracks().forEach((t) => t.stop());
streamRef.current = null;
};
// Stop the mic if the component unmounts mid-recording.
useEffect(() => {
cancelledRef.current = false;
return () => {
cancelledRef.current = true;
startingRef.current = false;
streamRef.current?.getTracks().forEach((t) => t.stop());
streamRef.current = null;
recorderRef.current = null;
};
}, []);
const start = useCallback(async () => {
if (startingRef.current || (recorderRef.current && recorderRef.current.state !== 'inactive')) return;
startingRef.current = true;
try {
const stream = await navigator.mediaDevices.getUserMedia({
audio: { echoCancellation: true, noiseSuppression: true },
});
if (cancelledRef.current) {
stream.getTracks().forEach((t) => t.stop());
return;
}
streamRef.current = stream;
const mimeType = pickMime();
const rec = mimeType ? new MediaRecorder(stream, { mimeType }) : new MediaRecorder(stream);
recorderRef.current = rec;
chunksRef.current = [];
rec.ondataavailable = (e) => {
if (e.data.size > 0) chunksRef.current.push(e.data);
};
rec.onstop = async () => {
stopTracks();
if (cancelledRef.current) return;
// Capture and clear the send intent for this stop before any async work.
const shouldSend = sendRef.current;
sendRef.current = false;
const type = rec.mimeType || 'audio/webm';
const blob = new Blob(chunksRef.current, { type });
if (blob.size < 800) {
setState('idle');
onError?.('Recording too short');
return;
}
setState('transcribing');
try {
const ext = type.includes('mp4') ? 'm4a' : type.includes('ogg') ? 'ogg' : 'webm';
const res = await transcribeVoice(blob, `recording.${ext}`);
if (!res.ok) throw new Error(`transcribe ${res.status}`);
const data = await res.json();
if (cancelledRef.current) return;
const text = String(data?.text || '').trim();
if (text) onTranscript(text, shouldSend);
else onError?.('No speech detected');
} catch (e) {
if (!cancelledRef.current) {
onError?.(`Transcription failed: ${e instanceof Error ? e.message : String(e)}`);
}
} finally {
if (!cancelledRef.current) setState('idle');
}
};
rec.start();
setState('recording');
} catch (e) {
recorderRef.current = null;
stopTracks();
if (cancelledRef.current) return;
const err = e as { name?: string; message?: string };
let msg = `Mic error: ${err?.message || e}`;
if (err?.name === 'NotAllowedError') msg = 'Microphone access denied.';
else if (err?.name === 'NotFoundError') msg = 'No microphone found.';
onError?.(msg);
setState('idle');
} finally {
startingRef.current = false;
}
}, [onTranscript, onError]);
// Stop recording. Pass { send: true } to auto-send the transcript once it's ready.
// Guard on the recorder's own state (not React state) so a double tap, or the mic
// and Send buttons both firing, can't call stop() on an already-inactive recorder.
const stop = useCallback((opts?: { send?: boolean }) => {
const rec = recorderRef.current;
if (rec && rec.state !== 'inactive') {
sendRef.current = opts?.send ?? false;
rec.stop();
}
}, []);
const toggle = useCallback(() => {
if (state === 'recording') stop();
else if (state === 'idle') start();
}, [state, start, stop]);
return { state, toggle, stop };
}

View File

@@ -173,6 +173,7 @@ function ChatInterface({
isDragActive, isDragActive,
openImagePicker, openImagePicker,
handleSubmit, handleSubmit,
handleVoiceTranscript,
handleInputChange, handleInputChange,
handleKeyDown, handleKeyDown,
handlePaste, handlePaste,
@@ -239,6 +240,7 @@ function ChatInterface({
selectedSession, selectedSession,
currentSessionId, currentSessionId,
setTokenBudget, setTokenBudget,
pendingPermissionRequests,
setPendingPermissionRequests, setPendingPermissionRequests,
streamTimerRef, streamTimerRef,
accumulatedStreamRef, accumulatedStreamRef,
@@ -405,6 +407,7 @@ function ChatInterface({
renderInputWithMentions={renderInputWithMentions} renderInputWithMentions={renderInputWithMentions}
textareaRef={textareaRef} textareaRef={textareaRef}
input={input} input={input}
onVoiceTranscript={handleVoiceTranscript}
onInputChange={handleInputChange} onInputChange={handleInputChange}
onTextareaClick={handleTextareaClick} onTextareaClick={handleTextareaClick}
onTextareaKeyDown={handleKeyDown} onTextareaKeyDown={handleKeyDown}

View File

@@ -1,4 +1,5 @@
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'react-i18next';
import { useCallback, useEffect, useRef, useState } from 'react';
import type { import type {
ChangeEvent, ChangeEvent,
ClipboardEvent, ClipboardEvent,
@@ -9,8 +10,10 @@ import type {
RefObject, RefObject,
TouchEvent, TouchEvent,
} from 'react'; } from 'react';
import { ImageIcon, MessageSquareIcon, XIcon, ArrowDownIcon } from 'lucide-react'; import { ImageIcon, MessageSquareIcon, XIcon, ArrowDownIcon, Loader2 } from 'lucide-react';
import { useVoiceInput } from '../../hooks/useVoiceInput';
import { useVoiceAvailable } from '../../hooks/useVoiceAvailable';
import type { SessionActivity } from '../../../../hooks/useSessionProtection'; import type { SessionActivity } from '../../../../hooks/useSessionProtection';
import type { PendingPermissionRequest, PermissionMode } from '../../types/types'; import type { PendingPermissionRequest, PermissionMode } from '../../types/types';
import { import {
@@ -27,6 +30,7 @@ import {
import CommandMenu from './CommandMenu'; import CommandMenu from './CommandMenu';
import ActivityIndicator from './ActivityIndicator'; import ActivityIndicator from './ActivityIndicator';
import ImageAttachment from './ImageAttachment'; import ImageAttachment from './ImageAttachment';
import VoiceInputButton from './VoiceInputButton';
import PermissionRequestsBanner from './PermissionRequestsBanner'; import PermissionRequestsBanner from './PermissionRequestsBanner';
import TokenUsageSummary from './TokenUsageSummary'; import TokenUsageSummary from './TokenUsageSummary';
@@ -89,6 +93,7 @@ interface ChatComposerProps {
renderInputWithMentions: (text: string) => ReactNode; renderInputWithMentions: (text: string) => ReactNode;
textareaRef: RefObject<HTMLTextAreaElement>; textareaRef: RefObject<HTMLTextAreaElement>;
input: string; input: string;
onVoiceTranscript?: (text: string, send?: boolean) => void;
onInputChange: (event: ChangeEvent<HTMLTextAreaElement>) => void; onInputChange: (event: ChangeEvent<HTMLTextAreaElement>) => void;
onTextareaClick: (event: MouseEvent<HTMLTextAreaElement>) => void; onTextareaClick: (event: MouseEvent<HTMLTextAreaElement>) => void;
onTextareaKeyDown: (event: KeyboardEvent<HTMLTextAreaElement>) => void; onTextareaKeyDown: (event: KeyboardEvent<HTMLTextAreaElement>) => void;
@@ -142,6 +147,7 @@ export default function ChatComposer({
renderInputWithMentions, renderInputWithMentions,
textareaRef, textareaRef,
input, input,
onVoiceTranscript,
onInputChange, onInputChange,
onTextareaClick, onTextareaClick,
onTextareaKeyDown, onTextareaKeyDown,
@@ -154,6 +160,28 @@ export default function ChatComposer({
sendByCtrlEnter, sendByCtrlEnter,
}: ChatComposerProps) { }: ChatComposerProps) {
const { t } = useTranslation('chat'); const { t } = useTranslation('chat');
// Voice state is hosted here (not in the mic button) so the main Send button can stop
// recording and send the transcript in one tap, the way the mic button drops it in the box.
const voiceAvailable = useVoiceAvailable();
const [voiceError, setVoiceError] = useState<string | null>(null);
const voiceErrorTimer = useRef<ReturnType<typeof setTimeout> | null>(null);
const handleVoiceError = useCallback((msg: string) => {
setVoiceError(msg);
if (voiceErrorTimer.current) clearTimeout(voiceErrorTimer.current);
voiceErrorTimer.current = setTimeout(() => setVoiceError(null), 4000);
}, []);
useEffect(() => () => {
if (voiceErrorTimer.current) clearTimeout(voiceErrorTimer.current);
}, []);
const noopTranscript = useCallback(() => {}, []);
const { state: voiceState, toggle: voiceToggle, stop: voiceStop } = useVoiceInput(
onVoiceTranscript ?? noopTranscript,
handleVoiceError,
);
const isRecording = voiceState === 'recording';
const isTranscribing = voiceState === 'transcribing';
const textareaRect = textareaRef.current?.getBoundingClientRect(); const textareaRect = textareaRef.current?.getBoundingClientRect();
const commandMenuPosition = { const commandMenuPosition = {
top: textareaRect ? Math.max(16, textareaRect.top - 316) : 0, top: textareaRect ? Math.max(16, textareaRect.top - 316) : 0,
@@ -309,6 +337,10 @@ export default function ChatComposer({
<ImageIcon /> <ImageIcon />
</PromptInputButton> </PromptInputButton>
{onVoiceTranscript && voiceAvailable && (
<VoiceInputButton state={voiceState} onToggle={voiceToggle} errorMsg={voiceError} />
)}
<button <button
type="button" type="button"
onClick={onModeSwitch} onClick={onModeSwitch}
@@ -387,10 +419,21 @@ export default function ChatComposer({
{sendByCtrlEnter ? t('input.hintText.ctrlEnter') : t('input.hintText.enter')} {sendByCtrlEnter ? t('input.hintText.ctrlEnter') : t('input.hintText.enter')}
</div> </div>
<PromptInputSubmit <PromptInputSubmit
onClick={isLoading ? onAbortSession : undefined} onClick={
disabled={!isLoading && !input.trim()} isLoading
? onAbortSession
: isRecording
? (e: MouseEvent<HTMLButtonElement>) => {
e.preventDefault();
voiceStop({ send: true });
}
: undefined
}
disabled={isLoading ? false : isRecording ? false : isTranscribing ? true : !input.trim()}
className="h-10 w-10 sm:h-10 sm:w-10" className="h-10 w-10 sm:h-10 sm:w-10"
/> >
{isTranscribing ? <Loader2 className="h-4 w-4 animate-spin" /> : undefined}
</PromptInputSubmit>
</div> </div>
</PromptInputFooter> </PromptInputFooter>
</PromptInput> </PromptInput>

View File

@@ -15,6 +15,7 @@ import { Reasoning, ReasoningTrigger, ReasoningContent } from '../../../../share
import { Markdown } from './Markdown'; import { Markdown } from './Markdown';
import MessageCopyControl from './MessageCopyControl'; import MessageCopyControl from './MessageCopyControl';
import MessageSpeakControl from './MessageSpeakControl';
type DiffLine = { type DiffLine = {
type: string; type: string;
@@ -415,6 +416,9 @@ const MessageComponent = memo(({ message, prevMessage, createDiff, onFileOpen, a
{shouldShowAssistantCopyControl && ( {shouldShowAssistantCopyControl && (
<MessageCopyControl content={assistantCopyContent} messageType="assistant" /> <MessageCopyControl content={assistantCopyContent} messageType="assistant" />
)} )}
{shouldShowAssistantCopyControl && (
<MessageSpeakControl content={assistantCopyContent} />
)}
{!isGrouped && <span>{formattedTime}</span>} {!isGrouped && <span>{formattedTime}</span>}
</div> </div>
)} )}

View File

@@ -0,0 +1,44 @@
import { Volume2, Loader2, Square } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { useTts } from '../../hooks/useTts';
import { useVoiceAvailable } from '../../hooks/useVoiceAvailable';
// Tap-to-speak button beside the copy control on assistant messages.
// Renders nothing unless the optional voice feature is enabled.
const MessageSpeakControl = ({ content }: { content: string }) => {
const { t } = useTranslation('chat');
const available = useVoiceAvailable();
const { state, toggle, error } = useTts(() => content);
if (!available) return null;
const title =
state === 'playing' ? t('voice.stopSpeaking') : state === 'loading' ? t('voice.loading') : t('voice.speak');
return (
<span className="relative inline-flex">
{error && (
<span className="absolute bottom-full left-1/2 z-10 mb-1 max-w-[240px] -translate-x-1/2 whitespace-normal rounded bg-red-600 px-2 py-1 text-center text-xs text-white shadow-lg">
{error}
</span>
)}
<button
type="button"
onClick={toggle}
title={title}
aria-label={title}
className="inline-flex items-center gap-1 rounded px-1 py-0.5 text-gray-400 transition-colors hover:text-gray-600 dark:text-gray-500 dark:hover:text-gray-300"
>
{state === 'playing' ? (
<Square className="h-3.5 w-3.5" />
) : state === 'loading' ? (
<Loader2 className="h-3.5 w-3.5 animate-spin" />
) : (
<Volume2 className="h-3.5 w-3.5" />
)}
</button>
</span>
);
};
export default MessageSpeakControl;

View File

@@ -0,0 +1,46 @@
import { useTranslation } from 'react-i18next';
import { Mic, Square, Loader2 } from 'lucide-react';
import { PromptInputButton } from '../../../../shared/view/ui';
import type { VoiceInputState } from '../../hooks/useVoiceInput';
type Props = {
state: VoiceInputState;
onToggle: () => void;
errorMsg?: string | null;
};
// Push-to-talk mic button (presentational). Recording state and the stop-and-send action
// are owned by the composer so the main Send button can drive them too. This button just
// starts recording and, while recording, stops and drops the transcript into the input box.
export default function VoiceInputButton({ state, onToggle, errorMsg }: Props) {
const { t } = useTranslation('chat');
const icon =
state === 'recording' ? (
<Square className="text-red-500" />
) : state === 'transcribing' ? (
<Loader2 className="animate-spin" />
) : (
<Mic />
);
return (
<span className="relative inline-flex">
{errorMsg && (
<span className="absolute bottom-full left-1/2 mb-1 -translate-x-1/2 whitespace-nowrap rounded bg-red-600 px-2 py-1 text-xs text-white shadow-lg">
{errorMsg}
</span>
)}
<PromptInputButton
tooltip={{ content: state === 'recording' ? t('voice.stopRecording') : t('voice.input') }}
onClick={(e: { preventDefault: () => void }) => {
e.preventDefault();
onToggle();
}}
>
{icon}
</PromptInputButton>
</span>
);
}

View File

@@ -4,6 +4,7 @@ import {
Eye, Eye,
Languages, Languages,
Maximize2, Maximize2,
Mic,
} from 'lucide-react'; } from 'lucide-react';
import type { PreferenceToggleItem } from './types'; import type { PreferenceToggleItem } from './types';
@@ -54,4 +55,9 @@ export const INPUT_SETTING_TOGGLES: PreferenceToggleItem[] = [
labelKey: 'quickSettings.sendByCtrlEnter', labelKey: 'quickSettings.sendByCtrlEnter',
icon: Languages, icon: Languages,
}, },
{
key: 'voiceEnabled',
labelKey: 'quickSettings.voiceEnabled',
icon: Mic,
},
]; ];

View File

@@ -6,7 +6,8 @@ export type PreferenceToggleKey =
| 'showRawParameters' | 'showRawParameters'
| 'showThinking' | 'showThinking'
| 'autoScrollToBottom' | 'autoScrollToBottom'
| 'sendByCtrlEnter'; | 'sendByCtrlEnter'
| 'voiceEnabled';
export type QuickSettingsPreferences = Record<PreferenceToggleKey, boolean>; export type QuickSettingsPreferences = Record<PreferenceToggleKey, boolean>;

View File

@@ -28,6 +28,9 @@ export default function QuickSettingsContent({
onPreferenceChange, onPreferenceChange,
}: QuickSettingsContentProps) { }: QuickSettingsContentProps) {
const { t } = useTranslation('settings'); const { t } = useTranslation('settings');
const inputSettingToggles = preferences.voiceEnabled
? INPUT_SETTING_TOGGLES
: INPUT_SETTING_TOGGLES.filter(({ key }) => key !== 'voiceEnabled');
const renderToggleRows = (items: PreferenceToggleItem[]) => ( const renderToggleRows = (items: PreferenceToggleItem[]) => (
items.map(({ key, labelKey, icon }) => ( items.map(({ key, labelKey, icon }) => (
@@ -67,7 +70,7 @@ export default function QuickSettingsContent({
</QuickSettingsSection> </QuickSettingsSection>
<QuickSettingsSection title={t('quickSettings.sections.inputSettings')}> <QuickSettingsSection title={t('quickSettings.sections.inputSettings')}>
{renderToggleRows(INPUT_SETTING_TOGGLES)} {renderToggleRows(inputSettingToggles)}
<p className="ml-3 text-xs text-gray-500 dark:text-gray-400"> <p className="ml-3 text-xs text-gray-500 dark:text-gray-400">
{t('quickSettings.sendByCtrlEnterDescription')} {t('quickSettings.sendByCtrlEnterDescription')}
</p> </p>

View File

@@ -27,12 +27,14 @@ export default function QuickSettingsPanelView() {
showThinking: preferences.showThinking, showThinking: preferences.showThinking,
autoScrollToBottom: preferences.autoScrollToBottom, autoScrollToBottom: preferences.autoScrollToBottom,
sendByCtrlEnter: preferences.sendByCtrlEnter, sendByCtrlEnter: preferences.sendByCtrlEnter,
voiceEnabled: preferences.voiceEnabled,
}), [ }), [
preferences.autoExpandTools, preferences.autoExpandTools,
preferences.autoScrollToBottom, preferences.autoScrollToBottom,
preferences.sendByCtrlEnter, preferences.sendByCtrlEnter,
preferences.showRawParameters, preferences.showRawParameters,
preferences.showThinking, preferences.showThinking,
preferences.voiceEnabled,
]); ]);
const handlePreferenceChange = useCallback( const handlePreferenceChange = useCallback(

View File

@@ -3,7 +3,7 @@ import type { Dispatch, SetStateAction } from 'react';
import type { LLMProvider } from '../../../types/app'; import type { LLMProvider } from '../../../types/app';
import type { ProviderAuthStatus } from '../../provider-auth/types'; import type { ProviderAuthStatus } from '../../provider-auth/types';
export type SettingsMainTab = 'agents' | 'appearance' | 'git' | 'api' | 'tasks' | 'browser' | 'notifications' | 'plugins' | 'about'; export type SettingsMainTab = 'agents' | 'appearance' | 'git' | 'api' | 'voice' | 'tasks' | 'browser' | 'notifications' | 'plugins' | 'about';
export type AgentProvider = LLMProvider; export type AgentProvider = LLMProvider;
export type AgentCategory = 'account' | 'permissions' | 'mcp' | 'skills'; export type AgentCategory = 'account' | 'permissions' | 'mcp' | 'skills';
export type ProjectSortOrder = 'name' | 'date'; export type ProjectSortOrder = 'name' | 'date';

View File

@@ -7,6 +7,7 @@ import SettingsSidebar from '../view/SettingsSidebar';
import AgentsSettingsTab from '../view/tabs/agents-settings/AgentsSettingsTab'; import AgentsSettingsTab from '../view/tabs/agents-settings/AgentsSettingsTab';
import AppearanceSettingsTab from '../view/tabs/AppearanceSettingsTab'; import AppearanceSettingsTab from '../view/tabs/AppearanceSettingsTab';
import CredentialsSettingsTab from '../view/tabs/api-settings/CredentialsSettingsTab'; import CredentialsSettingsTab from '../view/tabs/api-settings/CredentialsSettingsTab';
import VoiceSettingsTab from '../view/tabs/VoiceSettingsTab';
import GitSettingsTab from '../view/tabs/git-settings/GitSettingsTab'; import GitSettingsTab from '../view/tabs/git-settings/GitSettingsTab';
import BrowserUseSettingsTab from '../view/tabs/browser-use-settings/BrowserUseSettingsTab'; import BrowserUseSettingsTab from '../view/tabs/browser-use-settings/BrowserUseSettingsTab';
import NotificationsSettingsTab from '../view/tabs/NotificationsSettingsTab'; import NotificationsSettingsTab from '../view/tabs/NotificationsSettingsTab';
@@ -157,6 +158,8 @@ function Settings({ isOpen, onClose, projects = [], initialTab = 'agents' }: Set
{activeTab === 'api' && <CredentialsSettingsTab />} {activeTab === 'api' && <CredentialsSettingsTab />}
{activeTab === 'voice' && <VoiceSettingsTab />}
{activeTab === 'plugins' && <PluginSettingsTab />} {activeTab === 'plugins' && <PluginSettingsTab />}
{activeTab === 'about' && <AboutTab />} {activeTab === 'about' && <AboutTab />}

View File

@@ -1,5 +1,6 @@
import { Bell, Bot, GitBranch, Info, Key, ListChecks, MonitorPlay, Palette, Puzzle } from 'lucide-react'; import { Bell, Bot, GitBranch, Info, Key, ListChecks, Mic, MonitorPlay, Palette, Puzzle } from 'lucide-react';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'react-i18next';
import { cn } from '../../../lib/utils'; import { cn } from '../../../lib/utils';
import { PillBar, Pill } from '../../../shared/view/ui'; import { PillBar, Pill } from '../../../shared/view/ui';
import type { SettingsMainTab } from '../types/types'; import type { SettingsMainTab } from '../types/types';
@@ -20,6 +21,7 @@ const NAV_ITEMS: NavItem[] = [
{ id: 'appearance', labelKey: 'mainTabs.appearance', icon: Palette }, { id: 'appearance', labelKey: 'mainTabs.appearance', icon: Palette },
{ id: 'git', labelKey: 'mainTabs.git', icon: GitBranch }, { id: 'git', labelKey: 'mainTabs.git', icon: GitBranch },
{ id: 'api', labelKey: 'mainTabs.apiTokens', icon: Key }, { id: 'api', labelKey: 'mainTabs.apiTokens', icon: Key },
{ id: 'voice', labelKey: 'mainTabs.voice', icon: Mic },
{ id: 'tasks', labelKey: 'mainTabs.tasks', icon: ListChecks }, { id: 'tasks', labelKey: 'mainTabs.tasks', icon: ListChecks },
{ id: 'browser', labelKey: 'mainTabs.browser', icon: MonitorPlay }, { id: 'browser', labelKey: 'mainTabs.browser', icon: MonitorPlay },
{ id: 'plugins', labelKey: 'mainTabs.plugins', icon: Puzzle }, { id: 'plugins', labelKey: 'mainTabs.plugins', icon: Puzzle },

View File

@@ -0,0 +1,91 @@
import type { InputHTMLAttributes } from 'react';
import { useTranslation } from 'react-i18next';
import SettingsSection from '../SettingsSection';
import SettingsToggle from '../SettingsToggle';
import { useUiPreferences } from '../../../../hooks/useUiPreferences';
import { useVoiceConfig } from '../../../../hooks/useVoiceConfig';
const inputClass =
'w-full rounded-md border border-border bg-background px-3 py-2 text-sm text-foreground placeholder:text-muted-foreground focus:outline-none focus:ring-2 focus:ring-ring';
function Field({ label, ...props }: { label: string } & InputHTMLAttributes<HTMLInputElement>) {
return (
<label className="block space-y-1">
<span className="text-sm font-medium text-foreground">{label}</span>
<input className={inputClass} {...props} />
</label>
);
}
export default function VoiceSettingsTab() {
const { t } = useTranslation('settings');
const { preferences, setPreference } = useUiPreferences();
const { config, update } = useVoiceConfig();
const voiceEnabled = preferences.voiceEnabled;
return (
<div className="space-y-8">
<SettingsSection title={t('voiceSettings.title')} description={t('voiceSettings.description')}>
<div className="flex items-center justify-between rounded-lg border border-border p-3">
<div className="pr-3">
<div className="text-sm font-medium text-foreground">{t('voiceSettings.enable')}</div>
<div className="text-xs text-muted-foreground">{t('voiceSettings.enableDescription')}</div>
</div>
<SettingsToggle
checked={voiceEnabled}
onChange={(v) => setPreference('voiceEnabled', v)}
ariaLabel={t('voiceSettings.enable')}
/>
</div>
</SettingsSection>
{voiceEnabled && (
<SettingsSection title={t('voiceSettings.backendTitle')} description={t('voiceSettings.backendDescription')}>
<div className="space-y-4">
<Field
label={t('voiceSettings.baseUrl')}
placeholder="https://api.openai.com/v1"
value={config.baseUrl}
onChange={(e) => update({ baseUrl: e.target.value })}
/>
<Field
label={t('voiceSettings.apiKey')}
type="password"
autoComplete="off"
placeholder="sk-…"
value={config.apiKey}
onChange={(e) => update({ apiKey: e.target.value })}
/>
<div className="grid grid-cols-1 gap-4 sm:grid-cols-4">
<Field
label={t('voiceSettings.sttModel')}
placeholder="whisper-1"
value={config.sttModel}
onChange={(e) => update({ sttModel: e.target.value })}
/>
<Field
label={t('voiceSettings.ttsModel')}
placeholder="tts-1"
value={config.ttsModel}
onChange={(e) => update({ ttsModel: e.target.value })}
/>
<Field
label={t('voiceSettings.voice')}
placeholder="alloy"
value={config.ttsVoice}
onChange={(e) => update({ ttsVoice: e.target.value })}
/>
<Field
label={t('voiceSettings.format')}
placeholder="mp3"
value={config.ttsFormat}
onChange={(e) => update({ ttsFormat: e.target.value })}
/>
</div>
<p className="text-xs text-muted-foreground">{t('voiceSettings.note')}</p>
</div>
</SettingsSection>
)}
</div>
);
}

View File

@@ -1,6 +1,5 @@
import type { ITerminalOptions } from '@xterm/xterm'; import type { ITerminalOptions } from '@xterm/xterm';
export const CODEX_DEVICE_AUTH_URL = 'https://auth.openai.com/codex/device';
export const SHELL_RESTART_DELAY_MS = 200; export const SHELL_RESTART_DELAY_MS = 200;
export const TERMINAL_INIT_DELAY_MS = 100; export const TERMINAL_INIT_DELAY_MS = 100;
export const TERMINAL_RESIZE_DELAY_MS = 50; export const TERMINAL_RESIZE_DELAY_MS = 50;

View File

@@ -24,7 +24,6 @@ type UseShellConnectionOptions = {
autoConnect: boolean; autoConnect: boolean;
closeSocket: () => void; closeSocket: () => void;
clearTerminalScreen: () => void; clearTerminalScreen: () => void;
setAuthUrl: (nextAuthUrl: string) => void;
onOutputRef?: MutableRefObject<(() => void) | null>; onOutputRef?: MutableRefObject<(() => void) | null>;
}; };
@@ -49,7 +48,6 @@ export function useShellConnection({
autoConnect, autoConnect,
closeSocket, closeSocket,
clearTerminalScreen, clearTerminalScreen,
setAuthUrl,
onOutputRef, onOutputRef,
}: UseShellConnectionOptions): UseShellConnectionResult { }: UseShellConnectionOptions): UseShellConnectionResult {
const [isConnected, setIsConnected] = useState(false); const [isConnected, setIsConnected] = useState(false);
@@ -100,14 +98,8 @@ export function useShellConnection({
return; return;
} }
if (message.type === 'auth_url' || message.type === 'url_open') {
const nextAuthUrl = typeof message.url === 'string' ? message.url : '';
if (nextAuthUrl) {
setAuthUrl(nextAuthUrl);
}
}
}, },
[handleProcessCompletion, onOutputRef, setAuthUrl, terminalRef], [handleProcessCompletion, onOutputRef, terminalRef],
); );
const connectWebSocket = useCallback( const connectWebSocket = useCallback(
@@ -133,7 +125,6 @@ export function useShellConnection({
setIsConnected(true); setIsConnected(true);
setIsConnecting(false); setIsConnecting(false);
connectingRef.current = false; connectingRef.current = false;
setAuthUrl('');
window.setTimeout(() => { window.setTimeout(() => {
const currentTerminal = terminalRef.current; const currentTerminal = terminalRef.current;
@@ -196,7 +187,6 @@ export function useShellConnection({
isPlainShellRef, isPlainShellRef,
selectedProjectRef, selectedProjectRef,
selectedSessionRef, selectedSessionRef,
setAuthUrl,
terminalRef, terminalRef,
wsRef, wsRef,
], ],
@@ -225,8 +215,7 @@ export function useShellConnection({
setIsConnecting(false); setIsConnecting(false);
connectingRef.current = false; connectingRef.current = false;
forceRestartOnInitRef.current = false; forceRestartOnInitRef.current = false;
setAuthUrl(''); }, [clearTerminalScreen, closeSocket]);
}, [clearTerminalScreen, closeSocket, setAuthUrl]);
useEffect(() => { useEffect(() => {
if ( if (

View File

@@ -1,8 +1,9 @@
import { useCallback, useEffect, useRef, useState } from 'react'; import { useCallback, useEffect, useRef } from 'react';
import type { FitAddon } from '@xterm/addon-fit'; import type { FitAddon } from '@xterm/addon-fit';
import type { Terminal } from '@xterm/xterm'; import type { Terminal } from '@xterm/xterm';
import type { UseShellRuntimeOptions, UseShellRuntimeResult } from '../types/types'; import type { UseShellRuntimeOptions, UseShellRuntimeResult } from '../types/types';
import { copyTextToClipboard } from '../../../utils/clipboard';
import { useShellConnection } from './useShellConnection'; import { useShellConnection } from './useShellConnection';
import { useShellTerminal } from './useShellTerminal'; import { useShellTerminal } from './useShellTerminal';
@@ -22,15 +23,11 @@ export function useShellRuntime({
const fitAddonRef = useRef<FitAddon | null>(null); const fitAddonRef = useRef<FitAddon | null>(null);
const wsRef = useRef<WebSocket | null>(null); const wsRef = useRef<WebSocket | null>(null);
const [authUrl, setAuthUrl] = useState('');
const [authUrlVersion, setAuthUrlVersion] = useState(0);
const selectedProjectRef = useRef(selectedProject); const selectedProjectRef = useRef(selectedProject);
const selectedSessionRef = useRef(selectedSession); const selectedSessionRef = useRef(selectedSession);
const initialCommandRef = useRef(initialCommand); const initialCommandRef = useRef(initialCommand);
const isPlainShellRef = useRef(isPlainShell); const isPlainShellRef = useRef(isPlainShell);
const onProcessCompleteRef = useRef(onProcessComplete); const onProcessCompleteRef = useRef(onProcessComplete);
const authUrlRef = useRef('');
const lastSessionIdRef = useRef<string | null>(selectedSession?.id ?? null); const lastSessionIdRef = useRef<string | null>(selectedSession?.id ?? null);
// Keep mutable values in refs so websocket handlers always read current data. // Keep mutable values in refs so websocket handlers always read current data.
@@ -42,12 +39,6 @@ export function useShellRuntime({
onProcessCompleteRef.current = onProcessComplete; onProcessCompleteRef.current = onProcessComplete;
}, [selectedProject, selectedSession, initialCommand, isPlainShell, onProcessComplete]); }, [selectedProject, selectedSession, initialCommand, isPlainShell, onProcessComplete]);
const setCurrentAuthUrl = useCallback((nextAuthUrl: string) => {
authUrlRef.current = nextAuthUrl;
setAuthUrl(nextAuthUrl);
setAuthUrlVersion((previous) => previous + 1);
}, []);
const closeSocket = useCallback(() => { const closeSocket = useCallback(() => {
const activeSocket = wsRef.current; const activeSocket = wsRef.current;
if (!activeSocket) { if (!activeSocket) {
@@ -64,32 +55,6 @@ export function useShellRuntime({
wsRef.current = null; wsRef.current = null;
}, []); }, []);
const openAuthUrlInBrowser = useCallback((url = authUrlRef.current) => {
if (!url) {
return false;
}
const popup = window.open(url, '_blank');
if (popup) {
try {
popup.opener = null;
} catch {
// Ignore cross-origin restrictions when trying to null opener.
}
return true;
}
return false;
}, []);
const copyAuthUrlToClipboard = useCallback(async (url = authUrlRef.current) => {
if (!url) {
return false;
}
return copyTextToClipboard(url);
}, []);
const { isInitialized, clearTerminalScreen, disposeTerminal } = useShellTerminal({ const { isInitialized, clearTerminalScreen, disposeTerminal } = useShellTerminal({
terminalContainerRef, terminalContainerRef,
terminalRef, terminalRef,
@@ -98,10 +63,6 @@ export function useShellRuntime({
selectedProject, selectedProject,
minimal, minimal,
isRestarting, isRestarting,
initialCommandRef,
isPlainShellRef,
authUrlRef,
copyAuthUrlToClipboard,
closeSocket, closeSocket,
}); });
@@ -118,7 +79,6 @@ export function useShellRuntime({
autoConnect, autoConnect,
closeSocket, closeSocket,
clearTerminalScreen, clearTerminalScreen,
setAuthUrl: setCurrentAuthUrl,
onOutputRef, onOutputRef,
}); });
@@ -156,11 +116,7 @@ export function useShellRuntime({
isConnected, isConnected,
isInitialized, isInitialized,
isConnecting, isConnecting,
authUrl,
authUrlVersion,
connectToShell, connectToShell,
disconnectFromShell, disconnectFromShell,
openAuthUrlInBrowser,
copyAuthUrlToClipboard,
}; };
} }

View File

@@ -4,15 +4,18 @@ import { FitAddon } from '@xterm/addon-fit';
import { WebLinksAddon } from '@xterm/addon-web-links'; import { WebLinksAddon } from '@xterm/addon-web-links';
import { WebglAddon } from '@xterm/addon-webgl'; import { WebglAddon } from '@xterm/addon-webgl';
import { Terminal } from '@xterm/xterm'; import { Terminal } from '@xterm/xterm';
import type { Project } from '../../../types/app'; import type { Project } from '../../../types/app';
import { copyTextToClipboard } from '../../../utils/clipboard';
import { import {
CODEX_DEVICE_AUTH_URL,
TERMINAL_INIT_DELAY_MS, TERMINAL_INIT_DELAY_MS,
TERMINAL_OPTIONS, TERMINAL_OPTIONS,
TERMINAL_RESIZE_DELAY_MS, TERMINAL_RESIZE_DELAY_MS,
} from '../constants/constants'; } from '../constants/constants';
import { copyTextToClipboard } from '../../../utils/clipboard'; import {
import { isCodexLoginCommand } from '../utils/auth'; installMobileTerminalSelection,
type MobileTerminalSelectionManager,
} from '../utils/mobileTerminalSelection';
import { sendSocketMessage } from '../utils/socket'; import { sendSocketMessage } from '../utils/socket';
import { ensureXtermFocusStyles } from '../utils/terminalStyles'; import { ensureXtermFocusStyles } from '../utils/terminalStyles';
@@ -24,10 +27,6 @@ type UseShellTerminalOptions = {
selectedProject: Project | null | undefined; selectedProject: Project | null | undefined;
minimal: boolean; minimal: boolean;
isRestarting: boolean; isRestarting: boolean;
initialCommandRef: MutableRefObject<string | null | undefined>;
isPlainShellRef: MutableRefObject<boolean>;
authUrlRef: MutableRefObject<string>;
copyAuthUrlToClipboard: (url?: string) => Promise<boolean>;
closeSocket: () => void; closeSocket: () => void;
}; };
@@ -45,14 +44,11 @@ export function useShellTerminal({
selectedProject, selectedProject,
minimal, minimal,
isRestarting, isRestarting,
initialCommandRef,
isPlainShellRef,
authUrlRef,
copyAuthUrlToClipboard,
closeSocket, closeSocket,
}: UseShellTerminalOptions): UseShellTerminalResult { }: UseShellTerminalOptions): UseShellTerminalResult {
const [isInitialized, setIsInitialized] = useState(false); const [isInitialized, setIsInitialized] = useState(false);
const resizeTimeoutRef = useRef<number | null>(null); const resizeTimeoutRef = useRef<number | null>(null);
const mobileSelectionRef = useRef<MobileTerminalSelectionManager | null>(null);
const selectedProjectKey = selectedProject?.fullPath || selectedProject?.path || ''; const selectedProjectKey = selectedProject?.fullPath || selectedProject?.path || '';
const hasSelectedProject = Boolean(selectedProject); const hasSelectedProject = Boolean(selectedProject);
@@ -70,6 +66,11 @@ export function useShellTerminal({
}, [terminalRef]); }, [terminalRef]);
const disposeTerminal = useCallback(() => { const disposeTerminal = useCallback(() => {
if (mobileSelectionRef.current) {
mobileSelectionRef.current.dispose();
mobileSelectionRef.current = null;
}
if (terminalRef.current) { if (terminalRef.current) {
terminalRef.current.dispose(); terminalRef.current.dispose();
terminalRef.current = null; terminalRef.current = null;
@@ -80,7 +81,8 @@ export function useShellTerminal({
}, [fitAddonRef, terminalRef]); }, [fitAddonRef, terminalRef]);
useEffect(() => { useEffect(() => {
if (!terminalContainerRef.current || !hasSelectedProject || isRestarting || terminalRef.current) { const terminalContainer = terminalContainerRef.current;
if (!terminalContainer || !hasSelectedProject || isRestarting || terminalRef.current) {
return; return;
} }
@@ -102,7 +104,28 @@ export function useShellTerminal({
console.warn('[Shell] WebGL renderer unavailable, using Canvas fallback'); console.warn('[Shell] WebGL renderer unavailable, using Canvas fallback');
} }
nextTerminal.open(terminalContainerRef.current); nextTerminal.open(terminalContainer);
mobileSelectionRef.current = installMobileTerminalSelection(
nextTerminal,
terminalContainer,
{
onFontSizeChange: (fontSize) => {
nextTerminal.options.fontSize = fontSize;
const currentFitAddon = fitAddonRef.current;
if (currentFitAddon) {
currentFitAddon.fit();
sendSocketMessage(wsRef.current, {
type: 'resize',
cols: nextTerminal.cols,
rows: nextTerminal.rows,
});
} else {
nextTerminal.refresh(0, nextTerminal.rows - 1);
}
},
},
);
const copyTerminalSelection = async () => { const copyTerminalSelection = async () => {
const selection = nextTerminal.getSelection(); const selection = nextTerminal.getSelection();
@@ -133,29 +156,9 @@ export function useShellTerminal({
void copyTextToClipboard(selection); void copyTextToClipboard(selection);
}; };
terminalContainerRef.current.addEventListener('copy', handleTerminalCopy); terminalContainer.addEventListener('copy', handleTerminalCopy);
nextTerminal.attachCustomKeyEventHandler((event) => { nextTerminal.attachCustomKeyEventHandler((event) => {
const activeAuthUrl = isCodexLoginCommand(initialCommandRef.current)
? CODEX_DEVICE_AUTH_URL
: authUrlRef.current;
if (
event.type === 'keydown' &&
minimal &&
isPlainShellRef.current &&
activeAuthUrl &&
!event.ctrlKey &&
!event.metaKey &&
!event.altKey &&
event.key?.toLowerCase() === 'c'
) {
event.preventDefault();
event.stopPropagation();
void copyAuthUrlToClipboard(activeAuthUrl);
return false;
}
if ( if (
event.type === 'keydown' && event.type === 'keydown' &&
(event.ctrlKey || event.metaKey) && (event.ctrlKey || event.metaKey) &&
@@ -240,10 +243,10 @@ export function useShellTerminal({
}, TERMINAL_RESIZE_DELAY_MS); }, TERMINAL_RESIZE_DELAY_MS);
}); });
resizeObserver.observe(terminalContainerRef.current); resizeObserver.observe(terminalContainer);
return () => { return () => {
terminalContainerRef.current?.removeEventListener('copy', handleTerminalCopy); terminalContainer.removeEventListener('copy', handleTerminalCopy);
resizeObserver.disconnect(); resizeObserver.disconnect();
if (resizeTimeoutRef.current !== null) { if (resizeTimeoutRef.current !== null) {
window.clearTimeout(resizeTimeoutRef.current); window.clearTimeout(resizeTimeoutRef.current);
@@ -254,16 +257,12 @@ export function useShellTerminal({
disposeTerminal(); disposeTerminal();
}; };
}, [ }, [
authUrlRef,
closeSocket, closeSocket,
copyAuthUrlToClipboard,
disposeTerminal, disposeTerminal,
fitAddonRef, fitAddonRef,
initialCommandRef,
isPlainShellRef,
isRestarting, isRestarting,
minimal,
hasSelectedProject, hasSelectedProject,
minimal,
selectedProjectKey, selectedProjectKey,
terminalContainerRef, terminalContainerRef,
terminalRef, terminalRef,

View File

@@ -4,8 +4,6 @@ import type { Terminal } from '@xterm/xterm';
import type { Project, ProjectSession } from '../../../types/app'; import type { Project, ProjectSession } from '../../../types/app';
export type AuthCopyStatus = 'idle' | 'copied' | 'failed';
export type ShellInitMessage = { export type ShellInitMessage = {
type: 'init'; type: 'init';
projectPath: string; projectPath: string;
@@ -54,7 +52,6 @@ export type ShellSharedRefs = {
wsRef: MutableRefObject<WebSocket | null>; wsRef: MutableRefObject<WebSocket | null>;
terminalRef: MutableRefObject<Terminal | null>; terminalRef: MutableRefObject<Terminal | null>;
fitAddonRef: MutableRefObject<FitAddon | null>; fitAddonRef: MutableRefObject<FitAddon | null>;
authUrlRef: MutableRefObject<string>;
selectedProjectRef: MutableRefObject<Project | null | undefined>; selectedProjectRef: MutableRefObject<Project | null | undefined>;
selectedSessionRef: MutableRefObject<ProjectSession | null | undefined>; selectedSessionRef: MutableRefObject<ProjectSession | null | undefined>;
initialCommandRef: MutableRefObject<string | null | undefined>; initialCommandRef: MutableRefObject<string | null | undefined>;
@@ -69,10 +66,6 @@ export type UseShellRuntimeResult = {
isConnected: boolean; isConnected: boolean;
isInitialized: boolean; isInitialized: boolean;
isConnecting: boolean; isConnecting: boolean;
authUrl: string;
authUrlVersion: number;
connectToShell: (options?: { forceRestart?: boolean }) => void; connectToShell: (options?: { forceRestart?: boolean }) => void;
disconnectFromShell: (options?: { suppressAutoConnect?: boolean }) => void; disconnectFromShell: (options?: { suppressAutoConnect?: boolean }) => void;
openAuthUrlInBrowser: (url?: string) => boolean;
copyAuthUrlToClipboard: (url?: string) => Promise<boolean>;
}; };

View File

@@ -1,17 +1,4 @@
import type { ProjectSession } from '../../../types/app'; import type { ProjectSession } from '../../../types/app';
import { CODEX_DEVICE_AUTH_URL } from '../constants/constants';
export function isCodexLoginCommand(command: string | null | undefined): boolean {
return typeof command === 'string' && /\bcodex\s+login\b/i.test(command);
}
export function resolveAuthUrlForDisplay(command: string | null | undefined, authUrl: string): string {
if (isCodexLoginCommand(command)) {
return CODEX_DEVICE_AUTH_URL;
}
return authUrl;
}
export function getSessionDisplayName(session: ProjectSession | null | undefined): string | null { export function getSessionDisplayName(session: ProjectSession | null | undefined): string | null {
if (!session) { if (!session) {
@@ -21,4 +8,4 @@ export function getSessionDisplayName(session: ProjectSession | null | undefined
return session.__provider === 'cursor' return session.__provider === 'cursor'
? session.name || 'Untitled Session' ? session.name || 'Untitled Session'
: session.summary || 'New Session'; : session.summary || 'New Session';
} }

File diff suppressed because it is too large Load Diff

View File

@@ -59,12 +59,8 @@ export default function Shell({
isConnected, isConnected,
isInitialized, isInitialized,
isConnecting, isConnecting,
authUrl,
authUrlVersion,
connectToShell, connectToShell,
disconnectFromShell, disconnectFromShell,
openAuthUrlInBrowser,
copyAuthUrlToClipboard,
} = useShellRuntime({ } = useShellRuntime({
selectedProject, selectedProject,
selectedSession, selectedSession,
@@ -243,15 +239,7 @@ export default function Shell({
if (minimal) { if (minimal) {
return ( return (
<> <>
<ShellMinimalView <ShellMinimalView terminalContainerRef={terminalContainerRef} />
terminalContainerRef={terminalContainerRef}
authUrl={authUrl}
authUrlVersion={authUrlVersion}
initialCommand={initialCommand}
isConnected={isConnected}
openAuthUrlInBrowser={openAuthUrlInBrowser}
copyAuthUrlToClipboard={copyAuthUrlToClipboard}
/>
<TerminalShortcutsPanel <TerminalShortcutsPanel
wsRef={wsRef} wsRef={wsRef}
terminalRef={terminalRef} terminalRef={terminalRef}

View File

@@ -1,45 +1,12 @@
import { useEffect, useMemo, useState } from 'react';
import type { RefObject } from 'react'; import type { RefObject } from 'react';
import type { AuthCopyStatus } from '../../types/types';
import { resolveAuthUrlForDisplay } from '../../utils/auth';
type ShellMinimalViewProps = { type ShellMinimalViewProps = {
terminalContainerRef: RefObject<HTMLDivElement>; terminalContainerRef: RefObject<HTMLDivElement>;
authUrl: string;
authUrlVersion: number;
initialCommand: string | null | undefined;
isConnected: boolean;
openAuthUrlInBrowser: (url: string) => boolean;
copyAuthUrlToClipboard: (url: string) => Promise<boolean>;
}; };
export default function ShellMinimalView({ export default function ShellMinimalView({
terminalContainerRef, terminalContainerRef,
authUrl,
authUrlVersion,
initialCommand,
isConnected,
openAuthUrlInBrowser,
copyAuthUrlToClipboard,
}: ShellMinimalViewProps) { }: ShellMinimalViewProps) {
const [authUrlCopyStatus, setAuthUrlCopyStatus] = useState<AuthCopyStatus>('idle');
const [isAuthPanelHidden, setIsAuthPanelHidden] = useState(false);
const displayAuthUrl = useMemo(
() => resolveAuthUrlForDisplay(initialCommand, authUrl),
[authUrl, initialCommand],
);
// Keep auth panel UI state local to minimal mode and reset it when connection/url changes.
useEffect(() => {
setAuthUrlCopyStatus('idle');
setIsAuthPanelHidden(false);
}, [authUrlVersion, displayAuthUrl, isConnected]);
const hasAuthUrl = Boolean(displayAuthUrl);
const showMobileAuthPanel = hasAuthUrl && !isAuthPanelHidden;
const showMobileAuthPanelToggle = hasAuthUrl && isAuthPanelHidden;
return ( return (
<div className="relative h-full w-full bg-gray-900"> <div className="relative h-full w-full bg-gray-900">
<div <div
@@ -47,67 +14,6 @@ export default function ShellMinimalView({
className="h-full w-full focus:outline-none" className="h-full w-full focus:outline-none"
style={{ outline: 'none' }} style={{ outline: 'none' }}
/> />
{showMobileAuthPanel && (
<div className="absolute inset-x-0 bottom-14 z-20 border-t border-gray-700/80 bg-gray-900/95 p-3 backdrop-blur-sm md:hidden">
<div className="flex flex-col gap-2">
<div className="flex items-center justify-between gap-2">
<p className="text-xs text-gray-300">Open or copy the login URL:</p>
<button
type="button"
onClick={() => setIsAuthPanelHidden(true)}
className="rounded bg-gray-700 px-2 py-1 text-[10px] font-medium uppercase tracking-wide text-gray-100 hover:bg-gray-600"
>
Hide
</button>
</div>
<input
type="text"
value={displayAuthUrl}
readOnly
onClick={(event) => event.currentTarget.select()}
className="w-full rounded border border-gray-600 bg-gray-800 px-2 py-1 text-xs text-gray-100 focus:outline-none focus:ring-1 focus:ring-blue-500"
aria-label="Authentication URL"
/>
<div className="flex items-center gap-2">
<button
type="button"
onClick={() => {
openAuthUrlInBrowser(displayAuthUrl);
}}
className="flex-1 rounded bg-blue-600 px-3 py-2 text-xs font-medium text-white hover:bg-blue-700"
>
Open URL
</button>
<button
type="button"
onClick={async () => {
const copied = await copyAuthUrlToClipboard(displayAuthUrl);
setAuthUrlCopyStatus(copied ? 'copied' : 'failed');
}}
className="flex-1 rounded bg-gray-700 px-3 py-2 text-xs font-medium text-white hover:bg-gray-600"
>
{authUrlCopyStatus === 'copied' ? 'Copied' : 'Copy URL'}
</button>
</div>
</div>
</div>
)}
{showMobileAuthPanelToggle && (
<div className="absolute bottom-14 right-3 z-20 md:hidden">
<button
type="button"
onClick={() => setIsAuthPanelHidden(false)}
className="rounded bg-gray-800/95 px-3 py-2 text-xs font-medium text-gray-100 shadow-lg backdrop-blur-sm hover:bg-gray-700"
>
Show login URL
</button>
</div>
)}
</div> </div>
); );
} }

View File

@@ -26,6 +26,7 @@ const MOBILE_KEYS: Shortcut[] = [
{ type: 'arrow', id: 'arrow-down', sequence: '\x1b[B', icon: 'down' }, { type: 'arrow', id: 'arrow-down', sequence: '\x1b[B', icon: 'down' },
{ type: 'arrow', id: 'arrow-left', sequence: '\x1b[D', icon: 'left' }, { type: 'arrow', id: 'arrow-left', sequence: '\x1b[D', icon: 'left' },
{ type: 'arrow', id: 'arrow-right', sequence: '\x1b[C', icon: 'right' }, { type: 'arrow', id: 'arrow-right', sequence: '\x1b[C', icon: 'right' },
{ type: 'key', id: 'ctrl-c', label: 'Ctrl+C', sequence: '\x03' },
]; ];
const ARROW_ICONS = { const ARROW_ICONS = {

View File

@@ -43,7 +43,7 @@ function Sidebar({
}: SidebarProps) { }: SidebarProps) {
const { t } = useTranslation(['sidebar', 'common']); const { t } = useTranslation(['sidebar', 'common']);
const { isPWA } = useDeviceSettings({ trackMobile: false }); const { isPWA } = useDeviceSettings({ trackMobile: false });
const { updateAvailable, latestVersion, currentVersion, releaseInfo, installMode } = useVersionCheck( const { updateAvailable, restartRequired, latestVersion, currentVersion, releaseInfo, installMode } = useVersionCheck(
'siteboon', 'siteboon',
'claudecodeui', 'claudecodeui',
); );
@@ -224,6 +224,7 @@ function Sidebar({
onExpand={handleExpandSidebar} onExpand={handleExpandSidebar}
onShowSettings={onShowSettings} onShowSettings={onShowSettings}
updateAvailable={updateAvailable} updateAvailable={updateAvailable}
restartRequired={restartRequired}
onShowVersionModal={() => setShowVersionModal(true)} onShowVersionModal={() => setShowVersionModal(true)}
t={t} t={t}
/> />
@@ -296,6 +297,7 @@ function Sidebar({
onCreateProject={() => setShowNewProject(true)} onCreateProject={() => setShowNewProject(true)}
onCollapseSidebar={handleCollapseSidebar} onCollapseSidebar={handleCollapseSidebar}
updateAvailable={updateAvailable} updateAvailable={updateAvailable}
restartRequired={restartRequired}
releaseInfo={releaseInfo} releaseInfo={releaseInfo}
latestVersion={latestVersion} latestVersion={latestVersion}
currentVersion={currentVersion} currentVersion={currentVersion}

View File

@@ -1,4 +1,4 @@
import { Settings, Sparkles, PanelLeftOpen, Bug } from 'lucide-react'; import { Settings, Sparkles, PanelLeftOpen, Bug, AlertTriangle } from 'lucide-react';
import type { TFunction } from 'i18next'; import type { TFunction } from 'i18next';
const DISCORD_INVITE_URL = 'https://discord.gg/buxwujPNRE'; const DISCORD_INVITE_URL = 'https://discord.gg/buxwujPNRE';
@@ -16,6 +16,7 @@ type SidebarCollapsedProps = {
onExpand: () => void; onExpand: () => void;
onShowSettings: () => void; onShowSettings: () => void;
updateAvailable: boolean; updateAvailable: boolean;
restartRequired: boolean;
onShowVersionModal: () => void; onShowVersionModal: () => void;
t: TFunction; t: TFunction;
}; };
@@ -24,6 +25,7 @@ export default function SidebarCollapsed({
onExpand, onExpand,
onShowSettings, onShowSettings,
updateAvailable, updateAvailable,
restartRequired,
onShowVersionModal, onShowVersionModal,
t, t,
}: SidebarCollapsedProps) { }: SidebarCollapsedProps) {
@@ -75,6 +77,18 @@ export default function SidebarCollapsed({
<DiscordIcon className="h-4 w-4 text-muted-foreground transition-colors group-hover:text-foreground" /> <DiscordIcon className="h-4 w-4 text-muted-foreground transition-colors group-hover:text-foreground" />
</a> </a>
{/* Restart-required indicator */}
{restartRequired && (
<div
className="relative flex h-8 w-8 items-center justify-center rounded-lg"
aria-label={t('version.restartRequired')}
title={t('version.restartRequired')}
>
<AlertTriangle className="h-4 w-4 text-amber-500" />
<span className="absolute right-1.5 top-1.5 h-1.5 w-1.5 animate-pulse rounded-full bg-amber-500" />
</div>
)}
{/* Update indicator */} {/* Update indicator */}
{updateAvailable && ( {updateAvailable && (
<button <button

View File

@@ -141,6 +141,7 @@ type SidebarContentProps = {
onCreateProject: () => void; onCreateProject: () => void;
onCollapseSidebar: () => void; onCollapseSidebar: () => void;
updateAvailable: boolean; updateAvailable: boolean;
restartRequired: boolean;
releaseInfo: ReleaseInfo | null; releaseInfo: ReleaseInfo | null;
latestVersion: string | null; latestVersion: string | null;
currentVersion: string; currentVersion: string;
@@ -178,6 +179,7 @@ export default function SidebarContent({
onCreateProject, onCreateProject,
onCollapseSidebar, onCollapseSidebar,
updateAvailable, updateAvailable,
restartRequired,
releaseInfo, releaseInfo,
latestVersion, latestVersion,
currentVersion, currentVersion,
@@ -553,6 +555,7 @@ export default function SidebarContent({
<SidebarFooter <SidebarFooter
updateAvailable={updateAvailable} updateAvailable={updateAvailable}
restartRequired={restartRequired}
releaseInfo={releaseInfo} releaseInfo={releaseInfo}
latestVersion={latestVersion} latestVersion={latestVersion}
currentVersion={currentVersion} currentVersion={currentVersion}

View File

@@ -1,4 +1,4 @@
import { Settings, ArrowUpCircle, Bug } from 'lucide-react'; import { Settings, ArrowUpCircle, Bug, AlertTriangle } from 'lucide-react';
import type { TFunction } from 'i18next'; import type { TFunction } from 'i18next';
import { IS_PLATFORM } from '../../../../constants/config'; import { IS_PLATFORM } from '../../../../constants/config';
import type { ReleaseInfo } from '../../../../types/sharedTypes'; import type { ReleaseInfo } from '../../../../types/sharedTypes';
@@ -18,6 +18,7 @@ function DiscordIcon({ className }: { className?: string }) {
type SidebarFooterProps = { type SidebarFooterProps = {
updateAvailable: boolean; updateAvailable: boolean;
restartRequired: boolean;
releaseInfo: ReleaseInfo | null; releaseInfo: ReleaseInfo | null;
latestVersion: string | null; latestVersion: string | null;
currentVersion: string; currentVersion: string;
@@ -28,6 +29,7 @@ type SidebarFooterProps = {
export default function SidebarFooter({ export default function SidebarFooter({
updateAvailable, updateAvailable,
restartRequired,
releaseInfo, releaseInfo,
latestVersion, latestVersion,
currentVersion, currentVersion,
@@ -37,6 +39,22 @@ export default function SidebarFooter({
}: SidebarFooterProps) { }: SidebarFooterProps) {
return ( return (
<div className="flex-shrink-0" style={{ paddingBottom: 'env(safe-area-inset-bottom, 0)' }}> <div className="flex-shrink-0" style={{ paddingBottom: 'env(safe-area-inset-bottom, 0)' }}>
{/* Restart-required banner: the running server version differs from the
installed/frontend version (updated but not restarted). */}
{restartRequired && (
<>
<div className="nav-divider" />
<div className="px-2 py-1.5 md:px-2 md:py-1.5">
<div className="flex items-center gap-2.5 rounded-lg border border-amber-300/60 bg-amber-50/80 px-2.5 py-2 dark:border-amber-700/40 dark:bg-amber-900/15">
<AlertTriangle className="h-4 w-4 flex-shrink-0 text-amber-500 dark:text-amber-400" />
<span className="min-w-0 flex-1 text-xs font-medium text-amber-700 dark:text-amber-300">
{t('version.restartRequired')}
</span>
</div>
</div>
</>
)}
{/* Update banner */} {/* Update banner */}
{updateAvailable && ( {updateAvailable && (
<> <>

View File

@@ -7,6 +7,7 @@ type UiPreferences = {
autoScrollToBottom: boolean; autoScrollToBottom: boolean;
sendByCtrlEnter: boolean; sendByCtrlEnter: boolean;
sidebarVisible: boolean; sidebarVisible: boolean;
voiceEnabled: boolean;
}; };
type UiPreferenceKey = keyof UiPreferences; type UiPreferenceKey = keyof UiPreferences;
@@ -39,6 +40,7 @@ const DEFAULTS: UiPreferences = {
autoScrollToBottom: true, autoScrollToBottom: true,
sendByCtrlEnter: false, sendByCtrlEnter: false,
sidebarVisible: true, sidebarVisible: true,
voiceEnabled: false,
}; };
const PREFERENCE_KEYS = Object.keys(DEFAULTS) as UiPreferenceKey[]; const PREFERENCE_KEYS = Object.keys(DEFAULTS) as UiPreferenceKey[];

View File

@@ -28,20 +28,31 @@ export const useVersionCheck = (owner: string, repo: string) => {
const [latestVersion, setLatestVersion] = useState<string | null>(null); const [latestVersion, setLatestVersion] = useState<string | null>(null);
const [releaseInfo, setReleaseInfo] = useState<ReleaseInfo | null>(null); const [releaseInfo, setReleaseInfo] = useState<ReleaseInfo | null>(null);
const [installMode, setInstallMode] = useState<InstallMode>('git'); const [installMode, setInstallMode] = useState<InstallMode>('git');
const [runningVersion, setRunningVersion] = useState<string | null>(null);
const [restartRequired, setRestartRequired] = useState(false);
useEffect(() => { useEffect(() => {
const fetchInstallMode = async () => { const fetchHealth = async () => {
try { try {
const response = await fetch('/health'); const response = await fetch('/health');
const data = await response.json(); const data = await response.json();
if (data.installMode === 'npm' || data.installMode === 'git') { if (data.installMode === 'npm' || data.installMode === 'git') {
setInstallMode(data.installMode); setInstallMode(data.installMode);
} }
// `data.version` is the version the server process is actually running.
// This module's `version` is baked into the frontend bundle at build
// time, so it reflects the installed (on-disk) package. If they differ,
// the package was updated but the server process was not restarted, and
// DB-backed actions may silently fail until it is.
if (typeof data.version === 'string' && data.version.length > 0) {
setRunningVersion(data.version);
setRestartRequired(data.version !== version);
}
} catch { } catch {
// Default to git on error // Default to git / no restart hint on error
} }
}; };
fetchInstallMode(); fetchHealth();
}, []); }, []);
useEffect(() => { useEffect(() => {
@@ -84,5 +95,5 @@ export const useVersionCheck = (owner: string, repo: string) => {
return () => clearInterval(interval); return () => clearInterval(interval);
}, [owner, repo]); }, [owner, repo]);
return { updateAvailable, latestVersion, currentVersion: version, releaseInfo, installMode }; return { updateAvailable, latestVersion, currentVersion: version, releaseInfo, installMode, runningVersion, restartRequired };
}; };

View File

@@ -0,0 +1,68 @@
import { useState } from 'react';
export type VoiceConfig = {
baseUrl: string;
apiKey: string;
sttModel: string;
ttsModel: string;
ttsVoice: string;
ttsFormat: string;
};
const STORAGE_KEY = 'voiceConfig';
export const VOICE_CONFIG_SYNC_EVENT = 'voice-config:sync';
const DEFAULTS: VoiceConfig = { baseUrl: '', apiKey: '', sttModel: '', ttsModel: '', ttsVoice: '', ttsFormat: '' };
export function readVoiceConfig(): VoiceConfig {
try {
const raw = localStorage.getItem(STORAGE_KEY);
if (!raw) return { ...DEFAULTS };
const parsed = JSON.parse(raw);
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) return { ...DEFAULTS };
const config = { ...DEFAULTS };
for (const key of Object.keys(DEFAULTS) as (keyof VoiceConfig)[]) {
if (typeof parsed[key] === 'string') config[key] = parsed[key];
}
return config;
} catch {
return { ...DEFAULTS };
}
}
// Headers the voice proxy reads to target a per-user OpenAI-compatible backend.
// Empty fields are omitted so the server's env defaults apply.
export function voiceConfigHeaders(): Record<string, string> {
if (typeof window === 'undefined') return {};
const c = readVoiceConfig();
const h: Record<string, string> = {};
if (c.apiKey) h['x-voice-api-key'] = c.apiKey;
if (c.sttModel) h['x-voice-stt-model'] = c.sttModel;
if (c.ttsModel) h['x-voice-tts-model'] = c.ttsModel;
if (c.ttsVoice) h['x-voice-tts-voice'] = c.ttsVoice;
if (c.ttsFormat.trim()) h['x-voice-tts-format'] = c.ttsFormat.trim();
return h;
}
export function useVoiceConfig() {
const [config, setConfig] = useState<VoiceConfig>(() =>
typeof window === 'undefined' ? { ...DEFAULTS } : readVoiceConfig(),
);
const update = (patch: Partial<VoiceConfig>) => {
setConfig((prev) => {
const next = { ...prev, ...patch };
try {
const stored: Partial<VoiceConfig> = { ...next };
if (next.ttsFormat.trim()) stored.ttsFormat = next.ttsFormat.trim();
else delete stored.ttsFormat;
localStorage.setItem(STORAGE_KEY, JSON.stringify(stored));
window.dispatchEvent(new Event(VOICE_CONFIG_SYNC_EVENT));
} catch {
/* ignore persistence errors */
}
return next;
});
};
return { config, update };
}

View File

@@ -115,7 +115,8 @@
"restoreSessionError": "Fehler beim Wiederherstellen der Sitzung. Bitte erneut versuchen." "restoreSessionError": "Fehler beim Wiederherstellen der Sitzung. Bitte erneut versuchen."
}, },
"version": { "version": {
"updateAvailable": "Update verfügbar" "updateAvailable": "Update verfügbar",
"restartRequired": "Update installiert zum Anwenden Server neu starten"
}, },
"search": { "search": {
"modeProjects": "Projekte", "modeProjects": "Projekte",

View File

@@ -122,6 +122,14 @@
} }
} }
}, },
"voice": {
"input": "Voice input",
"stopRecording": "Stop recording",
"transcribing": "Transcribing…",
"speak": "Read aloud",
"stopSpeaking": "Stop",
"loading": "Loading…"
},
"input": { "input": {
"placeholder": "Type / for commands, @ for files, or ask {{provider}} anything...", "placeholder": "Type / for commands, @ for files, or ask {{provider}} anything...",
"placeholderDefault": "Type your message...", "placeholderDefault": "Type your message...",

View File

@@ -50,6 +50,21 @@
"resetToDefaults": "Reset to Defaults", "resetToDefaults": "Reset to Defaults",
"cancelChanges": "Cancel Changes" "cancelChanges": "Cancel Changes"
}, },
"voiceSettings": {
"title": "Voice",
"description": "Speech-to-text input and read-aloud, via an OpenAI-compatible audio backend.",
"enable": "Enable voice",
"enableDescription": "Show the mic button and the read-aloud button on messages.",
"backendTitle": "Backend",
"backendDescription": "Point at OpenAI, Groq, or a local server (LocalAI, Speaches, Kokoro-FastAPI). Leave blank to use the server default.",
"baseUrl": "Base URL",
"apiKey": "API key",
"sttModel": "Speech-to-text model",
"ttsModel": "Text-to-speech model",
"voice": "Voice",
"format": "Audio format",
"note": "A custom base URL is called directly by your browser and must allow browser CORS requests. Leave it blank to use the server-configured backend."
},
"quickSettings": { "quickSettings": {
"title": "Quick Settings", "title": "Quick Settings",
"sections": { "sections": {
@@ -64,6 +79,7 @@
"showThinking": "Show thinking", "showThinking": "Show thinking",
"autoScrollToBottom": "Auto-scroll to bottom", "autoScrollToBottom": "Auto-scroll to bottom",
"sendByCtrlEnter": "Send by Ctrl+Enter", "sendByCtrlEnter": "Send by Ctrl+Enter",
"voiceEnabled": "Voice (mic + read aloud)",
"sendByCtrlEnterDescription": "When enabled, pressing Ctrl+Enter will send the message instead of just Enter. This is useful for IME users to avoid accidental sends.", "sendByCtrlEnterDescription": "When enabled, pressing Ctrl+Enter will send the message instead of just Enter. This is useful for IME users to avoid accidental sends.",
"dragHandle": { "dragHandle": {
"dragging": "Dragging handle", "dragging": "Dragging handle",
@@ -94,6 +110,7 @@
"appearance": "Appearance", "appearance": "Appearance",
"git": "Git", "git": "Git",
"apiTokens": "API & Tokens", "apiTokens": "API & Tokens",
"voice": "Voice",
"tasks": "Tasks", "tasks": "Tasks",
"browser": "Browser", "browser": "Browser",
"notifications": "Notifications", "notifications": "Notifications",
@@ -114,7 +131,7 @@
}, },
"sound": { "sound": {
"title": "Sound", "title": "Sound",
"description": "Play a short tone when a chat run finishes.", "description": "Play a short tone when a chat run finishes or needs tool approval.",
"enabled": "Enabled", "enabled": "Enabled",
"test": "Test sound" "test": "Test sound"
}, },

View File

@@ -115,7 +115,8 @@
"restoreSessionError": "Error restoring session. Please try again." "restoreSessionError": "Error restoring session. Please try again."
}, },
"version": { "version": {
"updateAvailable": "Update available" "updateAvailable": "Update available",
"restartRequired": "Update installed — restart the server to apply"
}, },
"search": { "search": {
"modeProjects": "Projects", "modeProjects": "Projects",

View File

@@ -115,7 +115,8 @@
"restoreSessionError": "Erreur lors de la restauration de la session. Veuillez réessayer." "restoreSessionError": "Erreur lors de la restauration de la session. Veuillez réessayer."
}, },
"version": { "version": {
"updateAvailable": "Mise à jour disponible" "updateAvailable": "Mise à jour disponible",
"restartRequired": "Mise à jour installée — redémarrez le serveur pour l'appliquer"
}, },
"search": { "search": {
"modeProjects": "Projets", "modeProjects": "Projets",

View File

@@ -115,7 +115,8 @@
"restoreSessionError": "Errore durante il ripristino della sessione. Riprova." "restoreSessionError": "Errore durante il ripristino della sessione. Riprova."
}, },
"version": { "version": {
"updateAvailable": "Aggiornamento disponibile" "updateAvailable": "Aggiornamento disponibile",
"restartRequired": "Aggiornamento installato — riavvia il server per applicarlo"
}, },
"search": { "search": {
"modeProjects": "Progetti", "modeProjects": "Progetti",

View File

@@ -114,7 +114,8 @@
"restoreSessionError": "セッションの復元でエラーが発生しました。もう一度お試しください。" "restoreSessionError": "セッションの復元でエラーが発生しました。もう一度お試しください。"
}, },
"version": { "version": {
"updateAvailable": "アップデートあり" "updateAvailable": "アップデートあり",
"restartRequired": "更新が適用されていません。サーバーを再起動してください"
}, },
"deleteConfirmation": { "deleteConfirmation": {
"deleteProject": "プロジェクトを除去", "deleteProject": "プロジェクトを除去",

View File

@@ -114,7 +114,8 @@
"restoreSessionError": "세션 복원 오류. 다시 시도해주세요." "restoreSessionError": "세션 복원 오류. 다시 시도해주세요."
}, },
"version": { "version": {
"updateAvailable": "업데이트 가능" "updateAvailable": "업데이트 가능",
"restartRequired": "업데이트가 설치됨 — 적용하려면 서버를 재시작하세요"
}, },
"deleteConfirmation": { "deleteConfirmation": {
"deleteProject": "프로젝트 제거", "deleteProject": "프로젝트 제거",

View File

@@ -115,7 +115,8 @@
"restoreSessionError": "Ошибка при восстановлении сеанса. Попробуйте снова." "restoreSessionError": "Ошибка при восстановлении сеанса. Попробуйте снова."
}, },
"version": { "version": {
"updateAvailable": "Доступно обновление" "updateAvailable": "Доступно обновление",
"restartRequired": "Обновление установлено — перезапустите сервер для применения"
}, },
"search": { "search": {
"modeProjects": "Проекты", "modeProjects": "Проекты",

View File

@@ -115,7 +115,8 @@
"restoreSessionError": "Oturum geri yüklenirken hata oluştu. Lütfen tekrar dene." "restoreSessionError": "Oturum geri yüklenirken hata oluştu. Lütfen tekrar dene."
}, },
"version": { "version": {
"updateAvailable": "Güncelleme mevcut" "updateAvailable": "Güncelleme mevcut",
"restartRequired": "Güncelleme yüklendi — uygulamak için sunucuyu yeniden başlatın"
}, },
"search": { "search": {
"modeProjects": "Projeler", "modeProjects": "Projeler",

View File

@@ -115,7 +115,8 @@
"restoreSessionError": "恢复会话时出错,请重试。" "restoreSessionError": "恢复会话时出错,请重试。"
}, },
"version": { "version": {
"updateAvailable": "有可用更新" "updateAvailable": "有可用更新",
"restartRequired": "已安装更新 — 请重启服务器以生效"
}, },
"search": { "search": {
"modeProjects": "项目", "modeProjects": "项目",

View File

@@ -114,7 +114,8 @@
"restoreSessionError": "還原工作階段時出錯,請重試。" "restoreSessionError": "還原工作階段時出錯,請重試。"
}, },
"version": { "version": {
"updateAvailable": "有可用更新" "updateAvailable": "有可用更新",
"restartRequired": "已安裝更新 — 請重新啟動伺服器以套用"
}, },
"search": { "search": {
"modeProjects": "專案", "modeProjects": "專案",

View File

@@ -137,6 +137,12 @@
height: 100%; height: 100%;
margin: 0; margin: 0;
padding: 0; padding: 0;
/* The app shell is a fixed inset-0 container (see AppContent), so the
document itself never needs to scroll. Clipping it removes the phantom
full-height page scrollbar and disables the browser pull-to-refresh
gesture that reloads the page when scrolling up on mobile. */
overflow: hidden;
overscroll-behavior-y: contain;
} }
/* Root element with safe area padding for PWA */ /* Root element with safe area padding for PWA */

60
src/lib/voiceApi.ts Normal file
View File

@@ -0,0 +1,60 @@
import { authenticatedFetch } from '../utils/api';
import { readVoiceConfig, voiceConfigHeaders } from '../hooks/useVoiceConfig';
function directUrl(baseUrl: string, path: string): string {
return `${baseUrl.replace(/\/$/, '')}${path}`;
}
export function voiceConfigSignature(): string {
return JSON.stringify(readVoiceConfig());
}
export function transcribeVoice(blob: Blob, filename: string): Promise<Response> {
const config = readVoiceConfig();
const body = new FormData();
if (config.baseUrl.trim()) {
body.append('file', blob, filename);
body.append('model', config.sttModel || 'whisper-1');
return fetch(directUrl(config.baseUrl.trim(), '/audio/transcriptions'), {
method: 'POST',
headers: config.apiKey ? { Authorization: `Bearer ${config.apiKey}` } : {},
body,
});
}
body.append('audio', blob, filename);
return authenticatedFetch('/api/voice/transcribe', {
method: 'POST',
headers: voiceConfigHeaders(),
body,
});
}
export function synthesizeVoice(text: string, signal: AbortSignal): Promise<Response> {
const config = readVoiceConfig();
if (config.baseUrl.trim()) {
return fetch(directUrl(config.baseUrl.trim(), '/audio/speech'), {
method: 'POST',
headers: {
'Content-Type': 'application/json',
...(config.apiKey ? { Authorization: `Bearer ${config.apiKey}` } : {}),
},
body: JSON.stringify({
model: config.ttsModel || 'tts-1',
voice: config.ttsVoice || 'alloy',
input: text,
...(config.ttsFormat.trim() ? { response_format: config.ttsFormat.trim() } : {}),
}),
signal,
});
}
return authenticatedFetch('/api/voice/tts', {
method: 'POST',
body: JSON.stringify({ text }),
headers: voiceConfigHeaders(),
signal,
});
}

196
src/lib/voicePlayer.ts Normal file
View File

@@ -0,0 +1,196 @@
import { synthesizeVoice, voiceConfigSignature } from './voiceApi';
// A single app-level audio player for read-aloud. It owns one <audio> element, lives
// outside the React tree, and caches generated audio by content. Because playback is not
// tied to a component, switching chats or re-rendering a message can't revoke the blob URL
// out from under it (the cause of mid-play cutoffs). v1 plays one message at a time
// (a new play replaces the current one); the design leaves room for a queue later.
export type VoicePlayState = 'idle' | 'loading' | 'playing';
export type VoiceSnapshot = { state: VoicePlayState; error: string | null };
const IDLE: VoiceSnapshot = { state: 'idle', error: null };
const CACHE_MAX = 24;
const CLIENT_TIMEOUT_MS = 330000; // backstop; the server proxy already times out at 5 min
// Stable id / cache key from the text and voice settings that affect its audio (djb2).
export function voiceId(content: string, signature = voiceConfigSignature()): string {
const input = JSON.stringify([content, signature]);
let h = 5381;
for (let i = 0; i < input.length; i++) h = (((h << 5) + h) + input.charCodeAt(i)) | 0;
return (h >>> 0).toString(36);
}
class VoicePlayer {
private audio: HTMLAudioElement | null = null;
private unlocked = false;
private cache = new Map<string, string>(); // id -> blob URL (insertion order = LRU)
private currentId: string | null = null;
private state: VoicePlayState = 'idle';
private errorId: string | null = null;
private errorMsg: string | null = null;
private token = 0; // bumps to ignore stale in-flight results
private activeController: AbortController | null = null; // aborts the in-flight TTS fetch
private errorTimer: ReturnType<typeof setTimeout> | null = null;
private listeners = new Set<() => void>();
subscribe(listener: () => void): () => void {
this.listeners.add(listener);
return () => {
this.listeners.delete(listener);
};
}
private emit() {
this.listeners.forEach((l) => l());
}
getSnapshot(id: string): VoiceSnapshot {
const state = this.currentId === id ? this.state : 'idle';
const error = this.errorId === id ? this.errorMsg : null;
if (state === 'idle' && error === null) return IDLE;
return { state, error };
}
private ensureAudio(): HTMLAudioElement {
if (!this.audio) {
const audio = new Audio();
audio.addEventListener('ended', () => this.onEnded());
audio.addEventListener('error', () => {
// Only meaningful while we believe we're playing.
if (this.state === 'playing') this.onEnded();
});
this.audio = audio;
}
return this.audio;
}
// Call synchronously from the click handler so iOS grants the (reused) element playback.
unlock() {
if (this.unlocked) return;
const audio = this.ensureAudio();
try {
const p = audio.play();
if (p && typeof p.catch === 'function') p.catch(() => {});
audio.pause();
} catch {
/* priming attempt; ignore */
}
this.unlocked = true;
}
toggle(content: string) {
const id = voiceId(content);
if (this.currentId === id && (this.state === 'playing' || this.state === 'loading')) {
this.stop();
return;
}
void this.play(id, content);
}
stop() {
this.token++; // ignore any stale in-flight result
this.abortActive(); // and actually cancel the network request
if (this.audio) this.audio.pause();
this.state = 'idle';
this.currentId = null;
this.emit();
}
private abortActive() {
if (this.activeController) {
this.activeController.abort();
this.activeController = null;
}
}
private onEnded() {
this.state = 'idle';
this.currentId = null;
this.emit();
// (queue auto-advance would hook in here)
}
private setError(id: string, msg: string) {
this.state = 'idle';
this.currentId = id;
this.errorId = id;
this.errorMsg = msg;
this.emit();
if (this.errorTimer) clearTimeout(this.errorTimer);
this.errorTimer = setTimeout(() => {
if (this.errorId === id) {
this.errorId = null;
this.errorMsg = null;
if (this.currentId === id) this.currentId = null;
this.emit();
}
}, 6000);
}
private async play(id: string, content: string) {
const audio = this.ensureAudio();
audio.pause();
this.currentId = id;
this.errorId = null;
this.errorMsg = null;
this.state = 'loading';
this.emit();
const myToken = ++this.token;
this.abortActive(); // cancel any request this play supersedes
try {
let url = this.cache.get(id);
if (!url) {
const controller = new AbortController();
this.activeController = controller;
const timer = setTimeout(() => controller.abort(), CLIENT_TIMEOUT_MS);
const res = await synthesizeVoice(content, controller.signal).finally(() => {
clearTimeout(timer);
if (this.activeController === controller) this.activeController = null;
});
if (myToken !== this.token) return; // superseded by another play/stop
if (!res.ok) {
let msg = `Read-aloud failed (${res.status})`;
try {
const j = await res.json();
if (j?.error) msg = String(j.error);
} catch {
/* non-JSON error body */
}
throw new Error(msg);
}
const blob = await res.blob();
if (myToken !== this.token) return;
url = URL.createObjectURL(blob);
this.cacheSet(id, url);
}
if (myToken !== this.token) return;
audio.src = url;
audio.load();
await audio.play();
if (myToken !== this.token) return;
this.state = 'playing';
this.emit();
} catch (e) {
if (myToken !== this.token) return;
const aborted = e instanceof Error && e.name === 'AbortError';
this.setError(id, aborted ? 'Read-aloud timed out.' : e instanceof Error ? e.message : 'Read-aloud failed');
}
}
private cacheSet(id: string, url: string) {
this.cache.set(id, url);
while (this.cache.size > CACHE_MAX) {
const oldest = this.cache.keys().next().value as string | undefined;
if (oldest === undefined) break;
const oldUrl = this.cache.get(oldest);
this.cache.delete(oldest);
if (oldUrl && oldUrl !== this.audio?.src) URL.revokeObjectURL(oldUrl);
}
}
}
export const voicePlayer = new VoicePlayer();

View File

@@ -58,7 +58,7 @@ const playTone = (
oscillator.stop(startsAt + duration + 0.02); oscillator.stop(startsAt + duration + 0.02);
}; };
export const playChatCompletionSound = async ({ force = false } = {}): Promise<void> => { export const playNotificationSound = async ({ force = false } = {}): Promise<void> => {
if (!force && !isNotificationSoundEnabled()) { if (!force && !isNotificationSoundEnabled()) {
return; return;
} }
@@ -81,3 +81,5 @@ export const playChatCompletionSound = async ({ force = false } = {}): Promise<v
console.warn('Unable to play notification sound:', error); console.warn('Unable to play notification sound:', error);
} }
}; };
export const playChatCompletionSound = (options = {}): Promise<void> => playNotificationSound(options);