Compare commits

..

6 Commits

Author SHA1 Message Date
Simos Mikelatos
fff89e6132 Merge branch 'main' into camoufox-novnc-browser-use 2026-06-26 16:07:22 +02:00
Haile
591e8e7642 fix: voice tts format settings (#919)
* feat(voice): add optional speech-to-text input and read-aloud TTS

Adds a push-to-talk mic button in the composer and a read-aloud button on
assistant messages. Both are opt-in and hidden unless a voice backend is
configured via VOICE_SIDECAR_URL.

The auth-gated /api/voice proxy forwards to a configurable backend exposing
/transcribe and /tts (provider-agnostic); the frontend probes /api/voice/health
and hides the controls when disabled. Adds i18n keys and docs/voice.md.

Includes a local, no-API-key reference backend in voice-sidecar/ (faster-whisper
for STT, Kokoro-82M for TTS, both CPU-capable).

* refactor(voice): provider-agnostic backend and in-app config

Switches the voice proxy to the OpenAI audio API (/v1/audio/transcriptions and
/v1/audio/speech) so it works with OpenAI, Groq, or a local server. Adds a
Settings -> Voice tab (base URL, API key, models, voice) plus a Quick Settings
toggle, and removes the bundled Python sidecar.

Review fixes: stop mic tracks on unmount, clear the global TTS stop handler and
revoke leaked blob URLs, add fetch timeouts in the proxy, surface mic errors in
the button, trim before appending transcripts, and drop the repo-wide wav ignore.

* fix(voice): relax backend timeout and surface timeout errors

Bumps the proxy timeout to 5 minutes (VOICE_TIMEOUT_MS) since local TTS can
synthesize long messages at roughly real-time, and returns a clear timed-out
message (504) instead of failing silently. The read-aloud button now shows
backend errors.

* fix(voice): play read-aloud through an app-level player to stop cutoffs

Read-aloud now runs in a single module-level player outside the React tree instead
of per-message component state. Switching chats or re-rendering a message no longer
revokes the blob URL mid-play (the 'Invalid URI' cutoff). Adds content-keyed caching so
re-listening doesn't regenerate, and reuses one audio element (also unlocks iOS once).

* fix(voice): address review (SSRF guard, auth mapping, client timeout)

Validates the user-supplied backend URL (http/https only, blocks the link-local
metadata range) to prevent SSRF; remaps upstream 401/403 so a bad voice API key
isn't read as the app's own auth failing; adds a client-side AbortController timeout
on the read-aloud request so the button can't sit in loading if a request stalls.

* docs(voice): provider-agnostic wording and jsdoc on proxy functions

drop leftover sidecar/faster-whisper references now that the backend is any
openai-compatible voice api, and add jsdoc to the voice-proxy functions so the
docstring coverage check passes.

* fix(voice): harden timeout parsing, tts input check, and player abort

- fall back to the default when VOICE_TIMEOUT_MS is non-numeric or <= 0, so a
  bad override can't make the abort fire immediately
- type-check the tts `text` before calling .trim() so a non-string body returns
  400 instead of throwing
- abort the in-flight TTS fetch on stop() and on a superseding play, so tapping
  read-aloud repeatedly doesn't leave orphaned requests generating audio

* feat(voice): send transcript with the main send button while recording

while dictating, the main send button stops recording, transcribes, and sends
in one tap, matching the codex-style flow. the mic button still stops and drops
the transcript into the input box to edit before sending. voice recording state
is lifted into the composer so both buttons share it, and the send button is
enabled (not grayed) while recording. also fix a pre-existing type error: the
quick-settings preferences map was missing voiceEnabled.

* fix(voice): make stop() idempotent so a double tap can't throw

guard on the recorder's own state instead of react state, so a double tap or
the mic and send buttons both firing won't call stop() on an already-inactive
MediaRecorder.

* fix(voice): expose TTS format in user settings

* fix(voice): harden recording and backend behavior

Redirects could bypass the backend URL guard, and TTS playback waited for full buffering.

Recording could overlap or finish after teardown. Controls also ignored backend readiness.

Explicit formats and config-aware cache keys prevent stale audio after settings change.

* fix(voice): validate config and request boundaries

Malformed stored settings could break voice requests instead of using safe defaults.

Health results could outlive auth changes. URL checks also did not guard the fetch sink.

Remove constant recorder branches so lifecycle cancellation stays clear.

* fix(voice): separate client and server backends

User-selected backend URLs must remain usable without letting clients control server requests.

Call custom providers from the browser while keeping the server proxy bound to its configured host.

This restores voice controls for frontend settings without reopening the SSRF path.

* fix: hide voice options until enabled

---------

Co-authored-by: newsbubbles <nathaniel.gibson@gmail.com>
Co-authored-by: Simos Mikelatos <simosmik@gmail.com>
2026-06-26 16:06:40 +02:00
Simos Mikelatos
8adcdaa0e5 Merge branch 'main' into camoufox-novnc-browser-use 2026-06-24 23:16:13 +02:00
Simos Mikelatos
0610cc8333 fix: browser use set profile root folder 2026-06-24 19:21:52 +00:00
Simos Mikelatos
9457651fdd fix(browser-use): harden browser settings state 2026-06-24 15:36:25 +00:00
Simos Mikelatos
8c31ebcc63 feat(browser-use): add Camoufox noVNC session viewer 2026-06-24 14:39:41 +00:00
14 changed files with 1321 additions and 183 deletions

View File

@@ -69,7 +69,7 @@ const sessionIdSchema = {
const tools: ToolDefinition[] = [ const tools: ToolDefinition[] = [
{ {
name: 'browser_create_session', name: 'browser_create_session',
description: 'Create a temporary Browser session that the agent can control. Optionally provide a background profileName to reuse cookies and storage.', description: 'Create a Browser session that the agent can control. Provide profileName to use a specific persistent profile; when omitted, the configured persistent profile is used only if session persistence is enabled, otherwise a temporary session is created.',
inputSchema: { inputSchema: {
type: 'object', type: 'object',
properties: { properties: {

View File

@@ -64,7 +64,7 @@ import providerRoutes from './modules/providers/provider.routes.js';
import voiceRoutes from './voice-proxy.js'; import voiceRoutes from './voice-proxy.js';
import browserUseRoutes from './modules/browser-use/browser-use.routes.js'; import browserUseRoutes from './modules/browser-use/browser-use.routes.js';
import browserUseMcpRoutes from './modules/browser-use/browser-use-mcp.routes.js'; import browserUseMcpRoutes from './modules/browser-use/browser-use-mcp.routes.js';
import { browserUseService } from './modules/browser-use/browser-use.service.js'; import { browserUseService, VIEWER_COOKIE_NAME } from './modules/browser-use/index.js';
import { startEnabledPluginServers, stopAllPlugins, getPluginPort } from './utils/plugin-process-manager.js'; import { startEnabledPluginServers, stopAllPlugins, getPluginPort } from './utils/plugin-process-manager.js';
import { initializeDatabase, projectsDb, sessionsDb } from './modules/database/index.js'; import { initializeDatabase, projectsDb, sessionsDb } from './modules/database/index.js';
import { configureWebPush } from './services/vapid-keys.js'; import { configureWebPush } from './services/vapid-keys.js';
@@ -146,6 +146,8 @@ const wss = createWebSocketServer(server, {
shouldAutoOpenUrlFromOutput, shouldAutoOpenUrlFromOutput,
}, },
getPluginPort, getPluginPort,
browserUseViewer: (ws, pathname) => browserUseService.handleViewerWebSocket(ws, pathname),
authenticateBrowserUseViewer: authenticateBrowserUseViewerPath,
}); });
// Make WebSocket server available to routes // Make WebSocket server available to routes
@@ -211,11 +213,42 @@ app.use('/api/gemini', authenticateToken, geminiRoutes);
// Plugins API Routes (protected) // Plugins API Routes (protected)
app.use('/api/plugins', authenticateToken, pluginsRoutes); app.use('/api/plugins', authenticateToken, pluginsRoutes);
function readCookieValue(header, name) {
if (!header) return null;
const prefix = `${name}=`;
const cookie = String(header).split(';').map((part) => part.trim()).find((part) => part.startsWith(prefix));
return cookie ? decodeURIComponent(cookie.slice(prefix.length)) : null;
}
function authenticateBrowserUseViewerPath(pathname, token) {
const parts = String(pathname || '').split('/');
const sessionId = parts[4];
if (parts[1] !== 'api' || parts[2] !== 'browser-use' || parts[3] !== 'sessions' || parts[5] !== 'viewer' || parts[6] !== 'websockify') {
return false;
}
return browserUseService.validateViewerToken(decodeURIComponent(sessionId), token);
}
function authenticateBrowserUse(req, res, next) {
const match = /^\/sessions\/([^/]+)\/viewer(?:\/|$)/.exec(req.path || '');
if (match) {
const sessionId = decodeURIComponent(match[1]);
const token = typeof req.query.viewerToken === 'string'
? req.query.viewerToken
: readCookieValue(req.headers.cookie, VIEWER_COOKIE_NAME);
if (browserUseService.validateViewerToken(sessionId, token)) {
return next();
}
return res.status(401).json({ error: 'Browser viewer access requires a valid session token.' });
}
return authenticateToken(req, res, next);
}
// Browser MCP bridge API (local token protected) // Browser MCP bridge API (local token protected)
app.use('/api/browser-use-mcp', browserUseMcpRoutes); app.use('/api/browser-use-mcp', browserUseMcpRoutes);
// Browser API Routes (protected) // Browser API Routes (protected)
app.use('/api/browser-use', authenticateToken, browserUseRoutes); app.use('/api/browser-use', authenticateBrowserUse, browserUseRoutes);
// Unified provider MCP routes (protected) // Unified provider MCP routes (protected)
app.use('/api/providers', authenticateToken, providerRoutes); app.use('/api/providers', authenticateToken, providerRoutes);

View File

@@ -1,6 +1,7 @@
import express from 'express'; import express from 'express';
import { browserUseService } from '@/modules/browser-use/browser-use.service.js'; import { browserUseService } from '@/modules/browser-use/browser-use.service.js';
import { VIEWER_COOKIE_NAME, VIEWER_TOKEN_TTL_MS } from '@/modules/browser-use/browser-use.viewer.js';
const router = express.Router(); const router = express.Router();
@@ -8,6 +9,45 @@ function readParam(value: string | string[] | undefined): string {
return Array.isArray(value) ? value[0] || '' : value || ''; return Array.isArray(value) ? value[0] || '' : value || '';
} }
const SAFE_VIEWER_ROOT_FILES = new Set(['vnc.html', 'favicon.ico', 'manifest.json']);
const SAFE_VIEWER_ROOT_DIRS = new Set(['app', 'core', 'vendor', 'assets', 'images', 'utils']);
function isSafeViewerPath(viewerPath: string): boolean {
if (!viewerPath || viewerPath.startsWith('/') || viewerPath.includes('..') || viewerPath.includes('\\')) {
return false;
}
if (!/^[A-Za-z0-9][A-Za-z0-9._~/-]*$/.test(viewerPath)) {
return false;
}
if (SAFE_VIEWER_ROOT_FILES.has(viewerPath)) {
return true;
}
const [rootDir] = viewerPath.split('/');
return Boolean(rootDir && SAFE_VIEWER_ROOT_DIRS.has(rootDir));
}
function isSecureRequest(req: express.Request): boolean {
const forwardedProto = String(req.headers['x-forwarded-proto'] || '')
.split(',')[0]
.trim()
.toLowerCase();
return req.secure || forwardedProto === 'https';
}
function readQueryString(originalUrl: string): string {
const queryIndex = originalUrl.indexOf('?');
if (queryIndex < 0) {
return '';
}
const params = new URLSearchParams(originalUrl.slice(queryIndex + 1));
params.delete('viewerToken');
const nextQuery = params.toString();
return nextQuery ? `?${nextQuery}` : '';
}
router.get('/status', async (_req, res) => { router.get('/status', async (_req, res) => {
try { try {
res.json({ success: true, data: await browserUseService.getStatus() }); res.json({ success: true, data: await browserUseService.getStatus() });
@@ -62,13 +102,60 @@ router.get('/sessions', async (_req, res) => {
try { try {
res.json({ success: true, data: { sessions: await browserUseService.listSessions() } }); res.json({ success: true, data: { sessions: await browserUseService.listSessions() } });
} catch (error) { } catch (error) {
res.status(401).json({ res.status(500).json({
success: false, success: false,
error: error instanceof Error ? error.message : 'Failed to list browser sessions.', error: error instanceof Error ? error.message : 'Failed to list browser sessions.',
}); });
} }
}); });
router.get('/sessions/:sessionId/viewer/*', async (req, res) => {
try {
const sessionId = readParam(req.params.sessionId);
const originalPath = req.originalUrl.split('?')[0] || '';
const viewerMarker = `/sessions/${sessionId}/viewer/`;
const markerIndex = originalPath.indexOf(viewerMarker);
const rawViewerPath = markerIndex >= 0 ? originalPath.slice(markerIndex + viewerMarker.length) : 'vnc.html';
const viewerPath = decodeURIComponent(rawViewerPath).replace(/^\/+/, '') || 'vnc.html';
if (!isSafeViewerPath(viewerPath)) {
res.status(400).json({ success: false, error: 'Invalid Browser viewer path.' });
return;
}
const viewerToken = readParam(req.query.viewerToken as string | string[] | undefined);
if (viewerPath === 'vnc.html' && browserUseService.validateViewerToken(sessionId, viewerToken)) {
res.cookie(VIEWER_COOKIE_NAME, viewerToken, {
httpOnly: true,
sameSite: 'lax',
secure: isSecureRequest(req),
maxAge: VIEWER_TOKEN_TTL_MS,
path: '/api/browser-use/sessions/' + encodeURIComponent(sessionId) + '/viewer',
});
}
const target = browserUseService.getViewerProxyTarget(sessionId);
const query = readQueryString(req.originalUrl);
const upstream = await fetch(`http://127.0.0.1:${target.websockifyPort}/${viewerPath}${query}`, {
headers: {
accept: String(req.headers.accept || '*/*'),
},
});
const contentType = upstream.headers.get('content-type');
if (contentType) {
res.setHeader('content-type', contentType);
}
const cacheControl = viewerPath === 'vnc.html' ? 'no-store' : 'public, max-age=3600';
res.setHeader('cache-control', cacheControl);
res.status(upstream.status);
const body = Buffer.from(await upstream.arrayBuffer());
res.send(body);
} catch (error) {
res.status(404).json({
success: false,
error: error instanceof Error ? error.message : 'Browser viewer is not available.',
});
}
});
router.post('/sessions/:sessionId/stop', async (req, res) => { router.post('/sessions/:sessionId/stop', async (req, res) => {
try { try {
const result = await browserUseService.stopSession(readParam(req.params.sessionId)); const result = await browserUseService.stopSession(readParam(req.params.sessionId));

View File

@@ -1,128 +1,86 @@
import { createRequire } from 'node:module'; import { createRequire } from 'node:module';
import { randomBytes, randomUUID } from 'node:crypto'; import { randomUUID } from 'node:crypto';
import { spawn } from 'node:child_process'; import { execFileSync, spawn } from 'node:child_process';
import fs from 'node:fs'; import fs from 'node:fs';
import net from 'node:net';
import os from 'node:os'; import os from 'node:os';
import path from 'node:path'; import path from 'node:path';
import { appConfigDb } from '@/modules/database/index.js'; import { WebSocket } from 'ws';
import { providerMcpService } from '@/modules/providers/index.js'; import { providerMcpService } from '@/modules/providers/index.js';
import { getModuleDir } from '@/utils/runtime-paths.js'; import { getModuleDir } from '@/utils/runtime-paths.js';
import {
getOrCreateMcpToken,
getProfilePath,
normalizeBrowserBackend,
PROFILE_ROOT,
readSettings,
resolveSessionProfileName,
useVisibleCamoufoxBackend,
writeSettings,
} from './browser-use.settings.js';
import type {
BrowserUseSession,
BrowserUseSettings,
PublicBrowserUseSession,
RuntimeHandle,
RuntimeProbe,
RuntimeReadiness,
} from './browser-use.types.js';
import { getViewerUrl, handleViewerWebSocket, VIEWER_TOKEN_TTL_MS } from './browser-use.viewer.js';
const require = createRequire(import.meta.url); const require = createRequire(import.meta.url);
const __dirname = getModuleDir(import.meta.url); const __dirname = getModuleDir(import.meta.url);
const IS_PLATFORM = process.env.VITE_IS_PLATFORM === 'true'; const IS_PLATFORM = process.env.VITE_IS_PLATFORM === 'true';
const MAX_SESSIONS_PER_OWNER = Number.parseInt(process.env.CLOUDCLI_BROWSER_USE_MAX_SESSIONS_PER_OWNER || '3', 10); const MAX_SESSIONS_PER_OWNER = Number.parseInt(process.env.CLOUDCLI_BROWSER_USE_MAX_SESSIONS_PER_OWNER || '3', 10);
const SESSION_TTL_MS = Number.parseInt(process.env.CLOUDCLI_BROWSER_USE_SESSION_TTL_MS || String(30 * 60 * 1000), 10); const SESSION_TTL_MS = Number.parseInt(process.env.CLOUDCLI_BROWSER_USE_SESSION_TTL_MS || String(30 * 60 * 1000), 10);
const BROWSER_USE_SETTINGS_KEY = 'browser_use_settings';
const BROWSER_USE_MCP_TOKEN_KEY = 'browser_use_mcp_token';
type BrowserUseRuntime = 'cloud' | 'local';
type BrowserUseSessionStatus = 'ready' | 'stopped' | 'unavailable';
type BrowserUseSession = {
id: string;
ownerId: string;
createdBy: 'agent';
runtime: BrowserUseRuntime;
status: BrowserUseSessionStatus;
url: string | null;
title: string | null;
screenshotDataUrl: string | null;
createdAt: string;
updatedAt: string;
lastAction: string | null;
message: string | null;
profileName: string | null;
viewport: {
width: number;
height: number;
} | null;
cursor: {
x: number;
y: number;
actor: 'agent';
} | null;
};
type PublicBrowserUseSession = Omit<BrowserUseSession, 'ownerId'>;
type RuntimeHandle = {
browser?: any;
context?: any;
page?: any;
};
type BrowserUseSettings = {
enabled: boolean;
};
type RuntimeReadiness = {
playwright: any | null;
playwrightInstalled: boolean;
chromiumInstalled: boolean;
chromiumExecutablePath: string | null;
installInProgress: boolean;
installMessage: string | null;
};
type RuntimeProbe = Omit<RuntimeReadiness, 'installInProgress' | 'installMessage'>;
const sessions = new Map<string, BrowserUseSession>(); const sessions = new Map<string, BrowserUseSession>();
const handles = new Map<string, RuntimeHandle>(); const handles = new Map<string, RuntimeHandle>();
const reservedDisplays = new Set<string>();
const viewerTokens = new Map<string, { token: string; expiresAt: number }>();
let installPromise: Promise<{ success: boolean; message: string }> | null = null; let installPromise: Promise<{ success: boolean; message: string }> | null = null;
let lastInstallMessage: string | null = null; let lastInstallMessage: string | null = null;
let runtimeProbeCache: { value: RuntimeProbe; updatedAt: number } | null = null; let runtimeProbeCache: { value: RuntimeProbe; updatedAt: number } | null = null;
const DEFAULT_SETTINGS: BrowserUseSettings = {
enabled: false,
};
const AGENT_OWNER_ID = 'agent'; const AGENT_OWNER_ID = 'agent';
const PROFILE_ROOT = path.join(os.homedir(), '.cloudcli', 'browser-use', 'profiles');
const MCP_SERVER_NAME = 'cloudcli-browser'; const MCP_SERVER_NAME = 'cloudcli-browser';
const LEGACY_MCP_SERVER_NAMES = ['cloudcli-browser-use']; const LEGACY_MCP_SERVER_NAMES = ['cloudcli-browser-use'];
const RUNTIME_READINESS_CACHE_TTL_MS = 30_000; const RUNTIME_READINESS_CACHE_TTL_MS = 30_000;
const VISIBLE_BROWSER_ENABLED = process.env.CLOUDCLI_BROWSER_USE_VISIBLE !== 'false';
const RUNTIME_ROOT = process.env.CLOUDCLI_BROWSER_USE_RUNTIME_ROOT || '/opt/claudecodeui/.runtime-browser';
const NOVNC_ROOT = process.env.CLOUDCLI_BROWSER_USE_NOVNC_ROOT || path.join(RUNTIME_ROOT, 'novnc');
const X11VNC_BIN = process.env.CLOUDCLI_BROWSER_USE_X11VNC_BIN || path.join(RUNTIME_ROOT, 'rootfs/usr/bin/x11vnc');
const X11VNC_LIB_DIR = process.env.CLOUDCLI_BROWSER_USE_X11VNC_LIB_DIR || path.join(RUNTIME_ROOT, 'rootfs/usr/lib/x86_64-linux-gnu');
const X11VNC_EXTRA_LIB_DIR = process.env.CLOUDCLI_BROWSER_USE_X11VNC_EXTRA_LIB_DIR || path.join(RUNTIME_ROOT, 'rootfs/lib/x86_64-linux-gnu');
const LOG_RUNTIME_PROCESS_OUTPUT = process.env.CLOUDCLI_BROWSER_USE_RUNTIME_LOGS === 'true';
function getRuntime(): BrowserUseRuntime { function getRuntime(): 'cloud' | 'local' {
return IS_PLATFORM ? 'cloud' : 'local'; return IS_PLATFORM ? 'cloud' : 'local';
} }
function readSettings(): BrowserUseSettings { function getCamoufoxExecutablePath(): string | null {
const configured = process.env.CLOUDCLI_BROWSER_USE_CAMOUFOX_EXECUTABLE;
if (configured && fs.existsSync(configured)) {
return configured;
}
try { try {
const raw = appConfigDb.get(BROWSER_USE_SETTINGS_KEY); const output = execFileSync(path.join(os.homedir(), '.local/bin/camoufox'), ['path'], {
if (!raw) { encoding: 'utf8',
return DEFAULT_SETTINGS; stdio: ['ignore', 'pipe', 'ignore'],
} }).trim();
const executablePath = fs.statSync(output).isDirectory()
const parsed = JSON.parse(raw) as Partial<BrowserUseSettings>; ? path.join(output, 'camoufox')
return { : output;
enabled: parsed.enabled === true, return fs.existsSync(executablePath) ? executablePath : null;
}; } catch {
} catch (error: any) { return null;
console.warn('[Browser] Failed to read settings:', error?.message || error);
return DEFAULT_SETTINGS;
} }
} }
function writeSettings(settings: BrowserUseSettings): BrowserUseSettings {
const normalized = {
enabled: settings.enabled === true,
};
appConfigDb.set(BROWSER_USE_SETTINGS_KEY, JSON.stringify(normalized));
return normalized;
}
function getOrCreateMcpToken(): string {
const existing = appConfigDb.get(BROWSER_USE_MCP_TOKEN_KEY);
if (existing) {
return existing;
}
const token = randomBytes(32).toString('hex');
appConfigDb.set(BROWSER_USE_MCP_TOKEN_KEY, token);
return token;
}
function getSetupMessage(settings: BrowserUseSettings, readiness: RuntimeReadiness): string { function getSetupMessage(settings: BrowserUseSettings, readiness: RuntimeReadiness): string {
if (!settings.enabled) { if (!settings.enabled) {
return 'Browser is disabled in settings.'; return 'Browser is disabled in settings.';
@@ -132,6 +90,26 @@ function getSetupMessage(settings: BrowserUseSettings, readiness: RuntimeReadine
return 'Install Playwright and Chromium to use browser sessions.'; return 'Install Playwright and Chromium to use browser sessions.';
} }
if (settings.browserBackend === 'camoufox-vnc' && !getCamoufoxExecutablePath()) {
return 'Camoufox is selected, but Camoufox is not installed.';
}
if (useVisibleCamoufoxBackend(settings)) {
if (!VISIBLE_BROWSER_ENABLED) {
return 'Camoufox is selected, but visible browser sessions are disabled.';
}
if (!getCamoufoxExecutablePath()) {
return 'Camoufox is selected, but Camoufox is not installed.';
}
if (!fs.existsSync(X11VNC_BIN)) {
return 'Camoufox is selected, but x11vnc is missing.';
}
if (!fs.existsSync(path.join(NOVNC_ROOT, 'vnc.html'))) {
return 'Camoufox is selected, but noVNC is missing.';
}
return readiness.installMessage || 'Camoufox runtime is not ready.';
}
if (!readiness.chromiumInstalled) { if (!readiness.chromiumInstalled) {
return 'Playwright is installed, but Chromium is missing. Install the Chromium runtime to continue.'; return 'Playwright is installed, but Chromium is missing. Install the Chromium runtime to continue.';
} }
@@ -176,24 +154,6 @@ async function removeMcpServerFromAllProviders(name: string) {
return results.map((result) => ({ ...result, name })); return results.map((result) => ({ ...result, name }));
} }
function normalizeProfileName(profileName?: string | null): string | null {
const normalized = String(profileName || '').trim();
if (!normalized) {
return null;
}
return normalized.slice(0, 80);
}
function getProfilePath(profileName: string): string {
const safeName = profileName
.toLowerCase()
.replace(/[^a-z0-9._-]+/g, '-')
.replace(/^-+|-+$/g, '')
.slice(0, 80) || 'default';
return path.join(PROFILE_ROOT, safeName);
}
function probeRuntime(): RuntimeProbe { function probeRuntime(): RuntimeProbe {
const playwright = getPlaywright(); const playwright = getPlaywright();
const readiness: RuntimeProbe = { const readiness: RuntimeProbe = {
@@ -238,6 +198,175 @@ function getRuntimeReadiness(options: { force?: boolean } = {}): RuntimeReadines
}; };
} }
function findAvailablePort(): Promise<number> {
return new Promise((resolve, reject) => {
const server = net.createServer();
server.on('error', reject);
server.listen(0, '127.0.0.1', () => {
const address = server.address();
server.close(() => {
if (typeof address === 'object' && address?.port) {
resolve(address.port);
} else {
reject(new Error('Failed to reserve a browser runtime port.'));
}
});
});
});
}
function delay(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function isRuntimeProcessAlive(child: ReturnType<typeof spawn>): boolean {
return child.exitCode === null && child.signalCode === null && !child.killed;
}
function assertRuntimeProcessesAlive(processes: Array<ReturnType<typeof spawn>>, label: string) {
const exited = processes.find((child) => !isRuntimeProcessAlive(child));
if (exited) {
throw new Error(`${label} exited before the Browser viewer runtime was ready.`);
}
}
async function isPortListening(port: number): Promise<boolean> {
return new Promise((resolve) => {
const socket = net.createConnection({ host: '127.0.0.1', port });
let settled = false;
const finish = (listening: boolean) => {
if (settled) {
return;
}
settled = true;
socket.destroy();
resolve(listening);
};
socket.setTimeout(250);
socket.once('connect', () => finish(true));
socket.once('timeout', () => finish(false));
socket.once('error', () => finish(false));
});
}
async function waitForRuntimePort(
port: number,
label: string,
processes: Array<ReturnType<typeof spawn>>,
timeoutMs = 5_000,
) {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
assertRuntimeProcessesAlive(processes, label);
if (await isPortListening(port)) {
return;
}
await delay(100);
}
assertRuntimeProcessesAlive(processes, label);
throw new Error(`${label} did not start listening on 127.0.0.1:${port}.`);
}
function killRuntimeProcesses(processes?: Array<ReturnType<typeof spawn>>) {
processes?.forEach((child) => child.kill('SIGTERM'));
}
function reserveDisplay(): string {
for (let index = 90; index < 140; index += 1) {
const display = `:${index}`;
if (!reservedDisplays.has(display)) {
reservedDisplays.add(display);
return display;
}
}
throw new Error('No browser display slots are available.');
}
function spawnRuntimeProcess(command: string, args: string[], options: { env?: NodeJS.ProcessEnv } = {}) {
const child = spawn(command, args, {
env: { ...process.env, ...options.env },
stdio: ['ignore', 'ignore', 'pipe'],
});
child.stderr?.on('data', (chunk) => {
if (!LOG_RUNTIME_PROCESS_OUTPUT) {
return;
}
const text = String(chunk).trim();
if (text) {
console.warn(`[Browser runtime] ${path.basename(command)}: ${text}`);
}
});
child.on('error', (error) => {
console.warn(`[Browser runtime] ${path.basename(command)} failed:`, error.message);
});
return child;
}
async function startVisibleRuntime(): Promise<NonNullable<RuntimeHandle['viewer']> & { processes: Array<ReturnType<typeof spawn>> }> {
const display = reserveDisplay();
const vncPort = await findAvailablePort();
const websockifyPort = await findAvailablePort();
const processes: Array<ReturnType<typeof spawn>> = [];
try {
processes.push(spawnRuntimeProcess('Xvfb', [
display,
'-screen',
'0',
'1440x900x24',
'-ac',
'-nolisten',
'tcp',
]));
await delay(700);
assertRuntimeProcessesAlive(processes, 'Xvfb');
if (!fs.existsSync(X11VNC_BIN)) {
throw new Error(`x11vnc is missing at ${X11VNC_BIN}.`);
}
processes.push(spawnRuntimeProcess(X11VNC_BIN, [
'-display',
display,
'-localhost',
'-forever',
'-shared',
'-rfbport',
String(vncPort),
'-nopw',
'-quiet',
], {
env: {
LD_LIBRARY_PATH: `${X11VNC_LIB_DIR}:${X11VNC_EXTRA_LIB_DIR}:${process.env.LD_LIBRARY_PATH || ''}`,
},
}));
await waitForRuntimePort(vncPort, 'x11vnc', processes);
if (!fs.existsSync(path.join(NOVNC_ROOT, 'vnc.html'))) {
throw new Error(`noVNC is missing at ${NOVNC_ROOT}.`);
}
processes.push(spawnRuntimeProcess(path.join(os.homedir(), '.local/bin/websockify'), [
'--web',
NOVNC_ROOT,
`127.0.0.1:${websockifyPort}`,
`127.0.0.1:${vncPort}`,
]));
await waitForRuntimePort(websockifyPort, 'websockify', processes);
return {
display,
vncPort,
websockifyPort,
noVncRoot: NOVNC_ROOT,
processes,
};
} catch (error) {
killRuntimeProcesses(processes);
reservedDisplays.delete(display);
throw error;
}
}
const INSTALL_COMMAND_TIMEOUT_MS = Number.parseInt( const INSTALL_COMMAND_TIMEOUT_MS = Number.parseInt(
process.env.CLOUDCLI_BROWSER_USE_INSTALL_TIMEOUT_MS || String(10 * 60 * 1000), process.env.CLOUDCLI_BROWSER_USE_INSTALL_TIMEOUT_MS || String(10 * 60 * 1000),
10, 10,
@@ -350,6 +479,45 @@ function publicSession(session: BrowserUseSession): PublicBrowserUseSession {
return publicFields; return publicFields;
} }
function getSessionViewer(sessionId: string): RuntimeHandle['viewer'] | null {
const session = sessions.get(sessionId);
if (!session || session.ownerId !== AGENT_OWNER_ID || session.status !== 'ready') {
return null;
}
return handles.get(sessionId)?.viewer || null;
}
function createViewerToken(sessionId: string): string {
const token = randomUUID();
viewerTokens.set(sessionId, {
token,
expiresAt: Date.now() + VIEWER_TOKEN_TTL_MS,
});
return token;
}
function deleteViewerToken(sessionId: string) {
viewerTokens.delete(sessionId);
}
function validateViewerTokenForSession(sessionId: string, token: string | null | undefined): boolean {
if (!token) {
return false;
}
const session = sessions.get(sessionId);
const viewer = session?.ownerId === AGENT_OWNER_ID && session.status === 'ready'
? handles.get(sessionId)?.viewer || null
: null;
const stored = viewerTokens.get(sessionId);
if (!viewer || !stored || stored.token !== token || stored.expiresAt < Date.now()) {
if (stored?.expiresAt && stored.expiresAt < Date.now()) {
viewerTokens.delete(sessionId);
}
return false;
}
return true;
}
function ownerSessions(ownerId: string): BrowserUseSession[] { function ownerSessions(ownerId: string): BrowserUseSession[] {
return [...sessions.values()].filter((session) => session.ownerId === ownerId); return [...sessions.values()].filter((session) => session.ownerId === ownerId);
} }
@@ -357,8 +525,13 @@ function ownerSessions(ownerId: string): BrowserUseSession[] {
async function closeHandle(sessionId: string): Promise<void> { async function closeHandle(sessionId: string): Promise<void> {
const handle = handles.get(sessionId); const handle = handles.get(sessionId);
handles.delete(sessionId); handles.delete(sessionId);
deleteViewerToken(sessionId);
await handle?.context?.close?.().catch(() => undefined); await handle?.context?.close?.().catch(() => undefined);
await handle?.browser?.close().catch(() => undefined); await handle?.browser?.close().catch(() => undefined);
killRuntimeProcesses(handle?.processes);
if (handle?.viewer?.display) {
reservedDisplays.delete(handle.viewer.display);
}
} }
async function expireStaleSessions(now = Date.now()): Promise<void> { async function expireStaleSessions(now = Date.now()): Promise<void> {
@@ -424,6 +597,11 @@ export const browserUseService = {
const current = readSettings(); const current = readSettings();
const nextSettings = { const nextSettings = {
enabled: typeof settings.enabled === 'boolean' ? settings.enabled : current.enabled, enabled: typeof settings.enabled === 'boolean' ? settings.enabled : current.enabled,
persistSessions: typeof settings.persistSessions === 'boolean' ? settings.persistSessions : current.persistSessions,
defaultProfileName: typeof settings.defaultProfileName === 'string'
? settings.defaultProfileName
: current.defaultProfileName,
browserBackend: settings.browserBackend ? normalizeBrowserBackend(settings.browserBackend) : current.browserBackend,
}; };
const next = writeSettings(nextSettings); const next = writeSettings(nextSettings);
@@ -439,14 +617,28 @@ export const browserUseService = {
async getStatus() { async getStatus() {
const settings = readSettings(); const settings = readSettings();
const readiness = getRuntimeReadiness(); const readiness = getRuntimeReadiness();
const available = settings.enabled && readiness.playwrightInstalled && readiness.chromiumInstalled; const useVisibleBackend = useVisibleCamoufoxBackend(settings);
const visibleCamoufoxReady = useVisibleBackend
&& VISIBLE_BROWSER_ENABLED
&& readiness.playwrightInstalled
&& Boolean(getCamoufoxExecutablePath())
&& fs.existsSync(X11VNC_BIN)
&& fs.existsSync(path.join(NOVNC_ROOT, 'vnc.html'));
const available = settings.enabled
&& readiness.playwrightInstalled
&& (useVisibleBackend ? visibleCamoufoxReady : readiness.chromiumInstalled);
return { return {
enabled: settings.enabled, enabled: settings.enabled,
runtime: getRuntime(), runtime: getRuntime(),
backend: useVisibleBackend ? 'camoufox-vnc' : 'playwright',
browserBackend: settings.browserBackend,
available, available,
playwrightInstalled: readiness.playwrightInstalled, playwrightInstalled: readiness.playwrightInstalled,
chromiumInstalled: readiness.chromiumInstalled, chromiumInstalled: readiness.chromiumInstalled,
camoufoxInstalled: Boolean(getCamoufoxExecutablePath()),
noVncInstalled: fs.existsSync(path.join(NOVNC_ROOT, 'vnc.html')),
x11vncInstalled: fs.existsSync(X11VNC_BIN),
installInProgress: readiness.installInProgress, installInProgress: readiness.installInProgress,
sessionCount: sessions.size, sessionCount: sessions.size,
message: available message: available
@@ -505,7 +697,7 @@ export const browserUseService = {
} }
await expireStaleSessions(); await expireStaleSessions();
const profileName = normalizeProfileName(options?.profileName); const profileName = resolveSessionProfileName(settings, options?.profileName);
const now = new Date().toISOString(); const now = new Date().toISOString();
const session: BrowserUseSession = { const session: BrowserUseSession = {
@@ -521,6 +713,9 @@ export const browserUseService = {
updatedAt: now, updatedAt: now,
lastAction: 'create', lastAction: 'create',
message: null, message: null,
backend: useVisibleCamoufoxBackend(settings) ? 'camoufox-vnc' : 'playwright',
viewerUrl: null,
viewerEmbedUrl: null,
profileName, profileName,
viewport: { width: 1440, height: 900 }, viewport: { width: 1440, height: 900 },
cursor: null, cursor: null,
@@ -532,7 +727,13 @@ export const browserUseService = {
} }
const readiness = getRuntimeReadiness(); const readiness = getRuntimeReadiness();
if (!settings.enabled || !readiness.playwrightInstalled || !readiness.chromiumInstalled || !readiness.playwright) { const useVisibleBackend = useVisibleCamoufoxBackend(settings);
const visibleCamoufoxReady = useVisibleBackend
&& VISIBLE_BROWSER_ENABLED
&& Boolean(getCamoufoxExecutablePath())
&& fs.existsSync(X11VNC_BIN)
&& fs.existsSync(path.join(NOVNC_ROOT, 'vnc.html'));
if (!settings.enabled || !readiness.playwrightInstalled || !readiness.playwright || (useVisibleBackend ? !visibleCamoufoxReady : !readiness.chromiumInstalled)) {
session.message = getSetupMessage(settings, readiness); session.message = getSetupMessage(settings, readiness);
sessions.set(session.id, session); sessions.set(session.id, session);
return publicSession(session); return publicSession(session);
@@ -541,31 +742,73 @@ export const browserUseService = {
let browser: any | undefined; let browser: any | undefined;
let context: any | undefined; let context: any | undefined;
let page: any; let page: any;
const launchOptions = { let viewer: RuntimeHandle['viewer'];
headless: true, let processes: RuntimeHandle['processes'];
const launchOptions: Record<string, unknown> = {
headless: !useVisibleBackend,
args: ['--disable-dev-shm-usage'], args: ['--disable-dev-shm-usage'],
}; };
const contextOptions = { const contextOptions = useVisibleBackend
viewport: { width: 1440, height: 900 }, ? { viewport: null }
serviceWorkers: 'block', : {
}; viewport: { width: 1440, height: 900 },
serviceWorkers: 'block',
};
if (profileName) { try {
fs.mkdirSync(PROFILE_ROOT, { recursive: true }); if (useVisibleBackend) {
context = await readiness.playwright.chromium.launchPersistentContext(getProfilePath(profileName), { const camoufoxExecutable = getCamoufoxExecutablePath();
...launchOptions, if (!camoufoxExecutable) {
...contextOptions, throw new Error('Camoufox is not installed.');
}); }
page = context.pages()[0] || await context.newPage(); const runtime = await startVisibleRuntime();
} else { viewer = {
browser = await readiness.playwright.chromium.launch(launchOptions); display: runtime.display,
context = await browser.newContext(contextOptions); vncPort: runtime.vncPort,
page = await context.newPage(); websockifyPort: runtime.websockifyPort,
noVncRoot: runtime.noVncRoot,
};
processes = runtime.processes;
launchOptions.executablePath = camoufoxExecutable;
launchOptions.env = {
...process.env,
DISPLAY: runtime.display,
LD_LIBRARY_PATH: `${X11VNC_LIB_DIR}:${X11VNC_EXTRA_LIB_DIR}:${process.env.LD_LIBRARY_PATH || ''}`,
};
launchOptions.args = [];
session.backend = 'camoufox-vnc';
const viewerToken = createViewerToken(session.id);
session.viewerUrl = getViewerUrl(session.id, viewerToken);
session.viewerEmbedUrl = session.viewerUrl;
}
if (profileName) {
fs.mkdirSync(PROFILE_ROOT, { recursive: true });
const browserType = useVisibleBackend ? readiness.playwright.firefox : readiness.playwright.chromium;
context = await browserType.launchPersistentContext(getProfilePath(profileName), {
...launchOptions,
...contextOptions,
});
page = context.pages()[0] || await context.newPage();
} else {
const browserType = useVisibleBackend ? readiness.playwright.firefox : readiness.playwright.chromium;
browser = await browserType.launch(launchOptions);
context = await browser.newContext(contextOptions);
page = await context.newPage();
}
} catch (error) {
await context?.close?.().catch(() => undefined);
await browser?.close?.().catch(() => undefined);
killRuntimeProcesses(processes);
if (viewer?.display) {
reservedDisplays.delete(viewer.display);
}
throw error;
} }
session.status = 'ready'; session.status = 'ready';
session.message = 'Browser session is ready.'; session.message = 'Browser session is ready.';
sessions.set(session.id, session); sessions.set(session.id, session);
handles.set(session.id, { browser, context, page }); handles.set(session.id, { browser, context, page, processes, viewer });
await captureSession(session, page); await captureSession(session, page);
return publicSession(session); return publicSession(session);
}, },
@@ -812,6 +1055,25 @@ export const browserUseService = {
return { deleted: true, sessionId }; return { deleted: true, sessionId };
}, },
getViewerProxyTarget(sessionId: string) {
const viewer = getSessionViewer(sessionId);
if (!viewer) {
throw new Error('Browser viewer is not available for this session.');
}
return {
websockifyPort: viewer.websockifyPort,
noVncRoot: viewer.noVncRoot,
};
},
validateViewerToken(sessionId: string, token: string | null | undefined) {
return validateViewerTokenForSession(sessionId, token);
},
handleViewerWebSocket(clientWs: WebSocket, pathname: string) {
handleViewerWebSocket(clientWs, pathname, getSessionViewer);
},
async agentStopSession(sessionId: string) { async agentStopSession(sessionId: string) {
await this.getAgentSession(sessionId); await this.getAgentSession(sessionId);
return this.stopSession(sessionId); return this.stopSession(sessionId);

View File

@@ -0,0 +1,147 @@
import { randomBytes } from 'node:crypto';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { appConfigDb } from '@/modules/database/index.js';
import type { BrowserUseBackend, BrowserUseSettings } from './browser-use.types.js';
const IS_PLATFORM = process.env.VITE_IS_PLATFORM === 'true';
const BROWSER_USE_SETTINGS_KEY = 'browser_use_settings';
const BROWSER_USE_MCP_TOKEN_KEY = 'browser_use_mcp_token';
const MAX_PROFILE_NAME_LENGTH = 80;
export const DEFAULT_BROWSER_USE_SETTINGS: BrowserUseSettings = {
enabled: false,
persistSessions: false,
defaultProfileName: 'default',
browserBackend: IS_PLATFORM ? 'camoufox-vnc' : 'playwright',
};
export const PROFILE_ROOT = process.env.CLOUDCLI_BROWSER_USE_PROFILE_ROOT
|| path.join(os.homedir(), '.cloudcli', 'browser-use', 'profiles');
export function normalizeBrowserBackend(value: unknown): BrowserUseBackend {
return value === 'playwright' || value === 'camoufox-vnc'
? value
: DEFAULT_BROWSER_USE_SETTINGS.browserBackend;
}
function trimEdgeDashes(value: string): string {
let start = 0;
let end = value.length;
while (start < end && value[start] === '-') {
start += 1;
}
while (end > start && value[end - 1] === '-') {
end -= 1;
}
return value.slice(start, end);
}
export function normalizeProfileName(profileName?: string | null): string | null {
const sanitized = trimEdgeDashes(String(profileName || '')
.trim()
.toLowerCase()
.replace(/[^a-z0-9._-]+/g, '-'));
const normalized = sanitized
.slice(0, MAX_PROFILE_NAME_LENGTH)
.replace(/^-+|-+$/g, '');
if (!normalized) {
return null;
}
return /[a-z0-9]/.test(normalized) ? normalized : null;
}
export function normalizeDefaultProfileName(profileName?: string | null): string {
return normalizeProfileName(profileName) || DEFAULT_BROWSER_USE_SETTINGS.defaultProfileName;
}
export function resolveSessionProfileName(settings: BrowserUseSettings, profileName?: string | null): string | null {
const requestedProfileName = normalizeProfileName(profileName);
if (String(profileName || '').trim() && !requestedProfileName) {
throw new Error('Browser profile name must include at least one letter or number.');
}
if (requestedProfileName) {
validateRequestedProfileName(profileName, requestedProfileName);
return requestedProfileName;
}
return settings.persistSessions ? normalizeDefaultProfileName(settings.defaultProfileName) : null;
}
export function getProfilePath(profileName: string): string {
return path.join(PROFILE_ROOT, normalizeDefaultProfileName(profileName));
}
function validateRequestedProfileName(profileName: string | null | undefined, normalizedProfileName: string): void {
const requestedProfileName = String(profileName || '').trim();
const existingProfileName = findExistingProfileName(normalizedProfileName);
if (
existingProfileName
&& (requestedProfileName !== normalizedProfileName || existingProfileName !== normalizedProfileName)
) {
throw new Error(`Browser profile "${requestedProfileName}" resolves to existing profile "${existingProfileName}". Use "${normalizedProfileName}" instead.`);
}
}
function findExistingProfileName(normalizedProfileName: string): string | null {
try {
if (!fs.existsSync(PROFILE_ROOT)) {
return null;
}
const entries = fs.readdirSync(PROFILE_ROOT, { withFileTypes: true });
const match = entries.find((entry) => entry.isDirectory() && normalizeProfileName(entry.name) === normalizedProfileName);
return match?.name || null;
} catch {
return null;
}
}
export function useVisibleCamoufoxBackend(settings: BrowserUseSettings): boolean {
return settings.browserBackend === 'camoufox-vnc';
}
export function readSettings(): BrowserUseSettings {
try {
const raw = appConfigDb.get(BROWSER_USE_SETTINGS_KEY);
if (!raw) {
return DEFAULT_BROWSER_USE_SETTINGS;
}
const parsed = JSON.parse(raw) as Partial<BrowserUseSettings>;
return {
enabled: parsed.enabled === true,
persistSessions: parsed.persistSessions === true,
defaultProfileName: normalizeDefaultProfileName(parsed.defaultProfileName),
browserBackend: normalizeBrowserBackend(parsed.browserBackend),
};
} catch (error: any) {
console.warn('[Browser] Failed to read settings:', error?.message || error);
return DEFAULT_BROWSER_USE_SETTINGS;
}
}
export function writeSettings(settings: BrowserUseSettings): BrowserUseSettings {
const normalized = {
enabled: settings.enabled === true,
persistSessions: settings.persistSessions === true,
defaultProfileName: normalizeDefaultProfileName(settings.defaultProfileName),
browserBackend: normalizeBrowserBackend(settings.browserBackend),
};
appConfigDb.set(BROWSER_USE_SETTINGS_KEY, JSON.stringify(normalized));
return normalized;
}
export function getOrCreateMcpToken(): string {
const existing = appConfigDb.get(BROWSER_USE_MCP_TOKEN_KEY);
if (existing) {
return existing;
}
const token = randomBytes(32).toString('hex');
appConfigDb.set(BROWSER_USE_MCP_TOKEN_KEY, token);
return token;
}

View File

@@ -0,0 +1,66 @@
import type { spawn } from 'node:child_process';
export type BrowserUseRuntime = 'cloud' | 'local';
export type BrowserUseBackend = 'playwright' | 'camoufox-vnc';
export type BrowserUseSessionStatus = 'ready' | 'stopped' | 'unavailable';
export type BrowserUseSession = {
id: string;
ownerId: string;
createdBy: 'agent';
runtime: BrowserUseRuntime;
status: BrowserUseSessionStatus;
url: string | null;
title: string | null;
screenshotDataUrl: string | null;
createdAt: string;
updatedAt: string;
lastAction: string | null;
message: string | null;
backend: BrowserUseBackend;
viewerUrl: string | null;
viewerEmbedUrl: string | null;
profileName: string | null;
viewport: {
width: number;
height: number;
} | null;
cursor: {
x: number;
y: number;
actor: 'agent';
} | null;
};
export type PublicBrowserUseSession = Omit<BrowserUseSession, 'ownerId'>;
export type RuntimeHandle = {
browser?: any;
context?: any;
page?: any;
processes?: Array<ReturnType<typeof spawn>>;
viewer?: {
display: string;
vncPort: number;
websockifyPort: number;
noVncRoot: string;
};
};
export type BrowserUseSettings = {
enabled: boolean;
persistSessions: boolean;
defaultProfileName: string;
browserBackend: BrowserUseBackend;
};
export type RuntimeReadiness = {
playwright: any | null;
playwrightInstalled: boolean;
chromiumInstalled: boolean;
chromiumExecutablePath: string | null;
installInProgress: boolean;
installMessage: string | null;
};
export type RuntimeProbe = Omit<RuntimeReadiness, 'installInProgress' | 'installMessage'>;

View File

@@ -0,0 +1,76 @@
import { WebSocket } from 'ws';
import type { RuntimeHandle } from './browser-use.types.js';
type BrowserUseViewer = NonNullable<RuntimeHandle['viewer']>;
export const VIEWER_COOKIE_NAME = 'browser_use_viewer_token';
const DEFAULT_VIEWER_TOKEN_TTL_MS = 30 * 60 * 1000;
const parsedViewerTokenTtlMs = Number.parseInt(
process.env.CLOUDCLI_BROWSER_USE_VIEWER_TOKEN_TTL_MS || String(DEFAULT_VIEWER_TOKEN_TTL_MS),
10,
);
export const VIEWER_TOKEN_TTL_MS =
Number.isFinite(parsedViewerTokenTtlMs) && parsedViewerTokenTtlMs > 0
? parsedViewerTokenTtlMs
: DEFAULT_VIEWER_TOKEN_TTL_MS;
export function getViewerUrl(sessionId: string, viewerToken?: string): string {
const basePath = `/api/browser-use/sessions/${encodeURIComponent(sessionId)}/viewer`;
const websockifyPath = viewerToken
? `${basePath}/websockify?viewerToken=${encodeURIComponent(viewerToken)}`
: `${basePath}/websockify`;
const params = new URLSearchParams({
autoconnect: '1',
resize: 'scale',
reconnect: '1',
path: websockifyPath,
});
if (viewerToken) {
params.set('viewerToken', viewerToken);
}
return `${basePath}/vnc.html?${params.toString()}`;
}
export function handleViewerWebSocket(
clientWs: WebSocket,
pathname: string,
getSessionViewer: (sessionId: string) => BrowserUseViewer | null | undefined,
) {
const match = /^\/api\/browser-use\/sessions\/([^/]+)\/viewer\/websockify\/?$/.exec(pathname);
const sessionId = match ? decodeURIComponent(match[1]) : '';
const viewer = sessionId ? getSessionViewer(sessionId) : null;
if (!viewer) {
clientWs.close(4404, 'Browser viewer not found');
return;
}
const upstream = new WebSocket(`ws://127.0.0.1:${viewer.websockifyPort}`);
upstream.on('open', () => {
clientWs.on('message', (data) => {
if (upstream.readyState === WebSocket.OPEN) {
upstream.send(data);
}
});
upstream.on('message', (data) => {
if (clientWs.readyState === WebSocket.OPEN) {
clientWs.send(data);
}
});
});
upstream.on('close', (code, reason) => {
if (clientWs.readyState === WebSocket.OPEN) {
clientWs.close(code, reason);
}
});
upstream.on('error', () => {
if (clientWs.readyState === WebSocket.OPEN) {
clientWs.close(4502, 'Browser viewer upstream error');
}
});
clientWs.on('close', () => {
if (upstream.readyState === WebSocket.OPEN) {
upstream.close();
}
});
}

View File

@@ -0,0 +1,2 @@
export { browserUseService } from './browser-use.service.js';
export { VIEWER_COOKIE_NAME } from './browser-use.viewer.js';

View File

@@ -0,0 +1,73 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import test from 'node:test';
const originalProfileRoot = process.env.CLOUDCLI_BROWSER_USE_PROFILE_ROOT;
const testProfileRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'browser-use-profiles-'));
process.env.CLOUDCLI_BROWSER_USE_PROFILE_ROOT = testProfileRoot;
const {
getProfilePath,
normalizeDefaultProfileName,
normalizeProfileName,
PROFILE_ROOT,
resolveSessionProfileName,
} = await import('@/modules/browser-use/browser-use.settings.js');
test.after(() => {
if (originalProfileRoot === undefined) {
delete process.env.CLOUDCLI_BROWSER_USE_PROFILE_ROOT;
} else {
process.env.CLOUDCLI_BROWSER_USE_PROFILE_ROOT = originalProfileRoot;
}
fs.rmSync(testProfileRoot, { recursive: true, force: true });
});
test('browser profile names are canonicalized before storage and path resolution', () => {
assert.equal(normalizeProfileName(' Work Profile!! '), 'work-profile');
assert.equal(normalizeProfileName(`${'-'.repeat(100)}Work Profile`), 'work-profile');
assert.equal(normalizeDefaultProfileName(' Work Profile!! '), 'work-profile');
assert.equal(
getProfilePath(' Work Profile!! '),
`${PROFILE_ROOT}/work-profile`,
);
assert.equal(
resolveSessionProfileName({
enabled: true,
persistSessions: true,
defaultProfileName: ' Work Profile!! ',
browserBackend: 'playwright',
}),
'work-profile',
);
});
test('browser profile aliases are rejected when the normalized profile already exists', () => {
const profileName = `alias-test-${Date.now()}`;
fs.mkdirSync(getProfilePath(profileName), { recursive: true });
try {
assert.throws(
() => resolveSessionProfileName({
enabled: true,
persistSessions: false,
defaultProfileName: 'default',
browserBackend: 'playwright',
}, profileName.toUpperCase()),
/resolves to existing profile/,
);
assert.equal(
resolveSessionProfileName({
enabled: true,
persistSessions: false,
defaultProfileName: 'default',
browserBackend: 'playwright',
}, profileName),
profileName,
);
} finally {
fs.rmSync(getProfilePath(profileName), { recursive: true, force: true });
}
});

View File

@@ -1,8 +1,9 @@
import type { Server as HttpServer } from 'node:http'; import type { Server as HttpServer } from 'node:http';
import { WebSocketServer, type VerifyClientCallbackSync } from 'ws'; import { WebSocket, WebSocketServer, type VerifyClientCallbackSync } from 'ws';
import { handleChatConnection } from '@/modules/websocket/services/chat-websocket.service.js'; import { handleChatConnection } from '@/modules/websocket/services/chat-websocket.service.js';
import { VIEWER_COOKIE_NAME } from '@/modules/browser-use/index.js';
import { verifyWebSocketClient } from '@/modules/websocket/services/websocket-auth.service.js'; import { verifyWebSocketClient } from '@/modules/websocket/services/websocket-auth.service.js';
import { handlePluginWsProxy } from '@/modules/websocket/services/plugin-websocket-proxy.service.js'; import { handlePluginWsProxy } from '@/modules/websocket/services/plugin-websocket-proxy.service.js';
import { handleShellConnection } from '@/modules/websocket/services/shell-websocket.service.js'; import { handleShellConnection } from '@/modules/websocket/services/shell-websocket.service.js';
@@ -13,8 +14,21 @@ type WebSocketServerDependencies = {
chat: Parameters<typeof handleChatConnection>[2]; chat: Parameters<typeof handleChatConnection>[2];
shell: Parameters<typeof handleShellConnection>[1]; shell: Parameters<typeof handleShellConnection>[1];
getPluginPort: Parameters<typeof handlePluginWsProxy>[2]; getPluginPort: Parameters<typeof handlePluginWsProxy>[2];
browserUseViewer?: (ws: WebSocket, pathname: string) => void;
authenticateBrowserUseViewer?: (pathname: string, token: string | null) => boolean;
}; };
function readCookieValue(header: unknown, name: string): string | null {
if (!header) return null;
const prefix = `${name}=`;
const cookie = String(header).split(';').map((part) => part.trim()).find((part) => part.startsWith(prefix));
return cookie ? decodeURIComponent(cookie.slice(prefix.length)) : null;
}
function getBrowserUseViewerToken(url: URL, headers: Record<string, unknown>): string | null {
return url.searchParams.get('viewerToken') || readCookieValue(headers.cookie, VIEWER_COOKIE_NAME);
}
/** /**
* Creates and wires the server-wide websocket gateway used for chat, shell, and * Creates and wires the server-wide websocket gateway used for chat, shell, and
* plugin proxy routes. * plugin proxy routes.
@@ -27,7 +41,17 @@ export function createWebSocketServer(
server, server,
verifyClient: (( verifyClient: ((
info: Parameters<VerifyClientCallbackSync<AuthenticatedWebSocketRequest>>[0] info: Parameters<VerifyClientCallbackSync<AuthenticatedWebSocketRequest>>[0]
) => verifyWebSocketClient(info, dependencies.verifyClient)), ) => {
const requestUrl = new URL(info.req.url ?? '/', 'http://localhost');
if (
requestUrl.pathname.startsWith('/api/browser-use/sessions/')
&& requestUrl.pathname.endsWith('/viewer/websockify')
) {
const token = getBrowserUseViewerToken(requestUrl, info.req.headers as Record<string, unknown>);
return Boolean(dependencies.authenticateBrowserUseViewer?.(requestUrl.pathname, token));
}
return verifyWebSocketClient(info, dependencies.verifyClient);
}),
}); });
wss.on('connection', (ws, request) => { wss.on('connection', (ws, request) => {
@@ -68,6 +92,11 @@ export function createWebSocketServer(
return; return;
} }
if (pathname.startsWith('/api/browser-use/sessions/') && pathname.endsWith('/viewer/websockify')) {
dependencies.browserUseViewer?.(ws, pathname);
return;
}
console.log('[WARN] Unknown WebSocket path:', pathname); console.log('[WARN] Unknown WebSocket path:', pathname);
ws.close(); ws.close();
}); });

View File

@@ -1,4 +1,4 @@
import { useCallback, useEffect, useMemo, useState } from 'react'; import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
import { import {
Bot, Bot,
Clock3, Clock3,
@@ -7,6 +7,7 @@ import {
ExternalLink, ExternalLink,
Loader2, Loader2,
MonitorPlay, MonitorPlay,
MousePointer2,
RefreshCw, RefreshCw,
Settings, Settings,
Square, Square,
@@ -19,9 +20,14 @@ import { Badge, Button } from '../../../shared/view/ui';
import { authenticatedFetch } from '../../../utils/api'; import { authenticatedFetch } from '../../../utils/api';
import type { SettingsMainTab } from '../../settings/types/types'; import type { SettingsMainTab } from '../../settings/types/types';
const BROWSER_USE_GUIDE_URL = 'https://cloudcli.ai/docs/browser-use';
const BROWSER_USE_CACHE_TTL_MS = 30_000;
type BrowserUseStatus = { type BrowserUseStatus = {
enabled: boolean; enabled: boolean;
available: boolean; available: boolean;
backend: 'playwright' | 'camoufox-vnc';
browserBackend: 'playwright' | 'camoufox-vnc';
playwrightInstalled: boolean; playwrightInstalled: boolean;
chromiumInstalled: boolean; chromiumInstalled: boolean;
installInProgress: boolean; installInProgress: boolean;
@@ -39,6 +45,9 @@ type BrowserUseSession = {
updatedAt: string; updatedAt: string;
lastAction: string | null; lastAction: string | null;
message: string | null; message: string | null;
backend?: 'playwright' | 'camoufox-vnc';
viewerUrl?: string | null;
viewerEmbedUrl?: string | null;
createdBy: 'agent'; createdBy: 'agent';
profileName: string | null; profileName: string | null;
viewport: { viewport: {
@@ -54,17 +63,48 @@ type BrowserUseSession = {
type BrowserUsePanelProps = { type BrowserUsePanelProps = {
isVisible: boolean; isVisible: boolean;
projectId?: string | null;
onShowSettings?: (tab?: SettingsMainTab) => void; onShowSettings?: (tab?: SettingsMainTab) => void;
}; };
type BrowserUsePanelCacheEntry = {
status: BrowserUseStatus | null;
sessions: BrowserUseSession[];
selectedSessionId: string | null;
updatedAt: number;
};
const browserUsePanelCache = new Map<string, BrowserUsePanelCacheEntry>();
async function readJson<T>(response: Response): Promise<T> { async function readJson<T>(response: Response): Promise<T> {
const data = await response.json(); const text = await response.text();
let data: any = {};
if (text) {
try {
data = JSON.parse(text);
} catch {
throw new Error(response.ok ? 'Received an invalid Browser response.' : `Browser request failed (${response.status}).`);
}
}
if (!response.ok || data.success === false) { if (!response.ok || data.success === false) {
throw new Error(data.error || data.details || `Request failed (${response.status})`); throw new Error(data.error || data.details || `Request failed (${response.status})`);
} }
return data as T; return data as T;
} }
async function fetchBrowserPanelData() {
const [statusResponse, sessionsResponse] = await Promise.all([
authenticatedFetch('/api/browser-use/status'),
authenticatedFetch('/api/browser-use/sessions'),
]);
const statusData = await readJson<{ data: BrowserUseStatus }>(statusResponse);
const sessionsData = await readJson<{ data: { sessions: BrowserUseSession[] } }>(sessionsResponse);
return {
status: statusData.data,
sessions: [...sessionsData.data.sessions].sort((a, b) => Date.parse(b.createdAt) - Date.parse(a.createdAt)),
};
}
function formatRelativeTime(value: string | null): string { function formatRelativeTime(value: string | null): string {
if (!value) return 'Never'; if (!value) return 'Never';
@@ -119,20 +159,42 @@ function getStatusDot(status: BrowserUseSession['status']): string {
return 'bg-border'; return 'bg-border';
} }
function getEngineLabel(backend?: BrowserUseStatus['backend'] | BrowserUseSession['backend']): string {
return backend === 'camoufox-vnc' ? 'Visible browser' : 'Playwright';
}
const PROMPTS = [ const PROMPTS = [
'Use Browser to inspect the checkout flow and report any broken UI states.', 'Use Browser to inspect the checkout flow and report any broken UI states.',
'Open <url> with Browser, interact with the page, and summarize what changed after each step.', 'Open <url> with Browser, interact with the page, and summarize what changed after each step.',
]; ];
export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUsePanelProps) { function getBrowserUseCacheKey(projectId?: string | null): string {
const [status, setStatus] = useState<BrowserUseStatus | null>(null); return projectId ? `browser-use:project:${projectId}` : 'browser-use:global';
const [sessions, setSessions] = useState<BrowserUseSession[]>([]); }
const [selectedSessionId, setSelectedSessionId] = useState<string | null>(null);
function getFreshCacheEntry(cacheKey: string): BrowserUsePanelCacheEntry | null {
const entry = browserUsePanelCache.get(cacheKey);
if (!entry || Date.now() - entry.updatedAt > BROWSER_USE_CACHE_TTL_MS) {
return null;
}
return entry;
}
export default function BrowserUsePanel({ isVisible, projectId, onShowSettings }: BrowserUsePanelProps) {
const cacheKey = getBrowserUseCacheKey(projectId);
const initialCacheEntry = getFreshCacheEntry(cacheKey);
const [status, setStatus] = useState<BrowserUseStatus | null>(() => initialCacheEntry?.status ?? null);
const [sessions, setSessions] = useState<BrowserUseSession[]>(() => initialCacheEntry?.sessions ?? []);
const [selectedSessionId, setSelectedSessionId] = useState<string | null>(() => (
initialCacheEntry?.selectedSessionId || initialCacheEntry?.sessions[0]?.id || null
));
const [hasLoadedOnce, setHasLoadedOnce] = useState(Boolean(initialCacheEntry));
const [isRefreshing, setIsRefreshing] = useState(false); const [isRefreshing, setIsRefreshing] = useState(false);
const [isBusy, setIsBusy] = useState(false); const [isBusy, setIsBusy] = useState(false);
const [isInstalling, setIsInstalling] = useState(false); const [isInstalling, setIsInstalling] = useState(false);
const [isFullscreen, setIsFullscreen] = useState(false); const [isFullscreen, setIsFullscreen] = useState(false);
const [error, setError] = useState<string | null>(null); const [error, setError] = useState<string | null>(null);
const activeLoadIdRef = useRef(0);
const selectedSession = useMemo( const selectedSession = useMemo(
() => sessions.find((session) => session.id === selectedSessionId) || sessions[0] || null, () => sessions.find((session) => session.id === selectedSessionId) || sessions[0] || null,
@@ -140,8 +202,12 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
); );
const activeSessions = sessions.filter((session) => session.status === 'ready'); const activeSessions = sessions.filter((session) => session.status === 'ready');
const needsBrowserBinaries = Boolean(status?.enabled && (!status.playwrightInstalled || !status.chromiumInstalled)); const isInitialLoading = isRefreshing && !hasLoadedOnce && sessions.length === 0;
const runtimeLabel = !status?.enabled const isBackgroundRefreshing = isRefreshing && !isInitialLoading;
const needsBrowserBinaries = Boolean(status?.enabled && !status.available);
const runtimeLabel = isInitialLoading
? 'Loading'
: !status?.enabled
? 'Disabled' ? 'Disabled'
: status.available : status.available
? 'Ready' ? 'Ready'
@@ -157,29 +223,72 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
: null; : null;
const refresh = useCallback(async () => { const refresh = useCallback(async () => {
const loadId = activeLoadIdRef.current + 1;
activeLoadIdRef.current = loadId;
setIsRefreshing(true); setIsRefreshing(true);
try { try {
const [statusResponse, sessionsResponse] = await Promise.all([ let nextData: Awaited<ReturnType<typeof fetchBrowserPanelData>>;
authenticatedFetch('/api/browser-use/status'), try {
authenticatedFetch('/api/browser-use/sessions'), nextData = await fetchBrowserPanelData();
]); } catch (error) {
const statusData = await readJson<{ data: BrowserUseStatus }>(statusResponse); if (loadId !== activeLoadIdRef.current) {
const sessionsData = await readJson<{ data: { sessions: BrowserUseSession[] } }>(sessionsResponse); return;
const nextSessions = sessionsData.data.sessions; }
setStatus(statusData.data); await new Promise((resolve) => setTimeout(resolve, 350));
nextData = await fetchBrowserPanelData();
}
if (activeLoadIdRef.current !== loadId) {
return;
}
const nextSessions = nextData.sessions;
setStatus(nextData.status);
setSessions(nextSessions); setSessions(nextSessions);
setSelectedSessionId((current) => ( setHasLoadedOnce(true);
current && nextSessions.some((session) => session.id === current) let nextSelectedSessionId: string | null = null;
setSelectedSessionId((current) => {
nextSelectedSessionId = current && nextSessions.some((session) => session.id === current)
? current ? current
: nextSessions[0]?.id || null : nextSessions[0]?.id || null;
)); return nextSelectedSessionId;
});
browserUsePanelCache.set(cacheKey, {
status: nextData.status,
sessions: nextSessions,
selectedSessionId: nextSelectedSessionId,
updatedAt: Date.now(),
});
setError(null); setError(null);
} catch (err) { } catch (err) {
if (activeLoadIdRef.current !== loadId) {
return;
}
setHasLoadedOnce(true);
setError(err instanceof Error ? err.message : 'Failed to load Browser'); setError(err instanceof Error ? err.message : 'Failed to load Browser');
} finally { } finally {
setIsRefreshing(false); if (activeLoadIdRef.current === loadId) {
setIsRefreshing(false);
}
} }
}, []); }, [cacheKey]);
useEffect(() => {
const cachedEntry = browserUsePanelCache.get(cacheKey);
if (!cachedEntry) return;
browserUsePanelCache.set(cacheKey, {
...cachedEntry,
selectedSessionId,
});
}, [cacheKey, selectedSessionId]);
useEffect(() => {
const cachedEntry = getFreshCacheEntry(cacheKey);
setStatus(cachedEntry?.status ?? null);
setSessions(cachedEntry?.sessions ?? []);
setSelectedSessionId(cachedEntry?.selectedSessionId || cachedEntry?.sessions[0]?.id || null);
setHasLoadedOnce(Boolean(cachedEntry));
setError(null);
activeLoadIdRef.current += 1;
}, [cacheKey]);
useEffect(() => { useEffect(() => {
if (!isVisible) return; if (!isVisible) return;
@@ -253,6 +362,10 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
<span>{formatRelativeTime(session.updatedAt)}</span> <span>{formatRelativeTime(session.updatedAt)}</span>
<span className="truncate">- {formatAction(session.lastAction)}</span> <span className="truncate">- {formatAction(session.lastAction)}</span>
</div> </div>
<div className="mt-2 flex flex-wrap gap-1.5 pl-3.5 text-[10px] text-muted-foreground">
<span className="rounded border border-border/70 bg-background/70 px-1.5 py-0.5">{getEngineLabel(session.backend)}</span>
<span className="rounded border border-border/70 bg-background/70 px-1.5 py-0.5">{session.profileName || 'Temporary'}</span>
</div>
</button> </button>
); );
}; };
@@ -270,9 +383,18 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
</div> </div>
<p className="mt-1 max-w-xl text-sm leading-6 text-muted-foreground"> <p className="mt-1 max-w-xl text-sm leading-6 text-muted-foreground">
{status?.enabled {status?.enabled
? 'Agent browser sessions appear here while an AI task is using Browser.' ? 'When an agent opens a browser, you can watch the latest screenshot, take control in a new tab, or end the running session.'
: 'Enable Browser in settings to let agents open monitored browser sessions.'} : 'Enable Browser to let agents open websites, test flows, capture screenshots, and debug UI from a real page.'}
</p> </p>
<a
href={BROWSER_USE_GUIDE_URL}
target="_blank"
rel="noopener noreferrer"
className="mt-2 inline-flex items-center gap-1.5 text-sm font-medium text-primary hover:underline"
>
Read the Browser guide
<ExternalLink className="h-3.5 w-3.5" />
</a>
</div> </div>
</div> </div>
@@ -312,10 +434,19 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
</div> </div>
); );
const renderLoadingState = () => (
<div className="flex min-h-0 flex-1 items-center justify-center p-6">
<div className="flex items-center gap-3 rounded-md border border-border bg-card/40 px-4 py-3 text-sm text-muted-foreground shadow-sm">
<Loader2 className="h-4 w-4 animate-spin text-primary" />
Loading browser sessions...
</div>
</div>
);
const renderBrowserSurface = (fullscreen = false) => ( const renderBrowserSurface = (fullscreen = false) => (
<div className={cn('flex flex-1 items-center justify-center bg-neutral-950', fullscreen ? 'min-h-[80vh]' : 'min-h-[420px]')}> <div className={cn('flex flex-1 items-center justify-center bg-neutral-950', fullscreen ? 'min-h-[80vh]' : 'min-h-[420px]')}>
{selectedSession?.screenshotDataUrl ? ( {selectedSession?.screenshotDataUrl ? (
<div className="relative inline-block max-h-full"> <div className="group relative inline-block max-h-full">
<img <img
src={selectedSession.screenshotDataUrl} src={selectedSession.screenshotDataUrl}
alt="Browser session screenshot" alt="Browser session screenshot"
@@ -329,6 +460,18 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
<div className="absolute left-1/2 top-1/2 h-2 w-2 -translate-x-1/2 -translate-y-1/2 rounded-full bg-white" /> <div className="absolute left-1/2 top-1/2 h-2 w-2 -translate-x-1/2 -translate-y-1/2 rounded-full bg-white" />
</div> </div>
)} )}
{selectedSession?.viewerEmbedUrl && selectedSession.status === 'ready' && (
<button
type="button"
onClick={() => window.open(selectedSession.viewerUrl || selectedSession.viewerEmbedUrl || '', '_blank', 'noopener,noreferrer')}
className="absolute inset-0 flex items-center justify-center bg-black/0 opacity-0 transition focus-visible:bg-black/30 focus-visible:opacity-100 focus-visible:outline-none group-hover:bg-black/30 group-hover:opacity-100"
>
<span className="inline-flex items-center gap-2 rounded-md border border-white/20 bg-black/80 px-3 py-2 text-sm font-medium text-white shadow-lg">
<MousePointer2 className="h-4 w-4" />
Take control
</span>
</button>
)}
</div> </div>
) : ( ) : (
<div className="px-6 text-center"> <div className="px-6 text-center">
@@ -350,10 +493,29 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
<Badge variant="outline" className={cn('text-[10px]', getRuntimeTone(status, isInstalling))}> <Badge variant="outline" className={cn('text-[10px]', getRuntimeTone(status, isInstalling))}>
{runtimeLabel} {runtimeLabel}
</Badge> </Badge>
<Badge variant="outline" className="border-border bg-background text-[10px] text-muted-foreground">
{getEngineLabel(status?.backend)}
</Badge>
</div> </div>
<p className="mt-0.5 text-xs text-muted-foreground">Monitor browser sessions opened by AI agents.</p> <p className="mt-0.5 text-xs text-muted-foreground">Watch and manage browser sessions agents use to test real websites.</p>
{isBackgroundRefreshing && (
<div className="mt-1 flex items-center gap-1.5 text-xs text-muted-foreground">
<RefreshCw className="h-3 w-3 animate-spin" />
Refreshing sessions...
</div>
)}
</div> </div>
<div className="flex items-center gap-1.5"> <div className="flex items-center gap-1.5">
<Button
variant="ghost"
size="sm"
className="h-7 w-7 p-0"
onClick={() => window.open(BROWSER_USE_GUIDE_URL, '_blank', 'noopener,noreferrer')}
title="Open Browser guide"
aria-label="Open Browser guide"
>
<ExternalLink className="h-3.5 w-3.5" />
</Button>
{onShowSettings && ( {onShowSettings && (
<Button <Button
variant="ghost" variant="ghost"
@@ -425,7 +587,7 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
</div> </div>
{sessions.length === 0 ? ( {sessions.length === 0 ? (
renderEmptyState() isInitialLoading ? renderLoadingState() : renderEmptyState()
) : ( ) : (
<div className="min-h-0 flex-1 overflow-auto bg-muted/20 p-4"> <div className="min-h-0 flex-1 overflow-auto bg-muted/20 p-4">
<div className="mx-auto flex min-h-[500px] max-w-7xl flex-col overflow-hidden rounded-md border border-border bg-background shadow-sm"> <div className="mx-auto flex min-h-[500px] max-w-7xl flex-col overflow-hidden rounded-md border border-border bg-background shadow-sm">
@@ -441,14 +603,32 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
<ExternalLink className="h-3.5 w-3.5 shrink-0" /> <ExternalLink className="h-3.5 w-3.5 shrink-0" />
<span className="truncate">{selectedSession?.url || 'No page loaded'}</span> <span className="truncate">{selectedSession?.url || 'No page loaded'}</span>
</div> </div>
<div className="mt-1 flex flex-wrap gap-1.5 text-[10px] text-muted-foreground">
<span className="rounded border border-border/70 bg-muted/30 px-1.5 py-0.5">{getEngineLabel(selectedSession?.backend || status?.backend)}</span>
<span className="rounded border border-border/70 bg-muted/30 px-1.5 py-0.5">Profile: {selectedSession?.profileName || 'Temporary'}</span>
<span className="rounded border border-border/70 bg-muted/30 px-1.5 py-0.5">Updated {formatRelativeTime(selectedSession?.updatedAt || null)}</span>
</div>
</div> </div>
<div className="hidden text-xs text-muted-foreground md:block"> <div className="hidden text-xs text-muted-foreground md:block">
{formatAction(selectedSession?.lastAction || null)} {formatAction(selectedSession?.lastAction || null)}
</div> </div>
{selectedSession?.viewerUrl && selectedSession.status === 'ready' && (
<Button
variant="outline"
size="sm"
className="h-8"
onClick={() => window.open(selectedSession.viewerUrl || '', '_blank', 'noopener,noreferrer')}
title="Open live browser control in a new tab"
aria-label="Open live browser control in a new tab"
>
<MousePointer2 className="h-4 w-4" />
Take control
</Button>
)}
<Button variant="ghost" size="sm" className="h-8 w-8 p-0" onClick={() => setIsFullscreen(true)} disabled={!selectedSession?.screenshotDataUrl} title="Full screen" aria-label="Full screen"> <Button variant="ghost" size="sm" className="h-8 w-8 p-0" onClick={() => setIsFullscreen(true)} disabled={!selectedSession?.screenshotDataUrl} title="Full screen" aria-label="Full screen">
<Expand className="h-4 w-4" /> <Expand className="h-4 w-4" />
</Button> </Button>
<Button variant="ghost" size="sm" className="h-8 w-8 p-0 lg:hidden" onClick={stopSession} disabled={isBusy || !selectedSession || selectedSession.status !== 'ready'} title="Stop session" aria-label="Stop session"> <Button variant="ghost" size="sm" className="h-8 w-8 p-0 lg:hidden" onClick={stopSession} disabled={isBusy || !selectedSession || selectedSession.status !== 'ready'} title="End session" aria-label="End session">
<Square className="h-4 w-4" /> <Square className="h-4 w-4" />
</Button> </Button>
<Button variant="ghost" size="sm" className="h-8 w-8 p-0 lg:hidden" onClick={deleteSession} disabled={isBusy || !selectedSession} title="Delete session" aria-label="Delete session"> <Button variant="ghost" size="sm" className="h-8 w-8 p-0 lg:hidden" onClick={deleteSession} disabled={isBusy || !selectedSession} title="Delete session" aria-label="Delete session">
@@ -475,6 +655,11 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
<div className="min-h-0 flex-1 overflow-y-auto p-3"> <div className="min-h-0 flex-1 overflow-y-auto p-3">
{sessions.length > 0 ? ( {sessions.length > 0 ? (
<div className="space-y-2">{sessions.map(renderSessionItem)}</div> <div className="space-y-2">{sessions.map(renderSessionItem)}</div>
) : isInitialLoading ? (
<div className="flex items-center justify-center gap-2 rounded-md border border-dashed border-border/70 px-3 py-8 text-center text-xs text-muted-foreground">
<Loader2 className="h-3.5 w-3.5 animate-spin" />
Loading sessions...
</div>
) : ( ) : (
<div className="rounded-md border border-dashed border-border/70 px-3 py-8 text-center text-xs text-muted-foreground"> <div className="rounded-md border border-dashed border-border/70 px-3 py-8 text-center text-xs text-muted-foreground">
No agent browser sessions. No agent browser sessions.
@@ -505,7 +690,7 @@ export default function BrowserUsePanel({ isVisible, onShowSettings }: BrowserUs
<div className="mt-3 grid grid-cols-2 gap-2"> <div className="mt-3 grid grid-cols-2 gap-2">
<Button variant="outline" size="sm" onClick={stopSession} disabled={isBusy || !selectedSession || selectedSession.status !== 'ready'}> <Button variant="outline" size="sm" onClick={stopSession} disabled={isBusy || !selectedSession || selectedSession.status !== 'ready'}>
<Square className="h-4 w-4" /> <Square className="h-4 w-4" />
Stop End
</Button> </Button>
<Button variant="outline" size="sm" onClick={deleteSession} disabled={isBusy || !selectedSession}> <Button variant="outline" size="sm" onClick={deleteSession} disabled={isBusy || !selectedSession}>
<Trash2 className="h-4 w-4" /> <Trash2 className="h-4 w-4" />

View File

@@ -200,7 +200,11 @@ function MainContent({
{shouldShowBrowserTab && activeTab === 'browser' && ( {shouldShowBrowserTab && activeTab === 'browser' && (
<div className="h-full overflow-hidden"> <div className="h-full overflow-hidden">
<BrowserUsePanel isVisible={activeTab === 'browser'} onShowSettings={onShowSettings} /> <BrowserUsePanel
isVisible={activeTab === 'browser'}
projectId={selectedProject.projectId}
onShowSettings={onShowSettings}
/>
</div> </div>
)} )}

View File

@@ -73,7 +73,15 @@ export default function MainContentTitle({
<h2 className="scrollbar-hide overflow-x-auto whitespace-nowrap text-sm font-semibold leading-tight text-foreground"> <h2 className="scrollbar-hide overflow-x-auto whitespace-nowrap text-sm font-semibold leading-tight text-foreground">
{getSessionTitle(selectedSession)} {getSessionTitle(selectedSession)}
</h2> </h2>
<div className="truncate text-[11px] leading-tight text-muted-foreground">{selectedProject.displayName}</div> <div className="flex min-w-0 items-center gap-2 text-[11px] leading-tight text-muted-foreground">
<span className="min-w-0 truncate">{selectedProject.displayName}</span>
<span
className="hidden min-w-0 max-w-[45%] flex-shrink truncate border-l border-border/60 pl-2 font-mono text-[10px] sm:block"
title={selectedSession.id}
>
{selectedSession.id}
</span>
</div>
</div> </div>
) : showChatNewSession ? ( ) : showChatNewSession ? (
<div className="min-w-0"> <div className="min-w-0">

View File

@@ -1,22 +1,32 @@
import { useCallback, useEffect, useState } from 'react'; import { useCallback, useEffect, useState } from 'react';
import { Download, Loader2 } from 'lucide-react'; import { Download, ExternalLink, Eye, Loader2, Zap } from 'lucide-react';
import { Button } from '../../../../../shared/view/ui'; import { Button, Input } from '../../../../../shared/view/ui';
import { authenticatedFetch } from '../../../../../utils/api'; import { authenticatedFetch } from '../../../../../utils/api';
import SettingsCard from '../../SettingsCard'; import SettingsCard from '../../SettingsCard';
import SettingsRow from '../../SettingsRow'; import SettingsRow from '../../SettingsRow';
import SettingsSection from '../../SettingsSection'; import SettingsSection from '../../SettingsSection';
import SettingsToggle from '../../SettingsToggle'; import SettingsToggle from '../../SettingsToggle';
const BROWSER_USE_GUIDE_URL = 'https://cloudcli.ai/docs/browser-use';
type BrowserUseSettings = { type BrowserUseSettings = {
enabled: boolean; enabled: boolean;
persistSessions: boolean;
defaultProfileName: string;
browserBackend: 'playwright' | 'camoufox-vnc';
}; };
type BrowserUseStatus = { type BrowserUseStatus = {
enabled: boolean; enabled: boolean;
available: boolean; available: boolean;
backend: 'playwright' | 'camoufox-vnc';
browserBackend: 'playwright' | 'camoufox-vnc';
playwrightInstalled: boolean; playwrightInstalled: boolean;
chromiumInstalled: boolean; chromiumInstalled: boolean;
camoufoxInstalled: boolean;
noVncInstalled: boolean;
x11vncInstalled: boolean;
installInProgress: boolean; installInProgress: boolean;
message: string; message: string;
}; };
@@ -32,16 +42,20 @@ async function readJson<T>(response: Response): Promise<T> {
export default function BrowserUseSettingsTab() { export default function BrowserUseSettingsTab() {
const [settings, setSettings] = useState<BrowserUseSettings | null>(null); const [settings, setSettings] = useState<BrowserUseSettings | null>(null);
const [status, setStatus] = useState<BrowserUseStatus | null>(null); const [status, setStatus] = useState<BrowserUseStatus | null>(null);
const [hasLoadedSettings, setHasLoadedSettings] = useState(false);
const [isSettingsLoading, setIsSettingsLoading] = useState(true); const [isSettingsLoading, setIsSettingsLoading] = useState(true);
const [isStatusLoading, setIsStatusLoading] = useState(true); const [isStatusLoading, setIsStatusLoading] = useState(true);
const [isSaving, setIsSaving] = useState(false); const [isSaving, setIsSaving] = useState(false);
const [isInstalling, setIsInstalling] = useState(false); const [isInstalling, setIsInstalling] = useState(false);
const [error, setError] = useState<string | null>(null); const [error, setError] = useState<string | null>(null);
const [profileNameDraft, setProfileNameDraft] = useState('default');
const loadSettings = useCallback(async () => { const loadSettings = useCallback(async () => {
const settingsResponse = await authenticatedFetch('/api/browser-use/settings'); const settingsResponse = await authenticatedFetch('/api/browser-use/settings');
const settingsData = await readJson<{ data: { settings: BrowserUseSettings } }>(settingsResponse); const settingsData = await readJson<{ data: { settings: BrowserUseSettings } }>(settingsResponse);
setSettings(settingsData.data.settings); setSettings(settingsData.data.settings);
setHasLoadedSettings(true);
setProfileNameDraft(settingsData.data.settings.defaultProfileName || 'default');
}, []); }, []);
const loadStatus = useCallback(async () => { const loadStatus = useCallback(async () => {
@@ -52,6 +66,7 @@ export default function BrowserUseSettingsTab() {
useEffect(() => { useEffect(() => {
setError(null); setError(null);
setHasLoadedSettings(false);
setIsSettingsLoading(true); setIsSettingsLoading(true);
setIsStatusLoading(true); setIsStatusLoading(true);
@@ -74,6 +89,7 @@ export default function BrowserUseSettingsTab() {
}); });
const data = await readJson<{ data: { settings: BrowserUseSettings } }>(response); const data = await readJson<{ data: { settings: BrowserUseSettings } }>(response);
setSettings(data.data.settings); setSettings(data.data.settings);
setHasLoadedSettings(true);
window.dispatchEvent(new Event('browserUseSettingsChanged')); window.dispatchEvent(new Event('browserUseSettingsChanged'));
setIsStatusLoading(true); setIsStatusLoading(true);
await loadStatus(); await loadStatus();
@@ -101,8 +117,21 @@ export default function BrowserUseSettingsTab() {
} }
}; };
const saveProfileName = async () => {
const nextName = profileNameDraft.trim() || 'default';
setProfileNameDraft(nextName);
if (nextName === settings?.defaultProfileName) {
return;
}
await updateSettings({ defaultProfileName: nextName });
};
const browserEnabled = settings?.enabled === true; const browserEnabled = settings?.enabled === true;
const needsBrowserBinaries = Boolean(browserEnabled && status && (!status.playwrightInstalled || !status.chromiumInstalled)); const browserDisabled = hasLoadedSettings && settings?.enabled === false;
const persistSessions = settings?.persistSessions === true;
const selectedBackend = settings?.browserBackend || 'playwright';
const effectiveBackend = status?.backend || 'playwright';
const needsBrowserBinaries = Boolean(browserEnabled && status && !status.available);
const runtimeLabel = (installed?: boolean) => { const runtimeLabel = (installed?: boolean) => {
if (isStatusLoading && !status) { if (isStatusLoading && !status) {
return 'checking...'; return 'checking...';
@@ -114,33 +143,165 @@ export default function BrowserUseSettingsTab() {
<div className="space-y-8"> <div className="space-y-8">
<SettingsSection <SettingsSection
title="Browser" title="Browser"
description="Allow agents to create guarded Playwright browser sessions that you can monitor from the Browser tab." description="Give coding agents a working browser so they can open websites, test flows, capture screenshots, and help debug what users actually see."
> >
<SettingsCard divided> <SettingsCard divided>
<SettingsRow <SettingsRow
label="Enable Browser" label="Give Agents Browser Access"
description="Registers Browser for supported agents. Agents can create browser sessions; you can watch, stop, and delete them." description="Let agents use a browser during coding tasks while you can watch live sessions, open them in a tab, and stop them at any time."
> >
{isSettingsLoading && !settings ? ( {isSettingsLoading && !hasLoadedSettings ? (
<Loader2 className="h-4 w-4 animate-spin text-muted-foreground" /> <Loader2 className="h-4 w-4 animate-spin text-muted-foreground" />
) : ( ) : hasLoadedSettings ? (
<SettingsToggle <SettingsToggle
checked={browserEnabled} checked={browserEnabled}
onChange={(value) => void updateSettings({ enabled: value })} onChange={(value) => void updateSettings({ enabled: value })}
ariaLabel="Enable Browser" ariaLabel="Give Agents Browser Access"
disabled={isSaving} disabled={isSaving}
/> />
) : (
<span className="text-sm text-muted-foreground">Unavailable</span>
)} )}
</SettingsRow> </SettingsRow>
{browserDisabled && (
<div className="px-4 py-4">
<a
href={BROWSER_USE_GUIDE_URL}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1.5 text-sm font-medium text-primary hover:underline"
>
Read the Browser guide
<ExternalLink className="h-3.5 w-3.5" />
</a>
</div>
)}
{error && (
<div className="px-4 py-4">
<div className="rounded-md border border-red-200 bg-red-50 px-3 py-2 text-sm text-red-700 dark:border-red-900/50 dark:bg-red-950/30 dark:text-red-200">
{error}
</div>
</div>
)}
{browserEnabled && (
<>
<div className="space-y-3 px-4 py-4">
<div className="min-w-0">
<div className="text-sm font-medium text-foreground">Browser Engine</div>
<div className="mt-0.5 text-sm text-muted-foreground">
Pick the kind of browser experience agents should use for new sessions.
</div>
</div>
<div className="grid gap-2 sm:grid-cols-2">
{([
{
value: 'playwright' as const,
label: 'Playwright',
description: 'Best for quick checks, screenshots, and automated page interaction when no manual login is needed.',
icon: Zap,
},
{
value: 'camoufox-vnc' as const,
label: 'Camoufox + noVNC',
description: 'Best when a person may need to log in, approve a step, or watch the browser session live.',
icon: Eye,
},
]).map((option) => {
const Icon = option.icon;
const selected = selectedBackend === option.value;
return (
<button
key={option.value}
type="button"
onClick={() => void updateSettings({ browserBackend: option.value })}
disabled={isSaving || isSettingsLoading}
className={[
'group flex min-h-[88px] items-start gap-3 rounded-lg border px-3 py-3 text-left transition-colors',
'focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 focus-visible:ring-offset-background',
selected
? 'border-primary bg-primary/5 text-foreground shadow-sm'
: 'border-border bg-background hover:border-foreground/20 hover:bg-muted/40',
(isSaving || isSettingsLoading) ? 'cursor-not-allowed opacity-60' : '',
].join(' ')}
aria-pressed={selected}
>
<span className={[
'mt-0.5 flex h-8 w-8 flex-shrink-0 items-center justify-center rounded-md border',
selected ? 'border-primary/30 bg-primary/10 text-primary' : 'border-border bg-muted/40 text-muted-foreground',
].join(' ')}
>
<Icon className="h-4 w-4" />
</span>
<span className="min-w-0">
<span className="block text-sm font-medium">{option.label}</span>
<span className="mt-1 block text-xs leading-relaxed text-muted-foreground">{option.description}</span>
</span>
</button>
);
})}
</div>
</div>
<SettingsRow
label="Remember Browser Logins"
description="Keep cookies and site storage in a named profile so agents can reuse signed-in sessions instead of starting from scratch."
>
{isSettingsLoading && !settings ? (
<Loader2 className="h-4 w-4 animate-spin text-muted-foreground" />
) : (
<SettingsToggle
checked={persistSessions}
onChange={(value) => void updateSettings({ persistSessions: value })}
ariaLabel="Remember Browser Logins"
disabled={isSaving}
/>
)}
</SettingsRow>
{persistSessions && (
<SettingsRow
label="Default Browser Profile"
description="New browser sessions use this profile by default, so saved logins stay tied to a predictable workspace."
>
<Input
value={profileNameDraft}
onChange={(event) => setProfileNameDraft(event.target.value)}
onBlur={() => void saveProfileName()}
onKeyDown={(event) => {
if (event.key === 'Enter') {
event.currentTarget.blur();
}
}}
disabled={isSaving || isSettingsLoading}
className="w-40"
aria-label="Default Browser Profile"
/>
</SettingsRow>
)}
</>
)}
{browserEnabled && (
<div className="space-y-4 px-4 py-4"> <div className="space-y-4 px-4 py-4">
<div className="flex flex-wrap gap-2 text-xs text-muted-foreground"> <div className="flex flex-wrap gap-2 text-xs text-muted-foreground">
<span className="rounded-md border border-border px-2 py-1">
Backend: {effectiveBackend === 'camoufox-vnc' ? 'Camoufox + noVNC' : 'Playwright'}
</span>
<span className="rounded-md border border-border px-2 py-1"> <span className="rounded-md border border-border px-2 py-1">
Playwright: {runtimeLabel(status?.playwrightInstalled)} Playwright: {runtimeLabel(status?.playwrightInstalled)}
</span> </span>
<span className="rounded-md border border-border px-2 py-1"> <span className="rounded-md border border-border px-2 py-1">
Chromium: {runtimeLabel(status?.chromiumInstalled)} Chromium: {runtimeLabel(status?.chromiumInstalled)}
</span> </span>
<span className="rounded-md border border-border px-2 py-1">
Camoufox: {runtimeLabel(status?.camoufoxInstalled)}
</span>
<span className="rounded-md border border-border px-2 py-1">
noVNC: {runtimeLabel(status?.noVncInstalled)}
</span>
<span className="rounded-md border border-border px-2 py-1"> <span className="rounded-md border border-border px-2 py-1">
Status: {isStatusLoading && !status ? 'checking...' : status?.available ? 'ready' : browserEnabled ? 'setup required' : 'disabled'} Status: {isStatusLoading && !status ? 'checking...' : status?.available ? 'ready' : browserEnabled ? 'setup required' : 'disabled'}
</span> </span>
@@ -172,12 +333,17 @@ export default function BrowserUseSettingsTab() {
</div> </div>
)} )}
{error && ( <a
<div className="rounded-md border border-red-200 bg-red-50 px-3 py-2 text-sm text-red-700 dark:border-red-900/50 dark:bg-red-950/30 dark:text-red-200"> href={BROWSER_USE_GUIDE_URL}
{error} target="_blank"
</div> rel="noopener noreferrer"
)} className="inline-flex items-center gap-1.5 text-sm font-medium text-primary hover:underline"
>
Read the Browser guide
<ExternalLink className="h-3.5 w-3.5" />
</a>
</div> </div>
)}
</SettingsCard> </SettingsCard>
</SettingsSection> </SettingsSection>
</div> </div>