mirror of
https://github.com/siteboon/claudecodeui.git
synced 2026-05-09 05:58:27 +00:00
Refactor provider/session architecture to be DB-driven, modular, and sessionId-first across backend and frontend (#715)
* refactor: remove unused exports
* refactor: remove unused fields from project and session objects
* refactor: rename session_names table and related code to sessions for clarity and consistency
* refactor(database): move db into typescript
- Implemented githubTokensDb for managing GitHub tokens with CRUD operations.
- Created
otificationPreferencesDb to handle user notification preferences.
- Added projectsDb for project path management and related operations.
- Introduced pushSubscriptionsDb for managing browser push subscriptions.
- Developed scanStateDb to track the last scanned timestamp.
- Established sessionsDb for session management with CRUD functionalities.
- Created userDb for user management, including authentication and onboarding.
- Implemented apidKeysDb for storing and managing VAPID keys.
feat(database): define schema for new database tables
- Added SQL schema definitions for users, API keys, user credentials, notification preferences, VAPID keys, push subscriptions, projects, sessions, scan state, and app configuration.
- Included necessary indexes for performance optimization.
refactor(shared): enhance type definitions and utility functions
- Updated shared types and interfaces for improved clarity and consistency.
- Added new types for credential management and provider-specific operations.
- Refined utility functions for better error handling and message normalization.
* feat: added session indexer logic
* perf(projects): lazy-load TaskMaster metadata per selected project
Why:
- /api/projects is a hot path (initial load, sidebar refresh, websocket sync).
- Scanning .taskmaster for every project on each call added avoidable fs I/O and payload size.
- TaskMaster metadata is only needed after selecting a specific project.
- Moving it to a project-scoped endpoint makes loading cost match user intent.
- The UI now hydrates TaskMaster state on selection and keeps it across refresh events.
- This prevents status flicker/regression while still removing global scan overhead.
- Selection fetches are sequence-guarded to block stale async responses on fast switching.
- isManuallyAdded was removed from responses to keep the public project contract minimal.
- Project dumps now use incrementing snapshot files to preserve history for debugging.
What changed:
- Added GET /api/projects/:projectName/taskmaster and getProjectTaskMaster().
- Removed TaskMaster detection from bulk getProjects().
- Added api.projectTaskmaster(...) plus selection-time hydration in frontend contexts.
- Merged cached taskmaster values into refreshed project lists for continuity.
- Removed isManuallyAdded from manual project payloads.
* refactor: update import paths for database modules and remove legacy db.js and schema.js files
* refactor(projects): identify projects by DB projectId instead of folder-derived name
GET /api/projects used to scan ~/.claude/projects/ on every request, derive
each project's identity from the encoded folder name, and re-parse JSONL
files to build session lists. Using the folder-derived name as the project
identifier leaked the Claude CLI's on-disk encoding into every API route,
forced every downstream endpoint to re-resolve a real path via JSONL
'cwd' inspection, and made the project list endpoint O(projects x sessions)
on disk I/O.
This change switches the entire API surface to identify projects by the
stable primary key from the 'projects' table and drives the listing
straight from the DB:
- Add projectsDb.getProjectPathById as the canonical projectId -> path
resolver so routes no longer need to touch the filesystem to figure out
where a project lives.
- Rewrite getProjects so it reads the project list from the 'projects'
table and the per-project session list from the 'sessions' table (one
SELECT per project). No filesystem scanning happens for this endpoint
anymore, which removes the dependency on ~/.claude/projects existing,
on Cursor's MD5-hashed chat folders being discoverable, and on Codex's
JSONL history being on disk. Per the migration spec each session now
exposes 'summary' sourced from sessions.custom_name, 'messageCount' = 0
(message counting is not implemented), and sessionMeta.hasMore is
pinned to false since this endpoint doesn't drive session pagination.
- Introduce id-based wrappers (getSessionsById, renameProjectById,
deleteSessionById, deleteProjectById, getProjectTaskMasterById) so
every caller can pass projectId and resolve the real path through the
DB. renameProjectById also writes to projects.custom_project_name so
the DB-driven getProjects response reflects renames immediately; it
keeps project-config.json in sync for any legacy reader that still
consults the JSON file.
- Migrate every /api/projects/:projectName route in server/index.js,
server/routes/taskmaster.js, and server/routes/messages.js to
:projectId, and change server/routes/git.js so the 'project'
query/body parameter carries a projectId that is resolved through the
DB before any git command runs. TaskMaster WebSocket broadcasts emit
'projectId' for the same reason so the frontend can match
notifications against its current selection without another lookup.
- Delete helpers that existed only to feed the old getProjects path
(getCursorSessions, getGeminiCliSessions, getProjectTaskMaster) along
with their unused imports (better-sqlite3's Database,
applyCustomSessionNames). The legacy folder-name helpers (getSessions,
renameProject, deleteSession, deleteProject, extractProjectDirectory)
are kept as internal implementation details of the id-based wrappers
and of destructive cleanup / conversation search, but they are no
longer re-exported.
- searchConversations still walks JSONL to produce match snippets (that
data doesn't live in the DB), but it now includes the resolved
projectId in each result so the sidebar can cross-reference hits with
its already loaded project list without a second round-trip.
Frontend migration:
- Project.name is replaced by Project.projectId in src/types/app.ts, and
ProjectSession.__projectName becomes __projectId so session tagging
and sidebar state keys stay aligned with the backend identifier.
Settings continues to use SettingsProject.name for legacy consumers,
but it is populated from projectId by normalizeProjectForSettings.
- All places that previously indexed per-project state by project.name
(sidebar expanded/starred/loading/deletingProjects sets,
additionalSessions map, projectHasMoreOverrides, starredProjects
localStorage, command history and draft-input localStorage,
TaskMaster caches) now key on projectId so state survives
display-name edits and is consistent across the app.
- src/utils/api.js renames every endpoint parameter to projectId, the
unified messages endpoint takes projectId in its query string, and
useSessionStore forwards projectId on fetchFromServer / fetchMore /
refreshFromServer. Git panel, file tree, code editor, PRD editor,
plugins context, MCP server flows and TaskMaster hooks are all
updated to pass projectId.
- DEFAULT_PROJECT_FOR_EMPTY_SHELL is updated to carry a 'default'
projectId sentinel so the empty-shell placeholder still satisfies the
Project contract.
Bug fix bundled in:
- sessionsDb.setName no longer bumps updated_at when a row already
exists. Renaming is a label change, not activity, so there is no
reason for it to reset 'last activity' in the sidebar. It also no
longer relies on SQLite's CURRENT_TIMESTAMP, which stores a naive
'YYYY-MM-DD HH:MM:SS' value that JavaScript parses as local time and
caused renamed sessions to appear shifted backwards by the client's
UTC offset. When an INSERT actually happens it now writes ISO-8601
UTC with a 'Z' suffix.
- buildSessionsByProviderFromDb normalizes any legacy naive timestamps
in the sessions table to ISO-8601 UTC on the way out so rows written
before this change also render correctly on the client.
Other cleanup:
- Removed the filesystem-first project-discovery comment block at the
top of server/projects.js and replaced it with a short note that
describes the new DB-driven flow and lists the few remaining
filesystem-dependent helpers (message reads, search, destructive
delete, manual project registration).
- server/modules/providers/index.ts is added as a small barrel so the
providers module exposes a stable public surface.
Made-with: Cursor
* refactor(projects): reorganize project-related logic into dedicated modules
* refactor(projects): rename getProjects with getProjectsWithSessions
* refactor: update import path for getProjectsWithSessions to include file extension
* refactor: use updated session watcher
In addition, for projects_updated websocket response, send the sessionId instead
* refactor(websocket): move websocket logic to its own module
* refactor(sessions-watcher): remove redundant logging after session sync completion
* refactor(index.js): reorganize code structure
* refactor(index.js): fix import order
* refactor: remove unnecessary GitHub cloning logic from create-workspace endpoint
* refactor: modularize project services, and wizard create/clone flow
Restructure project creation, listing, GitHub clone progress, and TaskMaster
details behind a dedicated TypeScript module under server/modules/projects/,
and align the client wizard with a single path-based flow.
Server / routing
- Remove server/routes/projects.js and mount server/modules/projects/
projects.routes.ts at /api/projects (still behind authenticateToken).
- Drop duplicate handlers from server/index.js for GET /api/projects and
GET /api/projects/:projectId/taskmaster; those live on the new router.
- Import WORKSPACES_ROOT and validateWorkspacePath from shared utils in
index.js instead of the deleted projects route module.
Projects router (projects.routes.ts)
- GET /: list projects with sessions (existing snapshot behavior).
- POST /create-project: validate body, reject legacy workspaceType and
mixed clone fields, delegate to createProject service, return distinct
success copy when an archived path is reactivated.
- GET /clone-progress: Server-Sent Events for clone progress/complete/error;
requires authenticated user id for token resolution; wires startCloneProject.
- GET /:projectId/taskmaster: delegates to getProjectTaskMaster.
Services (new)
- project-management.service.ts: path validation, workspace directory
creation, persistence via projectsDb.createProjectPath, mapping to API
project shape; surfaces AppError for validation, conflict, and not-found
cases; optional dependency injection for tests.
- project-clone.service.ts: validates workspace, resolves GitHub auth
(stored token or inline token), runs git clone with progress callbacks,
registers project via createProject on success; sanitizes errors and
supports cancellation; injectable dependencies for tests.
- projects-has-taskmaster.service.ts: moves TaskMaster detection and
normalization out of server/projects.js; resolve-by-id and public
getProjectTaskMaster with structured AppError responses.
Persistence and shared types
- projectsDb.createProjectPath now returns CreateProjectPathResult
(created | reactivated_archived | active_conflict) using INSERT … ON
CONFLICT with selective update when the row is archived; normalizes
display name from path or custom name; repository row typing moves to
shared ProjectRepositoryRow.
- getProjectPaths() returns only non-archived rows (isArchived = 0).
- shared/types.ts: ProjectRepositoryRow, CreateProjectPathResult/outcome,
WorkspacePathValidationResult.
- shared/utils.ts: WORKSPACES_ROOT, forbidden path lists, validateWorkspacePath,
asyncHandler for Express async routes.
Legacy cleanup
- server/projects.js: remove detectTaskMasterFolder, normalizeTaskMasterInfo,
and getProjectTaskMasterById (logic lives in the new service).
- server/routes/agent.js: register external API project paths with
projectsDb.createProjectPath instead of addProjectManually try/catch;
treat active_conflict as an existing registration and continue.
Tests
- Add Node test suites for project-management, project-clone, and
projects-has-taskmaster services; update projects.service test import
for renamed projects-with-sessions-fetch.service.ts.
Rename
- projects.service.ts → projects-with-sessions-fetch.service.ts;
re-export from modules/projects/index.ts.
Client (project creation wizard)
- Remove StepTypeSelection and workspaceType from form state and types;
wizard is two steps (configure path/GitHub auth, then review).
- createWorkspaceRequest → createProjectRequest; clone vs create-only
inferred from githubUrl (pathUtils / isCloneWorkflow).
- Adjust step indices, WizardProgress, StepConfiguration/Review,
WorkspacePathField, and src/utils/api.js as needed for the new API.
Docs
- Minor websocket README touch-up.
Net: ~1.6k insertions / ~0.9k deletions across 29 files; behavior is
centralized in typed services with explicit HTTP errors and test seams.
* refactor: remove loading sessions logic from sidebar
* refactor: move project rename to module
* refactor: move project deletion to module
* refactor: move project star state from localStorage to backend
* refactor: implement optimistic UI for project star state management
* feat: optimistic update for session watcher
* fix(projects-state): stop websocket message reprocessing loop
The websocket projects effect in useProjectsState could re-handle the same
latestMessage after local state writes triggered re-renders.
Under bursty websocket traffic, this created an update feedback cycle
that surfaced as 'Maximum update depth exceeded', often from Sidebar.
What changed:
- Added lastHandledMessageRef so each latestMessage object is handled once.
- Added an early return guard when the current message was already handled.
- Made projects updates idempotent by comparing previous and merged payloads
before calling setProjects.
Result:
- Breaks the effect -> state update -> effect re-entry cycle.
- Reduces redundant renders during rapid projects_updated traffic while
preserving normal project/session synchronization.
* refactor: optimize project auto-expand logic
* refactor: move projects provider specific logic into respective session providers
* refactor: move rename and delete sessions to modules
* refactor: move fetching messages to module
* fix: remove unused var
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* Potential fix for pull request finding 'Useless assignment to local variable'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* refactor(projects/sidebar): remove temp snapshot side-effects and simplify session metadata UX
Why this change was needed:
- Project listing had an implicit side effect: every fetch wrote a debug snapshot under `.tmp/project-dumps`.
That added unnecessary disk I/O to a hot path, introduced hidden runtime behavior, and created maintenance
overhead for code that was not part of product functionality.
- Keeping snapshot-specific exports/tests around made the projects module API broader than needed and coupled
tests to temporary/debug behavior instead of user-visible behavior.
- Codex sessions could remain stuck with a placeholder name (`Untitled Codex Session`) even after a real title
became available from newer sync data, which degraded session discoverability in the UI.
- Sidebar session rows showed duplicated provider branding and long-form relative times, which added visual noise
and reduced scan speed when many sessions are listed.
What changed:
- Removed temporary projects snapshot dumping from `projects-with-sessions-fetch.service.ts`:
- deleted snapshot types/helpers and file-write flow
- removed the write call from `getProjectsWithSessions`
- Removed snapshot-related surface area from `projects/index.ts`.
- Removed the snapshot-focused test `projects.service.test.ts` that only validated removed debug behavior.
- Updated `codex-session-synchronizer.provider.ts` to upgrade session names when an existing session still has
the placeholder title but a real parsed name is now available.
- Updated `SidebarSessionItem.tsx`:
- removed duplicate provider logo rendering in each session row
- moved age indicator to the right side
- made age indicator fade on hover to prioritize action controls
- switched to compact relative time format (`<1m`, `Xm`, `Xhr`, `Xd`) for faster list scanning
Outcome:
- Lower overhead and fewer hidden side effects in project fetches.
- Cleaner module boundaries in projects.
- Better Codex session naming consistency after sync.
- Cleaner sidebar density and clearer hover/action behavior.
* refactor: implement pagination for project sessions loading
* refactor: move search to module
* fix: search performance
* refactor: add handling for internal Codex metadata in conversation search
* fix(migrations,projects,clone): normalize legacy schema before writes and harden conflict detection
Why
- Legacy installs can have a sessions table shape that predates provider/custom_name columns. Running migrateLegacySessionNames first caused its INSERT OR REPLACE INTO sessions (...) to target columns that may not exist and fail during startup migration.
- Some upgraded databases had projects.project_id as plain TEXT instead of a real PRIMARY KEY. That breaks assumptions used by id-based lookups and can allow invalid/duplicate identity semantics over time.
- projectsDb.createProjectPath inferred outcomes from
ow.isArchived, but the upsert path always returns the post-update row with isArchived=0, so archived-reactivation and fresh-create could be misclassified.
- git clone accepted user-controlled URLs directly in argv position, so inputs beginning with - could be interpreted as options instead of a repository argument.
What
- Added
ebuildProjectsTableWithPrimaryKeySchema in migrations: detect table shape via getTableInfo('projects'), verify project_id has pk=1, and rebuild when missing.
- Rebuild flow now creates a canonical projects__new table (project_id TEXT PRIMARY KEY), copies rows with transformation, backfills empty ids via SQLITE_UUID_SQL, deduplicates conflicting ids/paths, then swaps tables inside a transaction.
- Replaced the prior ddColumnToTableIfNotExists(...) + UPDATE project_id sequence with PK-aware detection/rebuild logic so legacy DBs converge to the required schema.
- Reordered migration sequence to run
ebuildSessionsTableWithProjectSchema before migrateLegacySessionNames, ensuring sessions is normalized before legacy session_names merge writes execute.
- Updated projectsDb.createProjectPath to generate an ttemptedId before insert, pass it into the prepared statement, and classify outcomes by comparing returned
ow.project_id to ttemptedId (created vs
eactivated_archived), with no-row remaining ctive_conflict.
- Hardened clone execution by inserting -- before clone URL in git argv and rejecting normalized GitHub URLs that start with - in startCloneProject.
Tests
- Added integration coverage for projectsDb.createProjectPath branches: fresh insert, archived reactivation, and active conflict.
- Added clone service test for option-prefixed githubUrl rejection (INVALID_GITHUB_URL).
* refactor(session-synchronizer): update last scanned timestamp based on synchronization results
* refactor: improve session limit and offset validation in provider routes
* refactor: normalize project paths across database and service modules
* refactor(database): make session id the primary key in sessions table
* fix(codex): preserve reasoning entries as thinking blocks
Codex history normalization was downgrading reasoning into plain assistant text
because of branch ordering, not because the raw data was missing.
Why this mattered:
- Codex reasoning JSONL entries are intentionally mapped to history items with
type thinking, but they also carry message.role assistant.
- normalizeHistoryEntry evaluated the assistant-role branch before the
thinking branch.
- As a result, reasoning content matched the assistant-text path first and was
emitted as kind text instead of kind thinking.
- This collapses semantic intent, so UI and downstream features that rely on
thinking blocks (separate rendering, filtering, and interpretation of model
thought process vs final answer) receive the wrong message kind.
What changed:
- Prioritized thinking detection (raw.type === thinking or raw.isReasoning)
before role-based assistant normalization.
- Kept a non-empty content guard for thinking payloads to avoid emitting empty
artifacts.
Impact:
- Reasoning entries from persisted Codex JSONL now remain thinking blocks
end-to-end.
- Regular assistant text normalization behavior remains unchanged.
* refactor: remove dead code
* refactor: directly use getProjectPathById from projectsDb
* refactor: add gemini jsonl session support
This commit is contained in:
Haile
@@ -9,6 +9,7 @@ import type {
|
||||
UpsertProviderMcpServerInput,
|
||||
} from '@/shared/types.js';
|
||||
|
||||
//----------------- PROVIDER CONTRACT INTERFACES ------------
|
||||
/**
|
||||
* Main provider contract for CLI and SDK integrations.
|
||||
*
|
||||
@@ -20,11 +21,16 @@ export interface IProvider {
|
||||
readonly mcp: IProviderMcp;
|
||||
readonly auth: IProviderAuth;
|
||||
readonly sessions: IProviderSessions;
|
||||
readonly sessionSynchronizer: IProviderSessionSynchronizer;
|
||||
}
|
||||
|
||||
|
||||
// ---------------------------
|
||||
//----------------- PROVIDER AUTH INTERFACE ------------
|
||||
/**
|
||||
* Auth contract for one provider.
|
||||
*
|
||||
* Implementations should return a complete installation/authentication status
|
||||
* without throwing for normal "not installed" or "not authenticated" states.
|
||||
*/
|
||||
export interface IProviderAuth {
|
||||
/**
|
||||
@@ -33,8 +39,13 @@ export interface IProviderAuth {
|
||||
getStatus(): Promise<ProviderAuthStatus>;
|
||||
}
|
||||
|
||||
// ---------------------------
|
||||
//----------------- PROVIDER MCP INTERFACE ------------
|
||||
/**
|
||||
* MCP contract for one provider.
|
||||
*
|
||||
* Implementations must map provider-native MCP config formats to shared
|
||||
* `ProviderMcpServer` records used by routes and frontend state.
|
||||
*/
|
||||
export interface IProviderMcp {
|
||||
listServers(options?: { workspacePath?: string }): Promise<Record<McpScope, ProviderMcpServer[]>>;
|
||||
@@ -45,10 +56,37 @@ export interface IProviderMcp {
|
||||
): Promise<{ removed: boolean; provider: LLMProvider; name: string; scope: McpScope }>;
|
||||
}
|
||||
|
||||
// ---------------------------
|
||||
//----------------- PROVIDER SESSION INTERFACE ------------
|
||||
/**
|
||||
* Session/history contract for one provider.
|
||||
*
|
||||
* Implementations normalize provider-specific events and message history into
|
||||
* shared transport shapes consumed by API routes and realtime streams.
|
||||
*/
|
||||
export interface IProviderSessions {
|
||||
normalizeMessage(raw: unknown, sessionId: string | null): NormalizedMessage[];
|
||||
fetchHistory(sessionId: string, options?: FetchHistoryOptions): Promise<FetchHistoryResult>;
|
||||
}
|
||||
|
||||
// ---------------------------
|
||||
//----------------- PROVIDER SESSION SYNCHRONIZER INTERFACE ------------
|
||||
/**
|
||||
* Session indexing contract for one provider.
|
||||
*
|
||||
* Implementations scan provider-specific session artifacts on disk and upsert
|
||||
* normalized session metadata into the database. The service layer uses this
|
||||
* interface for both full rescans and single-file incremental sync triggered
|
||||
* by filesystem watcher events.
|
||||
*/
|
||||
export interface IProviderSessionSynchronizer {
|
||||
/**
|
||||
* Scans provider session artifacts and upserts discovered sessions into DB.
|
||||
*/
|
||||
synchronize(since?: Date): Promise<number>;
|
||||
|
||||
/**
|
||||
* Parses and upserts one provider artifact file without running a full scan.
|
||||
*/
|
||||
synchronizeFile(filePath: string): Promise<string | null>;
|
||||
}
|
||||
|
||||
@@ -1,18 +1,77 @@
|
||||
// -------------- HTTP API response shapes for the server, shared across modules --------------
|
||||
import type { IncomingMessage } from 'node:http';
|
||||
|
||||
//----------------- HTTP RESPONSE SHAPES ------------
|
||||
/**
|
||||
* Canonical success envelope used by backend APIs that return a structured payload.
|
||||
*
|
||||
* Use this for route handlers that need a stable `success/data` shape so frontend
|
||||
* consumers can parse responses consistently across endpoints.
|
||||
*/
|
||||
export type ApiSuccessShape<TData = unknown> = {
|
||||
success: true;
|
||||
data: TData;
|
||||
};
|
||||
|
||||
/**
|
||||
* Generic plain-object record used when parsing loosely typed JSON payloads.
|
||||
*
|
||||
* Use this only after runtime shape checks, not as a replacement for validated
|
||||
* domain models.
|
||||
*/
|
||||
export type AnyRecord = Record<string, any>;
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// ---------------------------
|
||||
//----------------- WEBSOCKET TRANSPORT TYPES ------------
|
||||
/**
|
||||
* Minimal websocket client contract used by backend broadcaster services.
|
||||
*
|
||||
* Any transport object added to `connectedClients` must implement these two
|
||||
* members so shared services can safely send JSON strings and check whether the
|
||||
* socket is still open before broadcasting.
|
||||
*/
|
||||
export type RealtimeClientConnection = {
|
||||
readyState: number;
|
||||
send(data: string): void;
|
||||
};
|
||||
|
||||
/**
|
||||
* Authenticated user payload attached to websocket upgrade requests.
|
||||
*
|
||||
* Platform and OSS auth flows currently use either `id` or `userId`; both are
|
||||
* represented here so websocket handlers can resolve a stable writer user id.
|
||||
*/
|
||||
export type AuthenticatedWebSocketUser = {
|
||||
id?: string | number;
|
||||
userId?: string | number;
|
||||
username?: string;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
|
||||
/**
|
||||
* HTTP upgrade request shape after websocket authentication succeeds.
|
||||
*
|
||||
* `verifyClient` populates `request.user` with the authenticated payload, and
|
||||
* downstream websocket handlers rely on this extended request type.
|
||||
*/
|
||||
export type AuthenticatedWebSocketRequest = IncomingMessage & {
|
||||
user?: AuthenticatedWebSocketUser;
|
||||
};
|
||||
|
||||
// ---------------------------
|
||||
//----------------- PROVIDER MESSAGE MODEL ------------
|
||||
/**
|
||||
* Providers supported by the unified server runtime.
|
||||
*
|
||||
* Use this as the source of truth whenever a function or payload needs to identify
|
||||
* a specific LLM integration.
|
||||
*/
|
||||
export type LLMProvider = 'claude' | 'codex' | 'gemini' | 'cursor';
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Message/event variants emitted by provider adapters and normalized transports.
|
||||
*
|
||||
* Keep this union in sync with event kinds produced by provider session adapters.
|
||||
*/
|
||||
export type MessageKind =
|
||||
| 'text'
|
||||
| 'tool_use'
|
||||
@@ -30,11 +89,10 @@ export type MessageKind =
|
||||
| 'task_notification';
|
||||
|
||||
/**
|
||||
* Provider-neutral message event emitted over REST and realtime transports.
|
||||
* Provider-neutral message envelope used in REST responses and realtime channels.
|
||||
*
|
||||
* Providers all produce their own native SDK/CLI event shapes, so this type keeps
|
||||
* the common envelope strict while allowing provider-specific details to ride
|
||||
* along as optional properties.
|
||||
* Every provider-specific message must be converted into this shape before being
|
||||
* emitted outside provider-specific modules.
|
||||
*/
|
||||
export type NormalizedMessage = {
|
||||
id: string;
|
||||
@@ -73,21 +131,21 @@ export type NormalizedMessage = {
|
||||
};
|
||||
|
||||
/**
|
||||
* Pagination and provider lookup options for reading persisted session history.
|
||||
* Shared options used to fetch historical provider messages.
|
||||
*
|
||||
* Consumers should pass provider-specific lookup hints (`projectPath`) only
|
||||
* when the selected provider requires them.
|
||||
*/
|
||||
export type FetchHistoryOptions = {
|
||||
/** Claude project folder name. Required by Claude history lookup. */
|
||||
projectName?: string;
|
||||
/** Absolute workspace path. Required by Cursor to compute its chat hash. */
|
||||
projectPath?: string;
|
||||
/** Page size. `null` means all messages. */
|
||||
limit?: number | null;
|
||||
/** Pagination offset from the newest messages. */
|
||||
offset?: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Provider-neutral history result returned by the unified messages endpoint.
|
||||
* Standardized response payload returned from provider history readers.
|
||||
*
|
||||
* Use this as the contract for APIs that return paginated conversation history.
|
||||
*/
|
||||
export type FetchHistoryResult = {
|
||||
messages: NormalizedMessage[];
|
||||
@@ -98,21 +156,40 @@ export type FetchHistoryResult = {
|
||||
tokenUsage?: unknown;
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
|
||||
// ---------------------------
|
||||
//----------------- SHARED ERROR TYPES ------------
|
||||
/**
|
||||
* Optional metadata used when constructing application-level errors.
|
||||
*
|
||||
* `statusCode` should reflect the HTTP response status, while `code` identifies
|
||||
* the stable machine-readable error category.
|
||||
*/
|
||||
export type AppErrorOptions = {
|
||||
code?: string;
|
||||
statusCode?: number;
|
||||
details?: unknown;
|
||||
};
|
||||
|
||||
// -------------------- MCP related shared types --------------------
|
||||
// ---------------------------
|
||||
//----------------- MCP TYPES ------------
|
||||
/**
|
||||
* Scope where an MCP server definition is stored and resolved.
|
||||
*
|
||||
* `user` is global for a user account, `local` is provider-local, and `project`
|
||||
* is tied to a specific project path.
|
||||
*/
|
||||
export type McpScope = 'user' | 'local' | 'project';
|
||||
|
||||
/**
|
||||
* Transport protocol used by an MCP server definition.
|
||||
*/
|
||||
export type McpTransport = 'stdio' | 'http' | 'sse';
|
||||
|
||||
/**
|
||||
* Provider MCP server descriptor normalized for frontend consumption.
|
||||
* Normalized MCP server model exposed to frontend and route handlers.
|
||||
*
|
||||
* Provider adapters should map provider-native config to this structure before
|
||||
* returning results.
|
||||
*/
|
||||
export type ProviderMcpServer = {
|
||||
provider: LLMProvider;
|
||||
@@ -131,7 +208,10 @@ export type ProviderMcpServer = {
|
||||
};
|
||||
|
||||
/**
|
||||
* Shared payload shape for MCP server create/update operations.
|
||||
* Payload for create/update MCP server operations.
|
||||
*
|
||||
* Routes and services should accept this type, validate it, and then persist it
|
||||
* through provider-specific MCP repositories.
|
||||
*/
|
||||
export type UpsertProviderMcpServerInput = {
|
||||
name: string;
|
||||
@@ -149,18 +229,13 @@ export type UpsertProviderMcpServerInput = {
|
||||
envHttpHeaders?: Record<string, string>;
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
|
||||
// -------------------- Provider auth status types --------------------
|
||||
// ---------------------------
|
||||
//----------------- PROVIDER AUTH TYPES ------------
|
||||
/**
|
||||
* Result of a provider status check (installation + authentication).
|
||||
* Authentication status result returned by provider health checks.
|
||||
*
|
||||
* installed - Whether the provider's CLI/SDK is available
|
||||
* provider - Provider id the status belongs to
|
||||
* authenticated - Whether valid credentials exist
|
||||
* email - User email or auth method identifier
|
||||
* method - Auth method (e.g. 'api_key', 'credentials_file')
|
||||
* [error] - Error message if not installed or not authenticated
|
||||
* This shape is consumed by settings/status endpoints to report installation and
|
||||
* credential state for each provider.
|
||||
*/
|
||||
export type ProviderAuthStatus = {
|
||||
installed: boolean;
|
||||
@@ -170,3 +245,83 @@ export type ProviderAuthStatus = {
|
||||
method: string | null;
|
||||
error?: string;
|
||||
};
|
||||
|
||||
// ---------------------------
|
||||
//----------------- SHARED DATABASE CREDENTIAL TYPES ------------
|
||||
/**
|
||||
* Safe credential view returned by credential listing APIs.
|
||||
*
|
||||
* This intentionally excludes the raw credential secret while still exposing
|
||||
* metadata needed for UI rendering and management operations.
|
||||
*/
|
||||
export type CredentialPublicRow = {
|
||||
id: number;
|
||||
credential_name: string;
|
||||
credential_type: string;
|
||||
description: string | null;
|
||||
created_at: string;
|
||||
is_active: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Result returned after creating a credential record.
|
||||
*
|
||||
* Use this return shape when callers need the created id and display metadata,
|
||||
* but must never receive the stored secret value.
|
||||
*/
|
||||
export type CreateCredentialResult = {
|
||||
id: number | bigint;
|
||||
credentialName: string;
|
||||
credentialType: string;
|
||||
};
|
||||
|
||||
// ---------------------------
|
||||
//----------------- PROJECT PERSISTENCE TYPES ------------
|
||||
/**
|
||||
* Canonical project row shape returned by the projects repository.
|
||||
*
|
||||
* Use this type whenever backend services need to pass around one database
|
||||
* project record without leaking raw SQL row typing across modules.
|
||||
*/
|
||||
export type ProjectRepositoryRow = {
|
||||
project_id: string;
|
||||
project_path: string;
|
||||
custom_project_name: string | null;
|
||||
isStarred: number;
|
||||
isArchived: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Result category returned by `projectsDb.createProjectPath`.
|
||||
*
|
||||
* `created` means a fresh row was inserted, `reactivated_archived` means an
|
||||
* existing archived path was accepted and updated, and `active_conflict` means
|
||||
* an already-active path blocked project creation.
|
||||
*/
|
||||
export type CreateProjectPathOutcome =
|
||||
| 'created'
|
||||
| 'reactivated_archived'
|
||||
| 'active_conflict';
|
||||
|
||||
/**
|
||||
* Structured result returned by project-path upsert operations.
|
||||
*
|
||||
* Services should use this result to decide whether a request succeeded,
|
||||
* should return a conflict, or needs follow-up retrieval of row metadata.
|
||||
*/
|
||||
export type CreateProjectPathResult = {
|
||||
outcome: CreateProjectPathOutcome;
|
||||
project: ProjectRepositoryRow | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Validation result for user-supplied workspace/project paths.
|
||||
*
|
||||
* `resolvedPath` is present only when validation succeeds. `error` is present
|
||||
* only when validation fails and is suitable for user-facing diagnostics.
|
||||
*/
|
||||
export type WorkspacePathValidationResult = {
|
||||
valid: boolean;
|
||||
resolvedPath?: string;
|
||||
error?: string;
|
||||
};
|
||||
|
||||
@@ -1,7 +1,19 @@
|
||||
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { mkdir, readFile, writeFile } from 'node:fs/promises';
|
||||
import fs from 'node:fs';
|
||||
import {
|
||||
access,
|
||||
lstat,
|
||||
mkdir,
|
||||
readFile,
|
||||
readdir,
|
||||
readlink,
|
||||
realpath,
|
||||
stat,
|
||||
writeFile,
|
||||
} from 'node:fs/promises';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import readline from 'node:readline';
|
||||
|
||||
import type { NextFunction, Request, RequestHandler, Response } from 'express';
|
||||
|
||||
@@ -10,8 +22,17 @@ import type {
|
||||
ApiSuccessShape,
|
||||
AppErrorOptions,
|
||||
NormalizedMessage,
|
||||
WorkspacePathValidationResult,
|
||||
} from '@/shared/types.js';
|
||||
|
||||
//----------------- NORMALIZED MESSAGE HELPER INPUT TYPES ------------
|
||||
/**
|
||||
* Input payload accepted by `createNormalizedMessage`.
|
||||
*
|
||||
* Callers provide provider-specific fields plus the required `kind/provider`
|
||||
* pair; this helper fills missing envelope fields (`id`, `sessionId`,
|
||||
* `timestamp`) in a consistent way.
|
||||
*/
|
||||
type NormalizedMessageInput =
|
||||
{
|
||||
kind: NormalizedMessage['kind'];
|
||||
@@ -21,6 +42,14 @@ type NormalizedMessageInput =
|
||||
timestamp?: string | null;
|
||||
} & Record<string, unknown>;
|
||||
|
||||
// ---------------------------
|
||||
//----------------- HTTP HANDLER UTILITIES ------------
|
||||
/**
|
||||
* Wraps arbitrary data in the standard API success envelope.
|
||||
*
|
||||
* Use this helper in route handlers to keep successful JSON responses consistent
|
||||
* across endpoints.
|
||||
*/
|
||||
export function createApiSuccessResponse<TData>(
|
||||
data: TData,
|
||||
): ApiSuccessShape<TData> {
|
||||
@@ -30,6 +59,12 @@ export function createApiSuccessResponse<TData>(
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts an async Express handler into a standard `RequestHandler` and routes
|
||||
* rejected promises to Express error middleware.
|
||||
*
|
||||
* Use this to avoid repeating `try/catch(next)` in every async route.
|
||||
*/
|
||||
export function asyncHandler(
|
||||
handler: (req: Request, res: Response, next: NextFunction) => Promise<unknown>
|
||||
): RequestHandler {
|
||||
@@ -38,7 +73,14 @@ export function asyncHandler(
|
||||
};
|
||||
}
|
||||
|
||||
// --------- Global app error class for consistent error handling across the server ---------
|
||||
// ---------------------------
|
||||
//----------------- SHARED ERROR UTILITIES ------------
|
||||
/**
|
||||
* Shared application error with HTTP status and machine-readable code metadata.
|
||||
*
|
||||
* Throw this from service/route layers when the caller should receive a
|
||||
* controlled error response rather than a generic 500.
|
||||
*/
|
||||
export class AppError extends Error {
|
||||
readonly code: string;
|
||||
readonly statusCode: number;
|
||||
@@ -53,9 +95,226 @@ export class AppError extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// ---------------------------
|
||||
//----------------- WORKSPACE PATH VALIDATION UTILITIES ------------
|
||||
/**
|
||||
* Root directory that all workspace/project paths must stay under.
|
||||
*
|
||||
* This is resolved from `WORKSPACES_ROOT` when configured; otherwise it falls
|
||||
* back to the current user's home directory.
|
||||
*/
|
||||
export const WORKSPACES_ROOT = process.env.WORKSPACES_ROOT || os.homedir();
|
||||
|
||||
// ------------------------ Normalized provider message helpers ------------------------
|
||||
/**
|
||||
* System-critical paths that must never be used as workspace roots.
|
||||
*
|
||||
* The validation helper blocks these values directly and also blocks paths
|
||||
* nested under them (with explicit allow-list exceptions where necessary).
|
||||
*/
|
||||
export const FORBIDDEN_WORKSPACE_PATHS = [
|
||||
// Unix
|
||||
'/',
|
||||
'/etc',
|
||||
'/bin',
|
||||
'/sbin',
|
||||
'/usr',
|
||||
'/dev',
|
||||
'/proc',
|
||||
'/sys',
|
||||
'/var',
|
||||
'/boot',
|
||||
'/root',
|
||||
'/lib',
|
||||
'/lib64',
|
||||
'/opt',
|
||||
'/tmp',
|
||||
'/run',
|
||||
// Windows
|
||||
'C:\\Windows',
|
||||
'C:\\Program Files',
|
||||
'C:\\Program Files (x86)',
|
||||
'C:\\ProgramData',
|
||||
'C:\\System Volume Information',
|
||||
'C:\\$Recycle.Bin',
|
||||
];
|
||||
|
||||
function stripWindowsLongPathPrefix(inputPath: string): string {
|
||||
if (inputPath.startsWith('\\\\?\\UNC\\')) {
|
||||
return `\\\\${inputPath.slice('\\\\?\\UNC\\'.length)}`;
|
||||
}
|
||||
|
||||
if (inputPath.startsWith('\\\\?\\')) {
|
||||
return inputPath.slice('\\\\?\\'.length);
|
||||
}
|
||||
|
||||
return inputPath;
|
||||
}
|
||||
|
||||
function shouldUseWindowsPathNormalization(inputPath: string): boolean {
|
||||
if (process.platform === 'win32') {
|
||||
return true;
|
||||
}
|
||||
|
||||
return inputPath.startsWith('\\\\') || /^[a-zA-Z]:([\\/]|$)/.test(inputPath);
|
||||
}
|
||||
|
||||
/**
|
||||
* Canonicalizes project/workspace paths for stable DB keys and comparisons.
|
||||
*
|
||||
* Normalization rules:
|
||||
* - trim whitespace
|
||||
* - strip Windows long-path prefixes (`\\?\` and `\\?\UNC\`)
|
||||
* - normalize path separators and dot segments
|
||||
* - trim trailing separators except for filesystem roots
|
||||
*/
|
||||
export function normalizeProjectPath(inputPath: string): string {
|
||||
if (typeof inputPath !== 'string') {
|
||||
return '';
|
||||
}
|
||||
|
||||
const trimmed = inputPath.trim();
|
||||
if (!trimmed) {
|
||||
return '';
|
||||
}
|
||||
|
||||
const withoutLongPrefix = stripWindowsLongPathPrefix(trimmed);
|
||||
const useWindowsPathRules = shouldUseWindowsPathNormalization(withoutLongPrefix);
|
||||
const normalized = useWindowsPathRules
|
||||
? path.win32.normalize(withoutLongPrefix)
|
||||
: path.posix.normalize(withoutLongPrefix);
|
||||
|
||||
if (!normalized) {
|
||||
return '';
|
||||
}
|
||||
|
||||
const parser = useWindowsPathRules ? path.win32 : path.posix;
|
||||
const root = parser.parse(normalized).root;
|
||||
if (normalized === root) {
|
||||
return normalized;
|
||||
}
|
||||
|
||||
return normalized.replace(/[\\/]+$/, '');
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates that a user-supplied workspace path is safe to use.
|
||||
*
|
||||
* Call this before any filesystem mutation that creates or registers projects.
|
||||
* The function resolves symlinks, enforces `WORKSPACES_ROOT` containment, and
|
||||
* blocks known system directories.
|
||||
*/
|
||||
export async function validateWorkspacePath(requestedPath: string): Promise<WorkspacePathValidationResult> {
|
||||
try {
|
||||
const normalizedRequestedPath = normalizeProjectPath(requestedPath);
|
||||
if (!normalizedRequestedPath) {
|
||||
return {
|
||||
valid: false,
|
||||
error: 'Workspace path is required',
|
||||
};
|
||||
}
|
||||
|
||||
const absolutePath = path.resolve(normalizedRequestedPath);
|
||||
const normalizedPath = normalizeProjectPath(absolutePath);
|
||||
|
||||
if (FORBIDDEN_WORKSPACE_PATHS.includes(normalizedPath) || normalizedPath === '/') {
|
||||
return {
|
||||
valid: false,
|
||||
error: 'Cannot use system-critical directories as workspace locations',
|
||||
};
|
||||
}
|
||||
|
||||
for (const forbiddenPath of FORBIDDEN_WORKSPACE_PATHS) {
|
||||
const normalizedForbiddenPath = normalizeProjectPath(forbiddenPath);
|
||||
if (
|
||||
normalizedPath === normalizedForbiddenPath
|
||||
|| normalizedPath.startsWith(`${normalizedForbiddenPath}${path.sep}`)
|
||||
) {
|
||||
// Allow specific user-writable folders under /var.
|
||||
if (
|
||||
normalizedForbiddenPath === '/var'
|
||||
&& (normalizedPath.startsWith('/var/tmp') || normalizedPath.startsWith('/var/folders'))
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
return {
|
||||
valid: false,
|
||||
error: `Cannot create workspace in system directory: ${forbiddenPath}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
let resolvedPath = normalizeProjectPath(absolutePath);
|
||||
try {
|
||||
await access(absolutePath);
|
||||
resolvedPath = normalizeProjectPath(await realpath(absolutePath));
|
||||
} catch (error) {
|
||||
const fileError = error as NodeJS.ErrnoException;
|
||||
if (fileError.code !== 'ENOENT') {
|
||||
throw fileError;
|
||||
}
|
||||
|
||||
const parentPath = path.dirname(absolutePath);
|
||||
try {
|
||||
const parentRealPath = await realpath(parentPath);
|
||||
resolvedPath = normalizeProjectPath(path.join(parentRealPath, path.basename(absolutePath)));
|
||||
} catch (parentError) {
|
||||
const parentFileError = parentError as NodeJS.ErrnoException;
|
||||
if (parentFileError.code !== 'ENOENT') {
|
||||
throw parentFileError;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const resolvedWorkspaceRoot = normalizeProjectPath(await realpath(WORKSPACES_ROOT));
|
||||
if (
|
||||
!resolvedPath.startsWith(`${resolvedWorkspaceRoot}${path.sep}`)
|
||||
&& resolvedPath !== resolvedWorkspaceRoot
|
||||
) {
|
||||
return {
|
||||
valid: false,
|
||||
error: `Workspace path must be within the allowed workspace root: ${WORKSPACES_ROOT}`,
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
await access(absolutePath);
|
||||
const pathStats = await lstat(absolutePath);
|
||||
if (pathStats.isSymbolicLink()) {
|
||||
const symlinkTarget = await readlink(absolutePath);
|
||||
const resolvedSymlinkPath = path.resolve(path.dirname(absolutePath), symlinkTarget);
|
||||
const realSymlinkPath = await realpath(resolvedSymlinkPath);
|
||||
if (
|
||||
!realSymlinkPath.startsWith(`${resolvedWorkspaceRoot}${path.sep}`)
|
||||
&& realSymlinkPath !== resolvedWorkspaceRoot
|
||||
) {
|
||||
return {
|
||||
valid: false,
|
||||
error: 'Symlink target is outside the allowed workspace root',
|
||||
};
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
const fileError = error as NodeJS.ErrnoException;
|
||||
if (fileError.code !== 'ENOENT') {
|
||||
throw fileError;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
valid: true,
|
||||
resolvedPath,
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
valid: false,
|
||||
error: `Path validation failed: ${(error as Error).message}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------
|
||||
//----------------- NORMALIZED PROVIDER MESSAGE UTILITIES ------------
|
||||
/**
|
||||
* Generates a stable unique id for normalized provider messages.
|
||||
*/
|
||||
@@ -80,9 +339,8 @@ export function createNormalizedMessage(fields: NormalizedMessageInput): Normali
|
||||
};
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
|
||||
// ------------------------ The following are mainly for provider MCP runtimes ------------------------
|
||||
// ---------------------------
|
||||
//----------------- MCP CONFIG PARSING UTILITIES ------------
|
||||
/**
|
||||
* Safely narrows an unknown value to a plain object record.
|
||||
*
|
||||
@@ -154,6 +412,62 @@ export const readStringRecord = (value: unknown): Record<string, string> | undef
|
||||
return Object.keys(normalized).length > 0 ? normalized : undefined;
|
||||
};
|
||||
|
||||
// ---------------------------
|
||||
//----------------- WEBSOCKET PAYLOAD PARSING UTILITIES ------------
|
||||
/**
|
||||
* Parses one websocket message payload into a plain JSON object record.
|
||||
*
|
||||
* Use this in realtime handlers that receive raw websocket payloads as `string`,
|
||||
* `Buffer`, `ArrayBuffer`, or chunk arrays. The helper converts supported
|
||||
* payload formats to UTF-8 text, parses JSON, and returns only object payloads.
|
||||
* Primitive/array/invalid payloads return `null` so callers can handle bad input
|
||||
* without throwing from deeply nested message handlers.
|
||||
*/
|
||||
export const parseIncomingJsonObject = (payload: unknown): AnyRecord | null => {
|
||||
let text: string | null = null;
|
||||
|
||||
if (typeof payload === 'string') {
|
||||
text = payload;
|
||||
} else if (Buffer.isBuffer(payload)) {
|
||||
text = payload.toString('utf8');
|
||||
} else if (payload instanceof ArrayBuffer) {
|
||||
text = Buffer.from(payload).toString('utf8');
|
||||
} else if (Array.isArray(payload)) {
|
||||
const buffers = payload
|
||||
.map((entry) => {
|
||||
if (Buffer.isBuffer(entry)) {
|
||||
return entry;
|
||||
}
|
||||
|
||||
if (entry instanceof ArrayBuffer) {
|
||||
return Buffer.from(entry);
|
||||
}
|
||||
|
||||
if (ArrayBuffer.isView(entry)) {
|
||||
return Buffer.from(entry.buffer, entry.byteOffset, entry.byteLength);
|
||||
}
|
||||
|
||||
return null;
|
||||
})
|
||||
.filter((entry): entry is Buffer => entry !== null);
|
||||
|
||||
if (buffers.length > 0) {
|
||||
text = Buffer.concat(buffers).toString('utf8');
|
||||
}
|
||||
}
|
||||
|
||||
if (typeof text !== 'string' || text.trim().length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
const parsed = JSON.parse(text) as unknown;
|
||||
return readObjectRecord(parsed);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Reads a JSON config file and guarantees a plain object result.
|
||||
*
|
||||
@@ -189,5 +503,167 @@ export const writeJsonConfig = async (filePath: string, data: Record<string, unk
|
||||
await writeFile(filePath, `${JSON.stringify(data, null, 2)}\n`, 'utf8');
|
||||
};
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// ---------------------------
|
||||
//----------------- SESSION SYNCHRONIZER TITLE HELPERS ------------
|
||||
/**
|
||||
* Produces a compact session title suitable for UI rendering and DB storage.
|
||||
*
|
||||
* Use this when converting provider-native names into a consistent title value.
|
||||
* The helper collapses repeated whitespace, trims the result, and truncates it
|
||||
* to 120 characters so every provider writes stable and bounded metadata.
|
||||
* If the normalized input is empty, it returns the supplied fallback title.
|
||||
*/
|
||||
export function normalizeSessionName(rawValue: string | undefined, fallback: string): string {
|
||||
const normalized = (rawValue ?? '').replace(/\s+/g, ' ').trim();
|
||||
if (!normalized) {
|
||||
return fallback;
|
||||
}
|
||||
|
||||
return normalized.slice(0, 120);
|
||||
}
|
||||
|
||||
// ---------------------------
|
||||
//----------------- SESSION SYNCHRONIZER FILESYSTEM HELPERS ------------
|
||||
/**
|
||||
* Recursively discovers files that match one extension, with optional incremental filtering.
|
||||
*
|
||||
* Provider synchronizers call this to find transcript artifacts under provider
|
||||
* home directories. Pass `lastScanAt` to include only files created after the
|
||||
* previous scan, or pass `null` to perform a full rescan. Missing directories
|
||||
* are treated as empty because not every provider exists on every machine.
|
||||
*/
|
||||
export async function findFilesRecursivelyCreatedAfter(
|
||||
rootDir: string,
|
||||
extension: string,
|
||||
lastScanAt: Date | null,
|
||||
fileList: string[] = []
|
||||
): Promise<string[]> {
|
||||
try {
|
||||
const entries = await readdir(rootDir, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
const fullPath = path.join(rootDir, entry.name);
|
||||
|
||||
if (entry.isDirectory()) {
|
||||
await findFilesRecursivelyCreatedAfter(fullPath, extension, lastScanAt, fileList);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!entry.isFile() || !entry.name.endsWith(extension)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!lastScanAt) {
|
||||
fileList.push(fullPath);
|
||||
continue;
|
||||
}
|
||||
|
||||
const fileStat = await stat(fullPath);
|
||||
if (fileStat.birthtime > lastScanAt) {
|
||||
fileList.push(fullPath);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Missing provider folders are expected in first-run or partial setups.
|
||||
}
|
||||
|
||||
return fileList;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads file creation/update timestamps and maps them to DB-friendly ISO strings.
|
||||
*
|
||||
* Session indexers use this to persist `created_at` and `updated_at` metadata
|
||||
* when upserting sessions. If the file cannot be read, an empty object is
|
||||
* returned so indexing can continue for other files.
|
||||
*/
|
||||
export async function readFileTimestamps(
|
||||
filePath: string
|
||||
): Promise<{ createdAt?: string; updatedAt?: string }> {
|
||||
try {
|
||||
const fileStat = await stat(filePath);
|
||||
return {
|
||||
createdAt: fileStat.birthtime.toISOString(),
|
||||
updatedAt: fileStat.mtime.toISOString(),
|
||||
};
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------
|
||||
//----------------- SESSION SYNCHRONIZER JSONL PARSING HELPERS ------------
|
||||
/**
|
||||
* Builds a first-seen key/value lookup map from a JSONL file.
|
||||
*
|
||||
* Use this for provider index files where session id -> display name metadata
|
||||
* is stored line-by-line. The first value for each key wins, preserving the
|
||||
* earliest known label while avoiding repeated map overwrites.
|
||||
*/
|
||||
export async function buildLookupMap(
|
||||
filePath: string,
|
||||
keyField: string,
|
||||
valueField: string
|
||||
): Promise<Map<string, string>> {
|
||||
const lookup = new Map<string, string>();
|
||||
|
||||
try {
|
||||
const fileStream = fs.createReadStream(filePath);
|
||||
const lineReader = readline.createInterface({ input: fileStream, crlfDelay: Infinity });
|
||||
|
||||
for await (const line of lineReader) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const parsed = JSON.parse(trimmed) as Record<string, unknown>;
|
||||
const key = parsed[keyField];
|
||||
const value = parsed[valueField];
|
||||
|
||||
if (typeof key === 'string' && typeof value === 'string' && !lookup.has(key)) {
|
||||
lookup.set(key, value);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Missing or unreadable lookup files should not block session sync.
|
||||
}
|
||||
|
||||
return lookup;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads a JSONL file and returns the first extracted payload that matches caller criteria.
|
||||
*
|
||||
* The caller supplies an `extractor` that validates provider-specific row
|
||||
* shapes. This helper centralizes line-by-line parsing and lets indexers stop
|
||||
* scanning as soon as one valid row is found.
|
||||
*/
|
||||
export async function extractFirstValidJsonlData<T>(
|
||||
filePath: string,
|
||||
extractor: (parsedJson: unknown) => T | null | undefined
|
||||
): Promise<T | null> {
|
||||
try {
|
||||
const fileStream = fs.createReadStream(filePath);
|
||||
const lineReader = readline.createInterface({ input: fileStream, crlfDelay: Infinity });
|
||||
|
||||
for await (const line of lineReader) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const parsed = JSON.parse(trimmed);
|
||||
const extracted = extractor(parsed);
|
||||
if (extracted) {
|
||||
lineReader.close();
|
||||
fileStream.close();
|
||||
return extracted;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Ignore malformed or missing artifacts so full scans keep progressing.
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user