🤖 fix: remote mux server gap fixes and cleanup

[ThomasK33]

Summary

Comprehensive gap fixes and cleanup for the remote mux server feature branch. Addresses 12 issues identified during deep review of the ~5600 LoC remote server implementation, spanning federation middleware fixes, dead code removal, safety guards, and integration tests.

Background

The remote server architecture uses two proxy mechanisms: (1) a federation middleware that intercepts routes with remote-encoded IDs, and (2) explicit per-handler proxy blocks. The review identified that the per-handler blocks are dead code (federation runs first), terminal sessions aren't federated, mixed local/remote ID batches break, and several edge cases need hardening.

Implementation

Dead code removal (−194 LoC)

• Removed all 16 resolveRemoteWorkspaceProxy call sites from router.ts — unreachable since the federation middleware intercepts first

Federation fixes

• Terminal sessions: Added sessionId to FEDERATION_ID_KEYS, output value re-encoding for ID fields, path-specific handling for terminal.listSessions, and skipping local-only routes (openWindow, closeWindow, openNative)
• Mixed ID routing: getSessionUsageBatch now splits mixed local/remote IDs across servers with proper result merging
• Fork response: workspace.fork top-level projectPath now mapped from remote to local via remoteProjectPathMap
• Experiment gating: Backend now respects REMOTE_MUX_SERVERS experiment flag (was only checked in UI)

Safety & robustness

• Server removal guard: Prevents removing a server with active project mappings (avoids orphaned workspace crashes)
• Stale workspace cleanup: pumpRemoteMetadata emits metadata: null for stale workspaces on reconnect
• Client timeout: 30s timeout for non-streaming remote oRPC calls (AbortSignal.timeout)
• Config mutex: editConfig now uses AsyncMutex to prevent concurrent read-modify-write races
• HTTP auth warning: Logs when auth tokens are configured for HTTP (non-HTTPS) servers

UI guards

• BranchSelector and GitStatusIndicator hidden for remote workspaces (no local git repo)

Tests

• Extracted shared buildOrpcContext helper to tests/ipc/setup.ts
• Added enableExperimentForTesting helper (fixes existing remote tests broken by experiment gating)
• New integration tests for sendMessage, interruptStream, terminal.listSessions, terminal.create, terminal.sendInput, terminal.close federation

Validation

• make static-check passes (typecheck, lint, fmt, docs links, shellcheck)
• All 4 remote workspace test files pass: 5 pass, 2 skip (PTY-dependent), 0 fail
• Existing tests unaffected

Risks

• Output value re-encoding is now generic: any response field whose key is in FEDERATION_ID_KEYS with a non-encoded string value gets re-encoded. This is correct behavior but could interact with future FEDERATION_ID_KEYS additions.
• Server removal guard checks project mappings (not individual workspaces) — conservative but safe. Users must remove all project mappings before removing a server.

---

📋 Implementation Plan

Remote Mux Server: Gap Fixes & Cleanup

Context

This branch (~5600 LoC, 42 files) adds remote mux server support: a desktop Electron app connects to a mux server instance, proxies workspace operations, and displays remote workspaces alongside local ones. The architecture is Frontend ↔ Local mux ↔ Remote mux server with two proxy mechanisms:

1. Federation middleware (src/node/orpc/federationMiddleware.ts) — catch-all .use() on the router that intercepts any route whose input contains a remote-encoded workspace/task ID, proxies it to the remote server, and rewrites output IDs back.
2. Per-handler resolveRemoteWorkspaceProxy — explicit early-return blocks in ~15 handlers that do the same thing (and are now dead code since federation runs first).

A deep review identified several gaps. This plan addresses them in priority order. The plan preserves the existing assert() for missing servers (crash-on-invariant-violation philosophy) but ensures the invariant is enforced upstream.

Estimated net LoC: +350 / −250 (product code), +400 (tests)

---

1. Remove dead per-handler proxy blocks (cleanup, ~−200 LoC)

The federation middleware already intercepts all routes with workspaceId/taskId in input and never calls next() — so the explicit resolveRemoteWorkspaceProxy blocks in handlers are unreachable dead code.

Delete the early-return proxy blocks from these 15 handlers in src/node/orpc/router.ts:

Handler	Lines to delete
agents.list	L906–914
agents.get	L1005–1013
agentSkills.list	L1028–1036
agentSkills.listDiagnostics	L1049–1057
agentSkills.get	L1070–1078
workspace.archive	L2534–2537
workspace.unarchive	L2545–2548
workspace.sendMessage	L2573–2580
workspace.answerAskUserQuestion	L2598–2605
workspace.resumeStream	L2623–2629
workspace.interruptStream	L2648–2654
workspace.getInfo	L2712–2721
workspace.getFullReplay	L2735–2744
workspace.getSubagentTranscript	L2755–2792
workspace.onChat	L2989–3021

Then clean up imports from router.ts:

• Remove resolveRemoteWorkspaceProxy import (only used in deleted blocks)
• Remove rewriteRemoteWorkspaceChatMessageIds import (still used in federationMiddleware.ts)
• Remove rewriteRemoteFrontendWorkspaceMetadataIds import (still used in federationMiddleware.ts)
• Remove rewriteRemoteTaskToolPartsInMessage import (still used in federationMiddleware.ts)

Keep the 4 multi-server aggregation handlers (workspace.list, workspace.onMetadata, workspace.activity.list, workspace.activity.subscribe) — these fan out to ALL servers and can't use federation.

Validation: All existing remote workspace tests must still pass. The federation middleware tests (src/node/orpc/federationMiddleware.test.ts) are the source of truth.

---

2. Fix terminal sessions for remote workspaces (~+120 LoC)

Terminal operations close, resize, sendInput, onOutput, attach, onExit use sessionId (not workspaceId), so the federation middleware doesn't intercept them. After terminal.create is proxied to the remote, subsequent operations fail on the local server.

Approach: Encode server ID into session IDs

Use the same remote.{b64server}.{b64id} encoding for session IDs. The codec in remoteMuxIds.ts is generic enough to reuse.

Step 1: Add "sessionId" to FEDERATION_ID_KEYS

In src/node/orpc/federationMiddleware.ts L26–37:

const FEDERATION_ID_KEYS = new Set<string>([
  "workspaceId",
  "workspaceIds",
  "parentWorkspaceId",
  "sourceWorkspaceId",
  "taskId",
  "taskIds",
  "task_id",
  "task_ids",
  "sectionId",
  "sessionId",       // ← add this
]);

Step 2: Encode sessionId in terminal.create response rewriting

terminal.create returns { sessionId: string }. The federation middleware's generic output rewriter needs to re-encode the sessionId from the remote response. Add a federation output rewrite for sessionId fields — either:

• Add "sessionId" to the set of keys that rewriteFederationOutputValue re-encodes (in the generic object walker at federationMiddleware.ts L289–365), OR
• Add a path-specific rewrite for ["terminal", "create"] that encodes the returned sessionId.

The generic approach is cleaner: in rewriteFederationOutputValue, when walking object keys, if a key is in FEDERATION_ID_KEYS and the value is a string, re-encode it as encodeRemoteWorkspaceId(serverId, value). This already partially happens for record keys (L340–352); extend it to also rewrite values under known ID keys.

Step 3: Handle terminal.listSessions output

terminal.listSessions (input: workspaceId) returns { sessions: Array<{ id: string, ... }> }. The federation middleware proxies it, but the returned session id values need encoding. Ensure the generic output rewriter handles nested id fields inside sessions — or add a path-specific rewrite.

Validation: Write a test in tests/ipc/ that:

1. Creates a remote workspace
2. Calls terminal.create with the encoded workspace ID
3. Verifies the returned sessionId is remote-encoded
4. Calls terminal.sendInput with that encoded session ID
5. Subscribes to terminal.onOutput with that encoded session ID
6. Calls terminal.close with that encoded session ID
7. All calls proxy correctly to the remote server

---

3. Gate backend on REMOTE_MUX_SERVERS experiment (~+10 LoC)

The UI correctly hides remote features when the experiment is disabled, but the backend unconditionally fetches from all configured remote servers.

File: src/node/orpc/remoteMuxProxying.ts — getRemoteServersForWorkspaceViews() at L155–158:

export function getRemoteServersForWorkspaceViews(context: ORPCContext) {
  // Add at top:
  if (!context.experimentsService.isExperimentEnabled(EXPERIMENT_IDS.REMOTE_MUX_SERVERS)) {
    return [];
  }
  const config = context.config.loadConfigOrDefault();
  // ... rest unchanged
}

File: src/node/orpc/federationMiddleware.ts — createFederationMiddleware() at L404+:

// Early in the middleware function, before decoding IDs:
if (!options.context.experimentsService.isExperimentEnabled(EXPERIMENT_IDS.REMOTE_MUX_SERVERS)) {
  return options.next();
}

Validation: With experiment disabled, workspace.list should return only local workspaces. Sending a remote-encoded workspace ID should fall through to the local handler (which will fail to find it — correct behavior).

---

4. Fix getSessionUsageBatch mixed ID routing (~+40 LoC)

Input is { workspaceIds: string[] }. If the array mixes local and remote IDs, the federation middleware sends everything to one remote server — local IDs get lost.

Fix: Add explicit handler-level splitting in src/node/orpc/router.ts at L3662–3667.

Replace the simple passthrough with:

handler: async ({ context, input }) => {
  // Partition IDs into local vs per-remote-server buckets
  const localIds: string[] = [];
  const remoteByServer = new Map<string, { server: RemoteMuxServerConfig; ids: string[]; encodedToRemote: Map<string, string> }>();

  for (const id of input.workspaceIds) {
    const decoded = decodeRemoteWorkspaceId(id);
    if (!decoded) {
      localIds.push(id);
      continue;
    }
    let bucket = remoteByServer.get(decoded.serverId);
    if (!bucket) {
      const server = context.remoteServersService.getServer(decoded.serverId);
      assert(server, `Remote server not found: ${decoded.serverId}`);
      bucket = { server, ids: [], encodedToRemote: new Map() };
      remoteByServer.set(decoded.serverId, bucket);
    }
    bucket.ids.push(decoded.remoteId);
    bucket.encodedToRemote.set(decoded.remoteId, id);
  }

  // Fetch local
  const localResult = localIds.length > 0
    ? await context.sessionUsageService.getSessionUsageBatch(localIds)
    : {};

  // Fetch from each remote server, re-key with encoded IDs
  const remoteResults: Record<string, SessionUsageFile | undefined> = {};
  await Promise.all([...remoteByServer.values()].map(async (bucket) => {
    const authToken = context.remoteServersService.getAuthToken(bucket.server.id);
    const client = createRemoteClient({ baseUrl: bucket.server.baseUrl, authToken });
    const result = await client.workspace.getSessionUsageBatch({ workspaceIds: bucket.ids });
    for (const [remoteId, usage] of Object.entries(result)) {
      const encodedId = bucket.encodedToRemote.get(remoteId) ?? encodeRemoteWorkspaceId(bucket.server.id, remoteId);
      remoteResults[encodedId] = usage;
    }
  }));

  return { ...localResult, ...remoteResults };
}

Also add this route to the federation middleware skip list — the federation middleware should not intercept getSessionUsageBatch since the handler handles splitting itself. Add to federationMiddleware.ts:

// Skip routes that handle their own remote splitting
if (isExactPath(options.path, ["workspace", "getSessionUsageBatch"])) {
  return options.next();
}

---

5. Prevent server removal with orphaned workspaces (~+20 LoC)

When a server is removed, the assert(server, ...) in resolveRemoteWorkspaceProxy (and the federation middleware) will crash on any operation targeting orphaned workspaces. Enforce the invariant upstream.

File: src/node/orpc/router.ts — the remoteServers.remove handler (find it near L600+):

Before calling context.remoteServersService.remove(), check for active workspaces:

// Check for workspaces referencing this server
const allWorkspaces = await context.workspaceService.list();
const orphaned = allWorkspaces.filter(ws => {
  const decoded = decodeRemoteWorkspaceId(ws.id);
  return decoded?.serverId === input.id;
});
if (orphaned.length > 0) {
  return {
    success: false,
    error: `Cannot remove server: ${orphaned.length} workspace(s) still reference it. Archive or delete them first.`,
  };
}

If the remove endpoint currently returns void, update the schema to return Result<void> so it can communicate failure.

---

6. Add timeout to remote oRPC client (~+15 LoC)

src/node/remote/remoteOrpcClient.ts has no request timeout. A hanging remote server blocks the request indefinitely.

Add a timeoutMs option (default 30s for normal calls, no timeout for streaming):

export interface CreateRemoteClientOptions {
  baseUrl: string;
  authToken?: string;
  timeoutMs?: number;  // ← add
}

export function createRemoteClient<TClient = unknown>({
  baseUrl, authToken, timeoutMs,
}: CreateRemoteClientOptions): TClient {
  // ... existing validation ...

  const link = new HTTPRPCLink({
    url: orpcUrl,
    headers,
    fetch: timeoutMs
      ? (input, init) => globalThis.fetch(input, {
          ...init,
          signal: init?.signal
            ? AbortSignal.any([init.signal, AbortSignal.timeout(timeoutMs)])
            : AbortSignal.timeout(timeoutMs),
        })
      : undefined,
  });
  return createORPCClient(link) as unknown as TClient;
}

Then at call sites in federationMiddleware.ts and router.ts, pass timeoutMs: 30_000 for non-streaming calls. For streaming subscriptions (onMetadata, onChat, activity.subscribe), omit it (they have their own stall detection).

---

7. Fix stale workspaces after remote disconnect/reconnect (~+15 LoC)

In src/node/orpc/router.ts, the pumpRemoteMetadata function (inside workspace.onMetadata handler, ~L3124–3207) reconnects on failure but doesn't clear stale workspace entries.

Add cleanup at reconnect point (~L3131, before creating new client):

// On reconnect, emit removal for previously-visible workspaces
// They'll be re-added as the new stream delivers fresh events
if (visibleWorkspaceIds.size > 0) {
  for (const staleId of visibleWorkspaceIds) {
    const encoded = encodeRemoteIdBestEffort(server.id, staleId);
    push({ workspaceId: encoded, metadata: null });
  }
  visibleWorkspaceIds.clear();
}

---

8. Add editConfig mutex (~+5 LoC)

src/node/config.ts L455–462: editConfig is read-modify-write without locking. Concurrent async calls can lose updates.

import { AsyncMutex } from "@/node/utils/concurrency/asyncMutex";

// Add to Config class:
private readonly editMutex = new AsyncMutex();

async editConfig(fn: (config: ProjectsConfig) => ProjectsConfig): Promise<void> {
  await using _lock = await this.editMutex.acquire();
  const config = this.loadConfigOrDefault();
  const newConfig = fn(config);
  await this.saveConfig(newConfig);
}

---

9. Disable UI controls for remote workspaces (~+30 LoC)

Several UI controls silently fail or reference non-existent local paths for remote workspaces.

9a. Terminal button — src/browser/components/WorkspaceHeader.tsx

After terminal sessions are fixed (item 2), the terminal button should work for remote workspaces. No UI guard needed if item 2 is done first. If item 2 is deferred, add a disabled state:

// Near L402, wrap terminal button:
const isRemote = remoteWorkspaceInfo != null;
// ... in JSX:
<Button disabled={isRemote} title={isRemote ? "Terminal not yet supported for remote workspaces" : "Open terminal"} ...>

9b. BranchSelector & GitStatusIndicator — src/browser/components/WorkspaceHeader.tsx

These run local git commands against namedWorkspacePath. For remote workspaces, the path is rewritten but may not have a local git repo. Guard them:

// L248 — wrap BranchSelector:
{!remoteWorkspaceInfo && (
  <BranchSelector workspaceId={workspaceId} workspaceName={workspaceName} />
)}

// L249-255 — wrap GitStatusIndicator:
{!remoteWorkspaceInfo && (
  <GitStatusIndicator gitStatus={gitStatus} workspaceId={workspaceId} ... />
)}

Same pattern in WorkspaceListItem.tsx L514–520 for the GitStatusIndicator.

9c. Explorer tab — src/browser/components/RightSidebar/ExplorerTab.tsx

The file browser runs bash scripts via API. For remote workspaces, these would proxy to the remote (which is correct behavior). No guard needed — the federation middleware will proxy executeBash to the remote server. The explorer should work for remote workspaces.

However, verify that workspacePath is correctly mapped for remote workspaces. If ExplorerTab receives a local path that doesn't exist, it'll show an empty tree. Check that the path mapping from rewriteRemoteFrontendWorkspaceMetadataForLocalProject provides a usable path.

---

10. Handle workspace.fork for remote workspaces (~+10 LoC)

workspace.fork input has sourceWorkspaceId (in FEDERATION_ID_KEYS) and returns { metadata, projectPath }. The federation middleware already:

• Decodes sourceWorkspaceId in input ✅
• Rewrites metadata via rewriteRemoteFrontendMetadataBestEffort (which includes project path mapping) ✅
• Re-encodes metadata.id ✅

The gap: The top-level projectPath in the response is a remote path and isn't rewritten.

Fix in src/node/orpc/federationMiddleware.ts — add a path-specific rewrite:

// After the existing isExactPath checks (~L452):
if (isExactPath(options.path, ["workspace", "fork"])) {
  // The top-level projectPath needs mapping from remote to local
  if (typeof remoteResult === "object" && remoteResult !== null && "projectPath" in remoteResult) {
    const result = remoteResult as Record<string, unknown>;
    const remotePath = result.projectPath as string;
    const localPath = mapRemoteProjectPathToLocal(serverId, remotePath, options.context);
    if (localPath) {
      result.projectPath = localPath;
    }
  }
}

Where mapRemoteProjectPathToLocal uses the server's projectMappings to find the corresponding local path (similar to what rewriteRemoteFrontendWorkspaceMetadataForLocalProject does for metadata).

---

11. Log warning for auth tokens over HTTP (~+5 LoC)

File: src/node/services/remoteServersService.ts — in upsert() (~L154–180):

if (params.authToken && normalizedConfig.baseUrl.startsWith("http://")) {
  log.warn(
    `Remote server "${normalizedConfig.label}" uses http:// — auth token will be sent in cleartext. Consider using https://.`
  );
}

---

12. Write integration tests for core proxy paths (~+400 LoC)

Using the pattern from tests/ipc/archiveRemoteWorkspace.test.ts (two real test environments, one acting as remote), add tests for:

File: tests/ipc/sendMessageRemoteWorkspace.test.ts

• Create remote workspace → encode ID → send message via local → verify federation proxies it
• Subscribe to onChat via local → verify events arrive with encoded IDs
• Call interruptStream via local → verify it reaches remote

File: tests/ipc/terminalRemoteWorkspace.test.ts (after item 2)

• Create remote workspace → terminal.create → verify encoded session ID returned
• terminal.sendInput with encoded session ID → verify proxied
• terminal.onOutput subscription → verify output arrives
• terminal.close → verify cleanup

File: tests/ipc/resumeStreamRemoteWorkspace.test.ts

• Create remote workspace → start streaming → interrupt → call resumeStream via local
• Verify idempotency: call resumeStream while remote is already streaming → verify no error, no duplicate

Extract shared helper: The buildOrpcContext() function is duplicated across agentSkillsRemoteWorkspace.test.ts, archiveRemoteWorkspace.test.ts, and others. Extract to tests/ipc/helpers.ts.

---

Implementation Order

Execute in this order to minimize conflicts and enable incremental validation:

1. Item 1 (remove dead proxy blocks) — pure deletion, simplifies router.ts
2. Item 3 (experiment gating) — small, independent
3. Item 8 (editConfig mutex) — small, independent
4. Item 5 (prevent orphaned workspaces) — small, independent
5. Item 7 (stale workspace cleanup) — small, independent
6. Item 11 (HTTP warning) — small, independent
7. Item 6 (client timeout) — small, touches remoteOrpcClient.ts
8. Item 4 (batch splitting) — medium, modifies router.ts
9. Item 10 (fork path rewriting) — medium, modifies federation middleware
10. Item 2 (terminal sessions) — large, touches codec + federation + tests
11. Item 9 (UI guards) — depends on item 2 decision
12. Item 12 (integration tests) — depends on items above

Items 1–7 can be done in a single commit/PR. Items 8–12 could be a second pass.

---

Generated with mux • Model: anthropic:claude-opus-4-6 • Thinking: xhigh • Cost: $330.15

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a17f98c6cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed both review comments:

1. P1 (experiment gate bypass): Removed the redundant backend experiment gates entirely. The UI-level gate is sufficient — without the experiment enabled, users can't configure remote servers in Settings, so no remote server configs exist. Without configs, getRemoteServersForWorkspaceViews() already returns [] and the federation middleware can't look up servers for remote IDs. This avoids the mismatch between PostHog backend state and localStorage UI overrides.

2. P2 (path normalization): Added stripTrailingSlashes(metadata.projectPath.trim()) normalization in all 4 remoteProjectPathMap lookups across workspace.activity.list and workspace.activity.subscribe handlers.

Also rebased onto latest origin/main and resolved 7 merge conflicts + 6 post-rebase type errors.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 34856d4934

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed both round 2 comments:

1. P1 (experiment gate): Re-added backend experiment gate with a layered approach using config.getFeatureFlagOverride():

   • Config override "off" → force-disable (admin kill-switch via ~/.mux/config.json)
   • Config override "on" → force-enable (for self-hosted without PostHog)
   • Config override "default" → defer to experimentsService.isExperimentEnabled() (PostHog)
     This addresses both the kill-switch concern and the PostHog-unavailable scenario.

2. P2 (stale activity allowlist): Added periodic allowlist refresh (every 30s) inside the pumpRemoteActivity for await loop. The refresh is additive-only (never removes IDs) and best-effort (failures don't disrupt the stream). New workspaces created during a subscription now appear in activity within 30s.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0fcfb71525

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Remove hard timeout from federated streaming requests

This middleware creates the remote client with timeoutMs: 30_000 for every proxied route, including long-lived subscriptions such as workspace.onChat, workspace.onMetadata, workspace.activity.subscribe, and terminal streams. Because AbortSignal.timeout() is wall-clock based, healthy remote streams are forcibly aborted at 30s, which can terminate active chat/terminal sessions and lose in-flight streamed output. Apply the timeout only to non-streaming procedures (or bypass it for async-iterable routes).

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve sectionId when rewriting remote workspace metadata

Encoding sectionId as a remote ID breaks section matching on the frontend, which compares workspace.sectionId to plain section IDs from project sections (sectionIds.has(workspace.sectionId)). After this rewrite, remote workspaces no longer match their actual section IDs and are treated as unsectioned, causing incorrect grouping/placement behavior in the sidebar. sectionId should remain a section identifier, not be rewritten as a remote workspace/task ID.

Useful? React with 👍 / 👎.