System Architecture (Canonical Topology)

System Architecture (Canonical Topology)

This skill is the single source of truth for joelclaw system wiring. Use it for:

• "why did this run / not run"
• "which worker handles this function"
• "what is listening on port X"
• "how does event Y flow"
• full-stack routing/​debug across CLI → Inngest → workers → gateway → telemetry

Ground-Truth Scope + Evidence Snapshot

This document is grounded in direct reads of:

• apps/​docs-api/​src/​index.ts
• packages/​restate/​Dockerfile
• packages/​restate/​src/​index.ts
• packages/​restate/​src/​workflows/​dag-orchestrator.ts
• packages/​agent-execution/​src/​microvm.ts
• packages/​system-bus/​src/​serve.ts
• packages/​system-bus/​src/​inngest/​functions/​index.host.ts
• packages/​system-bus/​src/​inngest/​functions/​index.cluster.ts
• packages/​system-bus/​src/​inngest/​client.ts
• infra/​worker-supervisor/​src/​main.rs
• ~/​Library/​LaunchAgents/​com.joel*.plist
• k8s/​* (all files)
• infra/​pds/​values.yaml
• packages/​gateway/​src/​daemon.ts
• packages/​gateway/​src/​channels/​*.ts
• ~/​.joelclaw/​gateway/​AGENTS.md
• ~/​.joelclaw/​gateway/​.pi/​settings.json
• ~/​.local/​caddy/​Caddyfile
• ~/​.colima/​default/​colima.yaml + colima status --json
• packages/​cli/​src/​cli.ts, packages/​cli/​src/​config.ts, packages/​cli/​src/​inngest.ts
• packages/​system-bus/​src/​observability/​* (key files: emit.ts, otel-event.ts, store.ts)
• packages/​telemetry/​src/​emitter.ts
• packages/​system-bus/​src/​lib/​langfuse.ts
• packages/​inference-router/​src/​tracing.ts
• ADRs in ~/​Vault/​docs/​decisions/ (required + topology-adjacent)
• last 50 lines of ~/​Vault/​system/​system-log.jsonl

Related docs verified

• docs/​architecture.md — Restate/​Firecracker runtime + workload execution flow
• docs/​deploy.md — Restate worker deploy + auth/​identity/​PVC procedures
• docs/​cli.md — workload command tree + runtime bridge
• docs/​observability.md — not inspected in this update

---

1) Physical Topology

Mac Mini "Panda" (host macOS)
├─ launchd services (gateway, worker supervisor, caddy, talon, agent-mail, etc.)
├─ Colima VM (driver: VZ, arch: aarch64, runtime: docker, VM IP: 192.168.64.2)
│  └─ Talos node: joelclaw-controlplane-1 (k8s v1.35.0, internal IP 10.5.0.2)
│     ├─ namespace: joelclaw
│     │  ├─ inngest (StatefulSet + NodePort 8288/​8289)
│     │  ├─ redis (StatefulSet + NodePort 6379)
│     │  ├─ typesense (StatefulSet + ClusterIP 8108)
│     │  ├─ restate (StatefulSet + NodePort 8080/​9070/​9071)
│     │  ├─ system-bus-worker (Deployment + ClusterIP 3111)
│     │  ├─ restate-worker (Deployment + ClusterIP 9080; full agent image + Firecracker)
│     │  ├─ dkron (StatefulSet + ClusterIP 8080)
│     │  ├─ docs-api (Deployment + NodePort 3838)
│     │  ├─ livekit-server (Deployment + NodePort 7880/​7881)
│     │  ├─ bluesky-pds (Deployment + NodePort 3000)
│     │  └─ minio (StatefulSet + NodePort 30900/​30901)
│     └─ namespace: aistor
│        ├─ aistor operator (Deployments: adminjob-operator, object-store-operator)
│        └─ aistor-s3 object store (StatefulSet + NodePort 31000/​31001)
├─ Caddy reverse proxy (tailnet HTTPS fan-in)
├─ Gateway daemon (embedded pi session)
├─ Firecracker substrate (requires Colima nestedVirtualization=true for /​dev/​kvm; OFF by default — unstable under load)
└─ NAS "three-body" (NFS tiers per ADR-0088)

Known runtime endpoints

• Colima VM IP: 192.168.64.2 (colima status --json)
• Kubernetes API (stable operator tunnel): https://127.0.0.1:16443
• Talos API (stable operator tunnel): 127.0.0.1:15000
• Tailnet hostnames seen in config:
  • panda.tail7af24.ts.net (Caddy routes)
  • pds.panda.tail7af24.ts.net (PDS values)

Tailscale mesh state

• tailscale status --json failed in this environment: UNKNOWN — needs manual verification

---

2) Process Inventory (Long-Running)

Host launchd inventory (snapshot)

Snapshot source: launchctl print gui/​$(id -u)/<label> and plist inspection.

Process supervision behavior: worker-supervisor

Source: infra/​worker-supervisor/​src/​main.rs

• Default config:
  • worker dir: ~/​Code/​joelhooks/​joelclaw/​packages/​system-bus
  • command: bun run src/​serve.ts
  • port: 3111
  • health endpoint: /​api/​inngest
  • sync endpoint: /​api/​inngest (PUT)
  • health interval: 30s
  • restart after 3 consecutive health failures
  • restart backoff: 1s → 30s max
• Pre-start kills stale process on port 3111.
• Runs host import preflight before spawn:
  • bun --eval "await import('./​src/​inngest/​functions/​index.host.ts');"
  • on failure, skips spawn and retries with exponential backoff
• Loads env from ~/​.config/​system-bus.env plus leased secrets.
• Forces WORKER_ROLE=host for the supervised host worker.
• Emits OTEL events via CLI on supervisor failures/​restarts:
  • worker.supervisor.preflight.failed
  • worker.supervisor.worker_exit
  • worker.supervisor.health_check.restart

Worker supervision split note

• Talon is running (com.joel.talon), but host worker is still launched via com.joel.system-bus-worker -> worker-supervisor.
• ADR + system-log indicate Talon can defer worker supervision during coexistence.

---

Kubernetes process inventory

Node

• joelclaw-controlplane-1 (Talos v1.12.4, k8s v1.35.0, internal IP 10.5.0.2)

Core services

Restate / Firecracker runtime note

• deployment/​restate-worker is the current durable execution worker. The image bundles Bun + Node + pi + codex, the full repo checkout, and 76 symlinked skills.
• Runtime auth/​identity come from secret/​pi-auth and configmap/​agent-identity, which recreate /​root/​.pi/​agent/​auth.json plus the joelclaw identity chain inside the pod.
• Firecracker is enabled in-pod via privileged access to /​dev/​kvm on Colima VZ. The /​dev/​kvm hostPath mount uses type "" (optional) so the pod starts without it when nestedVirtualization is off.
• Persistent microVM assets live on PVC firecracker-images, mounted at /​tmp/​firecracker-test for kernel, rootfs, and snapshot files.
• Retry caps (2026-03-17): dagWorker maxAttempts=5, dagOrchestrator maxAttempts=3. Prevents Restate journal poisoning from infinite retries after code changes or infrastructure failures.
• Colima stability: nestedVirtualization is OFF by default (crashes VM under Docker build load). Toggle ON only for Firecracker testing sessions, then toggle OFF. See k8s skill for recovery procedures.

Control-plane access

• kube API exposed locally at 127.0.0.1:16443 via com.joel.kube-operator-access (ssh -S none -o ControlPath=none -L 16443:10.5.0.2:6443)
• Talos API exposed locally at 127.0.0.1:15000 via the same daemon (ssh -S none -o ControlPath=none -L 15000:10.5.0.2:50000)
• NodePort/​runtime app ports still come from Colima/​Lima forwarding; the operator daemon exists specifically because the direct host-published 6443 path was not boring after the rebuild

---

3) Worker Architecture (Role Split + Registration)

Source files:

• packages/​system-bus/​src/​serve.ts
• packages/​system-bus/​src/​inngest/​functions/​index.host.ts
• packages/​system-bus/​src/​inngest/​functions/​index.cluster.ts
• packages/​system-bus/​src/​inngest/​client.ts

Role model

• WORKER_ROLE parsed as host (default) or cluster.
• Registered function set is role-dependent:
  • host uses hostFunctionDefinitions
  • cluster uses clusterFunctionDefinitions

Ground-truth counts

• Host function set: 101
• Cluster function set: 12
• Cluster subset functions:
  • approvalRequest, approvalResolve
  • todoistCommentAdded, todoistTaskCompleted, todoistTaskCreated
  • frontMessageReceived, frontMessageSent, frontAssigneeChanged
  • todoistMemoryReviewBridge
  • githubWorkflowRunCompleted, githubPackagePublished
  • webhookSubscriptionDispatchGithubWorkflowRunCompleted

App registration isolation

From inngest/​client.ts:

• app id resolves to:
  • system-bus-host when role is host
  • system-bus-cluster when role is cluster
• explicit INNGEST_APP_ID overrides role-derived id.

This prevents host and cluster workers from overwriting each other’s function graphs.

serveHost behavior

From serve.ts:

• host role default serveHost: http://host.docker.internal:3111
• cluster role default serveHost: unset (connect-mode default)
• INNGEST_SERVE_HOST overrides either role.

Kubernetes cluster worker manifest sets:

• INNGEST_BASE_URL=http://inngest-svc:8288
• INNGEST_SERVE_HOST=http://system-bus-worker:3111

Registration mechanics

• Worker exposes GET|POST|PUT /​api/​inngest.
• Worker sends a delayed self-sync PUT /​api/​inngest ~5s after startup.
• worker-supervisor also performs startup PUT sync.

Host is primary today

From index comments + function lists:

• ADR-0089 transition: host remains authoritative for broad function ownership.
• Cluster is intentionally limited to cluster-safe subset (12 functions).

---

4) Event Flow (CLI → Inngest → Worker → Completion)

Canonical flow: joelclaw send

1. CLI joelclaw send <event> calls Inngest.send().
2. Inngest.send() POSTs event JSON to:
   • ${INNGEST_URL}/​e/​${INNGEST_EVENT_KEY}
   • default: http://localhost:8288/e/<key>
3. Inngest server persists the event and resolves matching function triggers.
4. Inngest dispatches function steps to the worker app graph that owns that function ID:
   • host app (system-bus-host) for 101-host set
   • cluster app (system-bus-cluster) for 12-cluster subset
5. Worker handles callbacks via /​api/​inngest (Hono + inngest/​hono handler).
6. Each step.run result is memoized by Inngest; next step executes when prior completes.
7. Completion/​failure is queryable via GraphQL (/​v0/​gql) and CLI commands (runs, run, event, events).

Queue flow: joelclaw queue emit → Restate drainer → durable dispatch

1. CLI joelclaw queue emit <event> persists a QueueEventEnvelope into Redis stream joelclaw:queue:events and indexes it in sorted set joelclaw:queue:priority.
2. The restate-worker k8s deployment (packages/​restate/​src/​index.ts) starts a deterministic queue drainer beside the channel callback listener.
3. On startup, the drainer claims pending + never-delivered entries via @joelclaw/​queue#getUnacked(), reindexes replayable entries, and emits OTEL replay evidence.
4. Each drain tick selects the next priority candidate from the sorted set, resolves its static registry target from packages/​queue/​src/​registry.ts, and POSTs a one-node DAG request to Restate /​dagOrchestrator/​{workflowId}/​run/​send.
5. When backlog remains and a dispatch slot frees, the drainer self-pulses immediately instead of waiting for the next QUEUE_DRAIN_INTERVAL_MS heartbeat. That interval is now the idle poll / retry cadence, not a mandatory 2-second tax between successful sends.
6. The current Story-3 bridge re-emits the queue item to its registered Inngest event target inside that one-node DAG request. This is deliberate: the deterministic queue/​drainer is proven first; per-family Restate cutovers remain Story 4 work.
7. On accepted Restate dispatch, the drainer acks the queue message; on failure it leaves the message in Redis, applies retry cooldown, and emits queue.dispatch.failed OTEL evidence.
8. If backlog remains in Redis but the drainer stops making progress past QUEUE_DRAIN_STALL_AFTER_MS, it emits queue.drainer.stalled and exits non-zero so k8s restarts deployment/​restate-worker. That is the self-heal path for a wedged drainer inside an otherwise-running Bun process.
9. Crash recovery comes from the Redis stream + consumer-group replay path, not from vibes: restart the restate-worker pod, let getUnacked() reclaim the inflight entries, then drain resumes.

Workload flow: joelclaw workload run → Redis → Restate DAG → execution

1. joelclaw workload plan ... --stages-from <file> can load an explicit stage DAG, validate unknown deps/​self-deps/​duplicates/​cycles, and preserve per-stage acceptance gates.
2. joelclaw workload run <plan-artifact> normalizes the selected stage into the canonical workload/​requested runtime request.
3. Queue admission writes the request into Redis, where the deterministic drainer forwards it into Restate as a dagOrchestrator/​{workflowId}/​run/​send request.
4. dagOrchestrator executes dependency waves: ready nodes in parallel, chained nodes only after every dependsOn node has terminal output.
5. dagWorker executes the node handler:
   • shell → subprocess work inside the restate-worker pod
   • infer → pi -p --no-session --no-extensions inside the pod, using the mounted auth + identity + skill set
   • microvm → Firecracker boot/​restore through /​dev/​kvm with kernel/​rootfs/​snapshot files on PVC firecracker-images
6. Each node emits OTEL (dag.node.*), and the workflow emits dag.workflow.* so queue → Restate → execution remains observable.
7. Current truthful limit: the microVM runtime boots and restores snapshots in-cluster, but the broader exec-in-VM workspace drive protocol is still incomplete for general coding slices.

Webhook flow

1. External service posts to /​webhooks/:provider.
2. Caddy routes /​webhooks/​* on localhost:8443 to worker localhost:3111.
3. webhookApp verifies signature, normalizes payload, emits Inngest events (provider/​event).
4. Inngest executes subscribed functions.

"Why did this run / not run" trace recipe

1. joelclaw send <event> -d '<payload>'
2. joelclaw events --prefix <event-prefix> --hours 1
3. joelclaw event <event-id> (fan-out to function runs)
4. joelclaw run <run-id> (step trace + errors)
5. joelclaw runs --count 20 --hours 1
6. joelclaw otel search "<component/​action>" --hours 1
7. Validate function ownership in index.host.ts / index.cluster.ts.

---

5) Port Map (Canonical)

Exposure sources: k8s service manifests, Caddyfile, kubectl get svc, lsof listeners.

Notes

• Host NodePort exposure appears through an ssh listener process (Colima portForwarder=ssh).
• Exact per-port ssh forward command line is UNKNOWN — needs manual verification (process introspection restricted in this environment).

---

6) Storage Topology

Redis

• Runtime: k8s StatefulSet (redis:7-alpine, appendonly enabled).
• Primary uses:
  • gateway queue/​session keys (joelclaw:events:*, joelclaw:notify:*, joelclaw:gateway:sessions)
  • webhook subscriptions (joelclaw:webhook:*)
  • gateway health mute/​streak keys (gateway:health:*)

Typesense

From observability code:

• otel_events collection (canonical telemetry event store)
• memory_observations collection (vector-aware memory index; schema validated at startup)
• docs-api also points at http://typesense:8108 for docs search/​index surfaces.

Firecracker runtime storage

• PVC: firecracker-images
• Mounted in deployment/​restate-worker at /​tmp/​firecracker-test
• Stores:
  • kernel (vmlinux)
  • rootfs (agent-rootfs.ext4)
  • snapshots (snapshots/​vm.snap, snapshots/​vm.mem)
• Firecracker snapshot restore is currently operator-proven at ~9ms on the Colima VZ nested-virt path.

Inngest state

• StatefulSet PVC mounted at /​data
• INNGEST_SQLITE_DIR=/​data

docs-api surface

• Deployment: docs-api on NodePort 3838
• Route count: 11 endpoints including /​health
• Key routes:
  • GET /​search — hybrid chunk search with concept, concepts, doc_id, expand, and assemble
  • GET /​docs/​search
  • GET /​docs
  • GET /​docs/:id
  • GET /​docs/:id/​toc
  • GET /​docs/:id/​chunks
  • GET /​chunks/:id
  • GET /​concepts
  • GET /​concepts/:id
  • GET /​concepts/:id/​docs
• Taxonomy surface: 21-concept SKOS graph (10 parents + 11 sub-concepts) with broader, narrower, and related edges.

NAS (ADR-0088 + ADR-0187)

Tiering policy:

• Tier 1 local SSD (hot runtime state)
• Tier 2 NAS NVMe (/​Volumes/​nas-nvme ↔ /​volume2/​data)
• Tier 3 NAS HDD (/​Volumes/​three-body)

Access paths

k8s ↔ NAS networking

k8s pods reach the NAS via a LAN route through the Colima col0 bridge: Talos → Docker NAT → VM col0 → macOS (ip.forwarding=1) → LAN → NAS

The VZ NAT on eth0 does NOT forward LAN traffic. Route persisted in Colima provision + colima-tunnel script: ip route replace 192.168.1.0/​24 via 192.168.64.1 dev col0

Always use IP 192.168.1.163, never hostname three-body — DNS doesn't resolve from k8s.

Degradation contract (ADR-0187):

• writes must fallback local -> remote -> queued
• queue spool default: /​tmp/​joelclaw/​nas-queue

Vault

• Obsidian vault at /​Users/​joel/​Vault
• system log file: /​Users/​joel/​Vault/​system/​system-log.jsonl

---

7) Networking Topology

Caddy reverse proxy routes (from ~/​.local/​caddy/​Caddyfile)

• https://panda.tail7af24.ts.net:9443 -> localhost:8288 (Inngest)
• https://panda.tail7af24.ts.net:8290 -> localhost:8289 (Inngest connect)
• https://panda.tail7af24.ts.net:3443 -> localhost:3111 (worker)
• https://panda.tail7af24.ts.net:5443 -> localhost:3838 (docs-api)
• https://panda.tail7af24.ts.net:7443 -> localhost:7880 (LiveKit)
• https://panda.tail7af24.ts.net:6443 -> localhost:6333 (Qdrant)
• http://localhost:8443 path router:
  • /​webhooks/​* -> localhost:3111
  • fallback -> localhost:8288

Tailscale + Funnel

• Config comments and ADR-0051 describe Funnel path :443 -> localhost:8443.
• Runtime tailscale status unavailable here: UNKNOWN — needs manual verification.

External webhook ingress

Expected path:

1. Internet provider -> Tailscale Funnel :443
2. Funnel -> local :8443
3. Caddy path route /​webhooks/​* -> worker :3111
4. worker /​webhooks/:provider verifies + emits Inngest event

---

8) CLI Wiring (Command Tree → Endpoint Surface)

Primary command tree root: packages/​cli/​src/​cli.ts.

Endpoint map by command family

Config source:

• ~/​.config/​system-bus.env (plus env overrides)
• defaults:
  • INNGEST_URL=http://localhost:8288
  • INNGEST_WORKER_URL=http://localhost:3111

---

9) Observability + Tracing Topology

OTEL event pipeline

• Worker emits via emitOtelEvent() / emitMeasuredOtelEvent().
• Gateway emits via @joelclaw/​telemetry (emitGatewayOtel) to:
  • default OTEL_EMIT_URL=http://localhost:3111/observability/emit
• Worker endpoint /​observability/​emit validates token (x-otel-emit-token) if configured.
• Store path (storeOtelEvent):
  1. Typesense otel_events (primary)
  2. optional Convex mirror for high-severity recent window
  3. optional Sentry forward for warn/​error/​fatal

Langfuse integration points

• Gateway boot: packages/​gateway/​src/​daemon.ts calls initTracing({}) from inference-router.
• Inference router traces model-route decisions:
  • packages/​inference-router/​src/​tracing.ts
  • used from packages/​inference-router/​src/​router.ts
• System-bus LLM traces:
  • packages/​system-bus/​src/​lib/​langfuse.ts (traceLlmGeneration)
  • called by packages/​system-bus/​src/​lib/​inference.ts and channel-message-classify.ts

---

10) Key ADR Topology Decisions

---

10.1) Sandbox Execution Contract (@joelclaw/​agent-execution)

Package: packages/​agent-execution/
Purpose: Canonical contract for sandboxed story execution shared between Restate workflows, system-bus Inngest functions, and k8s Job launcher.

Contract Types

Request: SandboxExecutionRequest

• workflowId, requestId, storyId: identifiers
• task: story prompt/​task to execute
• agent: { name, variant?, model?, program? }
• sandbox: "workspace-write" | "danger-full-access"
• baseSha: git SHA before execution
• cwd?: working directory
• timeoutSeconds?: timeout
• verificationCommands?: post-execution verification
• sessionId?: tracking identifier

Result: SandboxExecutionResult

• requestId: correlation ID
• state: "pending" | "running" | "completed" | "failed" | "cancelled"
• startedAt, completedAt?, durationMs?: timing
• artifacts?: execution artifacts (see below)
• error?: error message (failed state)
• output?: stdout/​stderr output

Artifacts: ExecutionArtifacts

• headSha: git SHA after execution
• touchedFiles: list of modified/​untracked files from git status --porcelain
• patch?: git patch content (format-patch or diff)
• verification?: { commands, success, output }
• logs?: { executionLog?, verificationLog? }

Repo Materialization (Story 3)

Function: materializeRepo(targetPath, baseSha, options)

Behavior:

• Clone repo if target path doesn't exist (requires remoteUrl)
• Fetch + checkout if target path exists
• SHA verification after checkout
• Automatic unshallow if SHA not in shallow clone
• Isolated sandbox-local workspace (host worktree untouched)

Returns: { path, sha, freshClone, durationMs }

Key options:

• remoteUrl?: remote URL for fresh clone
• branch?: branch/​ref to fetch (default: "main")
• depth?: shallow clone depth (default: 1)
• includeSubmodules?: include submodules
• timeoutSeconds?: timeout (default: 300)

Artifact Export (Story 3)

Function: generatePatchArtifact(options)

Behavior:

• Captures touched-file inventory via getTouchedFiles()
• Generates git patch from baseSha..headSha:
  • Uses git format-patch if commits exist in range
  • Uses git diff if only uncommitted changes
• Optionally includes untracked files as patch content
• Embeds verification summary and log references
• Serializable to JSON via writeArtifactBundle()

Key options:

• repoPath: path to git repo
• baseSha: base SHA (start of diff range)
• headSha?: head SHA (default: HEAD)
• includeUntracked?: include untracked files (default: true)
• verificationCommands?, verificationSuccess?, verificationOutput?: verification data
• executionLogPath?, verificationLogPath?: log references
• timeoutSeconds?: timeout (default: 60)

Returns: ExecutionArtifacts

Promotion Boundary (Phase 1)

Authoritative output is patch bundle + metadata.

Sandbox runs do not merge to main or push to remote. The runtime:

1. Materializes repo at baseSha in sandbox-local workspace
2. Executes agent task
3. Runs verification commands
4. Exports patch artifact with touched files and verification results
5. Emits SandboxExecutionResult event with ExecutionArtifacts

Promotion is a separate operator decision:

• Restate workflow receives ExecutionArtifacts
• Operator reviews patch + verification summary
• Operator applies patch to host repo (or discards)
• Operator commits and pushes (if approved)

This keeps sandbox runs isolated and reversible.

k8s Job Integration

Job spec generation: generateJobSpec(request, options)

Cold k8s Jobs for isolated story execution:

• Deterministic Job naming keyed by requestId
• Runtime image contract: Git, Bun, agent tooling, /​workspace directory
• Environment-driven config: WORKFLOW_ID, REQUEST_ID, STORY_ID, TASK_PROMPT_B64, BASE_SHA, etc.
• Resource limits: 500m-2 CPU, 1-4Gi memory (configurable)
• TTL cleanup: auto-delete after 5 minutes (default)
• Active deadline: 1 hour max runtime (default)
• No automatic retries (backoffLimit: 0)
• Security: non-root (UID 1000), no privilege escalation, capabilities dropped

Runtime contract:

1. Decode TASK_PROMPT_B64 from env
2. Call materializeRepo() at BASE_SHA
3. Execute agent with task
4. Run verification commands (if VERIFICATION_COMMANDS_B64 set)
5. Call generatePatchArtifact() with results
6. Emit SandboxExecutionResult event with ExecutionArtifacts
7. Exit 0 (success) or non-zero (failure)

Cancellation: Delete Job resource (SIGTERM to container)

Job deletion: generateJobDeletion(requestId) -> { name, namespace, propagationPolicy }

See k8s/​agent-runner.yaml for full runtime contract specification.

Topology Impact

• Story 2: Added contract types and Job spec generation
• Story 3: Added repo materialization and artifact export helpers
• ADR-0221 phase 1: added explicit local sandbox isolation primitives — deterministic sandbox identity, deterministic local sandbox paths, per-sandbox env materialization, minimal/​full mode vocabulary, and a JSON registry helper for host-worker sandboxes
• ADR-0221 phase 2: wired those local helpers into the real host-worker system/​agent-dispatch local backend so sandbox runs now allocate deterministic paths under ~/​.joelclaw/​sandboxes/, materialize .sandbox.env, persist registry state, and carry localSandbox metadata in inbox snapshots
• ADR-0221 phase 3/​4/​5/​6: phase 3 added terminal retention/​cleanup policy (cleanupAfter + registry metadata), opportunistic pruning of expired local sandboxes on new-run startup, copy-first .devcontainer materialization helpers with exclusion rules for env/​secret junk, live sandbox env injection so the agent process actually sees the reserved runtime identity, a hash-preserving sandbox identity fix after live dogfood exposed path collisions from long shared requestId prefixes, abbreviated-baseSha acceptance during repo materialization, truthful failed inbox snapshots when dispatch crashes before normal terminal writeback, and a repeatable operator probe at bun scripts/​verify-local-sandbox-dispatch.ts; phase 4 adds sandboxMode=minimal|full through the workload front door, requested-cwd mapping inside the cloned checkout, compose-backed full local mode startup, the reality that stale Restate workers can reject workload/​requested until restarted and reloaded, a recursion guard because sandboxed stage runs were able to call scripts/​verify-workload-full-mode.ts / joelclaw workload run from inside the sandbox and spawn nested canaries instead of terminating honestly, and a guarded workflow-rig proof run (WR_20260310_013158) that completes terminally with healthy compose startup plus clean teardown; phase 5 adds the operator-facing CLI surface joelclaw workload sandboxes list|cleanup|janitor so retained sandboxes can be inspected and janitored on demand instead of only during startup opportunistic pruning, and the operator surfaces now reconcile registry entries against per-sandbox metadata before reporting or deleting so old partial writeback residue stops lying about terminal state; phase 6 makes janitoring scheduled instead of purely manual via repo-managed launchd service com.joel.local-sandbox-janitor, which runs scripts/​local-sandbox-janitor.sh → joelclaw workload sandboxes janitor at load and every 30 minutes
• Future: Runtime image build, hot-image CronJob, warm-pool scheduler, Restate integration

Current state: the host-worker local sandbox path is now using the local-isolation helpers in production code, the package has a concurrent proof that two local sandboxes keep distinct compose identity plus copied devcontainer state, guarded full-mode workflow-rig dogfood closes terminally, and cleanup now has both on-demand CLI surfaces and scheduled launchd janitoring. Follow-on work is now about deeper runtime ergonomics and debugging any remaining non-terminal stale residues, not missing basic cleanup automation.

---

11) Verification Commands (Health + Wiring)

Core topology

# Colima + VM IP
colima status --json

# Kubernetes control plane + node
kubectl cluster-info
kubectl get nodes -o wide

# Core workloads
kubectl get pods -n joelclaw -o wide
kubectl get svc -n joelclaw -o wide

Host supervision

# Worker supervisor launchd state
launchctl print gui/​$(id -u)/​com.joel.system-bus-worker | rg "state =|pid =|last exit code"

# Gateway / Caddy / Talon
launchctl print gui/​$(id -u)/​com.joel.gateway | rg "state =|pid ="
launchctl print gui/​$(id -u)/​com.joel.caddy | rg "state =|pid ="
launchctl print gui/​$(id -u)/​com.joel.talon | rg "state =|pid ="

# Talon health
curl -s http://127.0.0.1:9999/health

Worker role split

# Parse role counts directly from source lists
python - <<'PY'
import re
from pathlib import Path
for f,name in [('packages/​system-bus/​src/​inngest/​functions/​index.host.ts','host'),('packages/​system-bus/​src/​inngest/​functions/​index.cluster.ts','cluster')]:
    txt=Path(f).read_text()
    body=re.search(rf'export const {name}FunctionDefinitions = \[(.*?)\];', txt, re.S).group(1)
    count=sum(1 for line in body.splitlines() if line.strip() and not line.strip().startswith('//'))
    print(name, count)
PY

# Inngest app ID derivation logic
rg -n "INNGEST_APP_ID|system-bus-host|system-bus-cluster|WORKER_ROLE" packages/​system-bus/​src/​inngest/​client.ts

Event flow trace

# Send event
joelclaw send <event> -d '<json>'

# Trace event and resulting runs
joelclaw events --prefix <event-prefix> --hours 1 --count 20
joelclaw event <event-id>
joelclaw runs --hours 1 --count 20
joelclaw run <run-id>

# Telemetry correlation
joelclaw otel search "<component_or_action>" --hours 1

Networking

# Caddy route config
caddy validate --config ~/​.local/​caddy/​Caddyfile

# Listening ports snapshot
/​usr/​sbin/​lsof -iTCP -sTCP:LISTEN -n -P

# Tailscale runtime (if daemon available)
tailscale status --json

---

12) Known Unknowns (Do Not Guess)

• Tailscale daemon state is not readable in this environment.
  • tailscale status --json -> failed to connect.
  • UNKNOWN — needs manual verification
• docs/​architecture.md, docs/​deploy.md, docs/​observability.md are absent in-repo.
  • UNKNOWN — needs manual verification
• Exact command-line ownership of all Colima ssh forwarding ports (64784, 64785, 9627, etc.)
  • UNKNOWN — needs manual verification
• Ingress controller runtime status for k8s/​docs-api-ingress.yaml
  • UNKNOWN — needs manual verification

---

13) Mandatory Update Policy (Non-Optional)

Update this skill in the same change whenever any of these change:

1. Worker runtime wiring
   • serve.ts, client.ts, index.host.ts, index.cluster.ts
   • WORKER_ROLE, app IDs, serveHost behavior, registration path
2. Supervision/​process topology
   • any ~/​Library/​LaunchAgents/​com.joel*.plist
   • infra/​worker-supervisor/​*, Talon behavior, gateway launch script/​label
3. Kubernetes topology
   • any file under k8s/
   • Helm values affecting core services (livekit, pds, etc.)
   • Service type/​port changes (NodePort/​ClusterIP)
4. Networking/​ingress
   • Caddyfile route/​port changes
   • Tailscale/​Funnel hostnames or ingress path changes
   • Colima/​VM networking model changes
5. Storage topology
   • Redis keyspace contracts for gateway/​webhook routing
   • Typesense telemetry collection/​schema changes
   • NAS mount/​fallback/​queue contract changes
6. Observability/​tracing
   • OTEL emit endpoint/​token behavior
   • telemetry storage path changes (Typesense/​Convex/​Sentry)
   • Langfuse integration points
7. CLI control-plane routing
   • command families moved to different endpoints/​services
8. ADR status changes affecting topology
   • especially ADR-0048, 0088, 0089, 0144, 0155, 0156, 0159, 0182, 0187

If any item above changed and this skill was not updated, this skill is stale and non-canonical.