Compute and managed environments
Two related capabilities own the runtime substrate: compute-plane
materializes a profile into an isolated, ephemeral pod and governs its lifecycle,
egress, budget, and surfaces; managed-environments is the user-facing
desired-state object (an immutable, sealed EnvironmentRevision sequence) that
pods and hosts are launched from. Neither redefines memory, ranking, or
metering — those remain the memory, retrieval, and economics capabilities.
Tenancy is shared: every table here is scoped on installation_id under the
platform RLS contract (tenant_installation_matches(), ENABLE + FORCE
on every table, GUC-keyed, zero rows when unbound).
Status at a glance
| Concern | State | Evidence |
|---|---|---|
| Pod lifecycle seam | Ships — SandboxRuntimeAdapter port + compute_pod table (0128/0224) | packages/compute-plane/src/types.ts, packages/compute-plane/src/sandbox-runtime.ts |
| Egress + budget | Ships — EgressGuard, vCPU-second ceiling (default 7200s) | packages/compute-plane/src/egress-guard.ts, budget.ts |
| Runtime implementation | Evolving — E2B + local-kvm seam live; Firecracker-direct is Wave 6 | packages/compute-plane/src/sandbox-runtime.ts (RuntimeMode) |
| Managed environments | Evolving — data model + read/create ship, no UI yet (W1) | apps/web/src/lib/cloud/managed-environments/data-access.ts |
| Billing / duration | Not metered — compute time is a standing gap, vCPU budget only | spec C-3; add-compute-plane-economics |
Architecture at a glance
flowchart TD
A[Desired state: EnvironmentRevision] --> B[compute-plane worker]
B --> C[createSandboxRuntime]
C -->|e2b / firecracker / local-kvm| D[microVM / sandbox]
D --> E[lens builders index mounted repos]
D --> F[agent fleet + MCP / LSP servers]
D --> G[exec / ACP / A2A surfaces]
G --> H[EgressGuard: DNS-pin + allowlist]
F --> I[outbox emit on commit]
G --> J[credential-bound surface creds]
K[reaper] -->|lease / vCPU budget / orphan| D
L[revocation watcher] -->|bounded interval| DThe control plane is provider-neutral: SandboxRuntimeAdapter
(provisionPod / terminatePod / pausePod / resumePod / getHandle /
listLiveSandboxIds) is injected per environment via createSandboxRuntime() with
RuntimeMode = 'e2b' | 'firecracker' | 'local-kvm' and the ZEPHR_COMPUTE_RUNTIME
env var. The shipped runtime walks E2B and a local-kvm fallback;
Firecracker-direct is Wave 6 (ZEP-2345) and is not live on main. The
spec's Firecracker-microVM-plus-devcontainer framing is the architectural target,
not the current default runtime.
Pod lifecycle
A pod's durable state lives in the compute_pod row (migration 0128_compute_pod.sql
plus the quota columns in 0224_compute_pod_quota_columns.sql). Its status column
CHECK is the exact PodStatus surface:
export type PodStatus =
'pending' | 'provisioning' | 'running' | 'paused' | 'terminating' | 'terminated' | 'orphaned';Legal edges: pending → provisioning → running; running ⇄ paused; any live state
→ terminating → terminated; any live state → orphaned → terminated. orphaned
is the reaper's marker when the substrate and the control plane disagree.
- Auto-suspend. A running pod with no activity for 15 minutes is snapshotted and
transitioned to
paused; the user can resume within the retention window. - Auto-terminate. The reaper enforces the vCPU-time ceiling and
lease_expires_at(see Budget); crossing either terminates the pod and records the terminal state. - Orphan reap (negative). The reaper reconciles the provider's live sandbox list
against
compute_pod— notlast_heartbeat_at, which is guest-supplied and untrustworthy for a VM that is still alive. A pod stuck interminatingpast a bounded interval is reaped even while heartbeating, so a crash mid-terminate cannot leak a live VM that burns vCPU, holds resolved secrets, and keeps egress open. - Idempotent terminate. Re-invoking
terminateon aterminating/terminatedpod is a no-op success, never a double-destroy and never an error. - Live handle.
SandboxHandle.statusis the runtime-level view ('running' | 'paused' | 'terminated'); the fourRuntimeErrorcodes (pod_not_found,already_terminated,invalid_state) are refusal shapes, not crash surfaces.
Configuration
| Knob | Source | Default | Notes |
|---|---|---|---|
| Runtime mode | env ZEPHR_COMPUTE_RUNTIME | inject | local-kvm fallback | local/local-kvm/firecracker/e2b; local always wins as the safe default |
| vCPU budget | compute_pod.vcpu_seconds_budget (0224) | 7200 (2h) | DEFAULT_VCPU_SECONDS_BUDGET = 2*60*60; NON-MONETARY |
| Lease expiry | compute_pod.lease_expires_at | unset | optional wall-clock hard stop |
| Egress pin TTL | DEFAULT_PIN_TTL_MS | 300000 (5 min) | control-plane DNS pin freshness |
| Network allowlist | NetworkScopeEntry[] on scope binding | tenant scope | { host, port? }; narrows the platform maximum, never widens |
| Pod template | profile / worktreeId / templateId | — | materializes the profile's MCP servers, rules, hooks, model prefs |
Scope, tenancy, and consent
Every operation keyed by a caller-supplied podId first resolves the pod within the
caller's installation_id; a pod belonging to another installation returns the same
tenant-scoped not-found as a missing pod (no existence oracle).
Operations that mutate a pod or attach a surface — launch | pause | resume | terminate | exec | mcp | acp — additionally require a capability grant and pass a
consent gate. mcp and acp are gated at the same strength as exec: attaching the
pod's tool surface or driving its agent over ACP confers at least exec authority, so a
gate that covers exec but omits mcp/acp would invert the threat model. A refused
operation appends a trust evidence row at review_state = 'unreviewed' that an agent
cannot self-confirm.
Egress (default-deny, platform-maximum)
Egress is default-deny at the microVM network layer, bounded by a
platform-maximum allowlist that a tenant scope may only narrow, never widen.
Enforcement is on the resolved destination address, not a presented name: DNS is
resolved by the control plane, the answer is pinned for the connection, and the guest
connects only to pinned addresses. EgressGuard returns either
{ allow: true, pinned } or { allow: false, reason, audit } with these block reasons:
egress_refused_unpinned_address; // destination never pinned by control-plane DNS
egress_refused_dns_rebinding; // resolved address != pinned address for the name
egress_refused_sni_not_allowlisted;
egress_refused_ip_literal; // no DNS/SNI (e.g. IP-literal connect)
egress_refused_stale_pin;
BLOCK_REASON_GUEST_TO_HOST_ESCAPE; // metadata service / host / peer-tenant / link-localName-only matching is explicitly insufficient: an IP-literal connect performs no DNS
lookup and sends no SNI; ECH/ESNI encrypts the SNI; and a low-TTL rebind can return an
attacker address for a legitimately allowlisted name. Link-local and RFC1918
destinations are denied outright — including the canonical metadata address
(169.254.169.254), reached with no DNS, no SNI, no TLS. Every refused attempt is
auditable: timestamp, source process, destination, and matching reason.
Compute budget and miner detection
Budget bounds vCPU-seconds only — it is not a bound on token-denominated work. An
agent generating output tokens is waiting on a remote model API and burns near-zero
CPU, so it can run far past any token ceiling without approaching this stop. Pod
compute time is not metered: neither this change nor add-compute-plane-economics
charges for vCPU-time or wall-clock residency (add-compute-plane-economics adds only
the token kinds run_llm_agent_task and run_agent_lens_index). The vCPU budget is
the only cost control and a standing gap, not a deferral.
The accrued ceiling is enforced host-side and is the primary miner control: the
maximum overrun a pod can achieve is the sum of the budget-flush interval and the
reaper-evaluation interval. Syscall-signature monitoring is best-effort and
guest-defeatable, explicitly not a fail-closed gate: Firecracker's seccomp filters
restrict the host VMM system calls, while the guest runs its own kernel behind KVM, so
guest execve is not observable to a host filter. An in-guest agent may exist but is
labelled best-effort and guest-defeatable — a hostile guest with root disables it in one
syscall, so it is not counted as defence-in-depth. The false-positive ceiling is
documented: cargo build, tsc --build, test suites, dev servers, and LLM inference are
the intended workloads and must not trip the heuristic.
Secrets and residency
- Runtime-only injection.
secret://references resolve only at runtime on the target installation; a saved/sharedPodTemplatecarries references, never resolved values. A template containing a resolved secret is rejected/stripped at save. - Memory snapshots are secret-bearing. A Firecracker full snapshot writes the
mem_file_pathas a complete copy of guest memory, so a claim that resolved secrets never reach the persisted snapshot is true for disk/template artifacts and false for memory snapshots. The memory file is encrypted at rest with a tenant-scoped key, retention-bounded, and deleted on terminate; on resume everysecret://reference is re-resolved rather than trusting the value captured in the snapshot. - Exec is the blast radius. A principal permitted to
execcan read everysecret://value present in the guest environment by construction. Capability + consent gating (not sandboxing) is the control;execruns unprivileged (uid1000, never0), bounded byexec_timeout_secondsandexec_max_output_bytes, and every exec is audited — including the allowed path — with principal,podId, command string, exit code, and duration. - Residency. A repo with a declared residency (
lens_repo.residency∈('global','eu','us')) must not be materialized into a non-matching region. The placement check runs onlaunchand independently onresume(which does not pass throughprovisionPod, so it re-checks against the restore region). Violations of a mislabelled repo mount the non-compliant repo, mark the podterminating, and write a residency-incident audit row — both forward-looking remedies, because the crossing has already happened.
Managed environments (desired-state model)
A Managed Environment is the user-facing desired-state object; an
EnvironmentRevision is an immutable, sealed desired-state row. Switching, rollback,
and sync operate on revisions, never on a mutable live blob.
export interface ManagedEnvironmentRow {
readonly id: string;
readonly installationId: string;
readonly name: string;
readonly activeRevisionId: string | null;
readonly createdAt: string;
readonly updatedAt: string;
}
export interface ManagedEnvironmentRevisionRow {
readonly id: string;
readonly environmentId: string;
readonly revision: number; // 1,2,3... linear, no gaps
readonly snapshot: Record<string, unknown>; // sealed; secrets are secret:// refs only
readonly sealedAt: string;
}managed_environment(migration 0312_managed_environment.sql):active_revision_idFKsmanaged_environment_revision(id)ON DELETE SET NULL;next_revisionis an atomic counter claimed by row-lockingSELECT ... FOR UPDATE(no colliding revisions, enforced byUNIQUE (environment_id, revision)). Secrets never live in the sealedsnapshot— the same CHECK that protectsmcp_fabric_catalog.env_refsrefutessk-,AKIA…, andBearer …literals.managed_environment_revision: immutable; the historical row is never mutated, so a rollback creates a new deployment targeting the prior revision instead of rewriting history (apps/web/src/lib/cloud/managed-environments/data-access.tsexposeslistManagedEnvironments,getManagedEnvironment, androllbackEnvironmentRevision).
Availability and limits
- Cold start. A pod reaches
readyvia warm-pool snapshot-resume (preferred) or cold snapshot-restore; the target cold-start bound is stated in the compute-plane design, not asserted here. - Compute horizon. Bounded by
vcpu_seconds_budget(default 7200s) andlease_expires_at; auto-suspend on 15 min idle; auto-terminate on budget/lease exhaustion. - Scale.
compute_pod_installation_status_idxindexes live (running/provisioning) pods for the reaper;compute_pod_profile_idxis partial on non-nullprofile_id. - No duration metering. Pod compute is a non-monetary quota surface; charges for vCPU-time are a tracked gap (see Billing).
Failure and recovery
- Termination crash mid-flight. A lifecycle worker that sets
status = 'terminating'and then crashes before destroying the guest VM leaves a live machine still heartbeating. The reaper force-reaps by reconciling the provider's live sandbox list (not the guest'slast_heartbeat_at), so the leaked VM is reclaimed and its resolvedsecret://values stop being live. - Revocation crosses instances. Capability/consent revocation is evaluated at
invocation and swept into running pods within a bounded, stated interval; the
revocation watcher (
packages/compute-plane/src/revocation-watcher.ts) terminates pods launched under a revoked grant and cuts in-flightexec/logsstreams. - Tenant isolation is refuse-closed. Cross-tenant
podIdreference, unauthenticated exec, wrong-tenant pod exec, and leaked endpoint URLs all resolve to a structured not-found/refusal — never to another tenant's data. - Warm-pool hygiene. A pooled microVM previously serving installation A is either single-tenant for its whole lifetime or fully reset (memory zeroed, disk reverted to the pristine template) before reassignment, with no residual state readable by installation B.
Recovery summary
| Incident | Recovery |
|---|---|
| Orphaned pod (substrate/control-plane split) | Reaper reconciles provider live list, force-reaps to terminated |
| Terminated mid-shutdown | Same reconciliation; idempotent terminate no-ops on re-invocation |
| Lease / vCPU budget exceeded | Pod hard-stopped, terminal state recorded, user notified |
| Revoked capability while pod runs | Revocation watcher pauses/terminates pods within a bounded interval; exec streams cut |
| Mislabelled residency after launch | Non-compliant repo unmounted, pod marked terminating, incident audit row written |
| Rotated/revoked secret on a paused pod | On resume, secret:// refs re-resolve; no persisted secret value is trusted |