Zephr Docs

Compute and managed environments

Two related capabilities own the runtime substrate: compute-plane materializes a profile into an isolated, ephemeral pod and governs its lifecycle, egress, budget, and surfaces; managed-environments is the user-facing desired-state object (an immutable, sealed EnvironmentRevision sequence) that pods and hosts are launched from. Neither redefines memory, ranking, or metering — those remain the memory, retrieval, and economics capabilities. Tenancy is shared: every table here is scoped on installation_id under the platform RLS contract (tenant_installation_matches(), ENABLE + FORCE on every table, GUC-keyed, zero rows when unbound).

Status at a glance

ConcernStateEvidence
Pod lifecycle seamShips — SandboxRuntimeAdapter port + compute_pod table (0128/0224)packages/compute-plane/src/types.ts, packages/compute-plane/src/sandbox-runtime.ts
Egress + budgetShips — EgressGuard, vCPU-second ceiling (default 7200s)packages/compute-plane/src/egress-guard.ts, budget.ts
Runtime implementationEvolving — E2B + local-kvm seam live; Firecracker-direct is Wave 6packages/compute-plane/src/sandbox-runtime.ts (RuntimeMode)
Managed environmentsEvolving — data model + read/create ship, no UI yet (W1)apps/web/src/lib/cloud/managed-environments/data-access.ts
Billing / durationNot metered — compute time is a standing gap, vCPU budget onlyspec C-3; add-compute-plane-economics

Architecture at a glance

flowchart TD
    A[Desired state: EnvironmentRevision] --> B[compute-plane worker]
    B --> C[createSandboxRuntime]
    C -->|e2b / firecracker / local-kvm| D[microVM / sandbox]
    D --> E[lens builders index mounted repos]
    D --> F[agent fleet + MCP / LSP servers]
    D --> G[exec / ACP / A2A surfaces]
    G --> H[EgressGuard: DNS-pin + allowlist]
    F --> I[outbox emit on commit]
    G --> J[credential-bound surface creds]
    K[reaper] -->|lease / vCPU budget / orphan| D
    L[revocation watcher] -->|bounded interval| D

The control plane is provider-neutral: SandboxRuntimeAdapter (provisionPod / terminatePod / pausePod / resumePod / getHandle / listLiveSandboxIds) is injected per environment via createSandboxRuntime() with RuntimeMode = 'e2b' | 'firecracker' | 'local-kvm' and the ZEPHR_COMPUTE_RUNTIME env var. The shipped runtime walks E2B and a local-kvm fallback; Firecracker-direct is Wave 6 (ZEP-2345) and is not live on main. The spec's Firecracker-microVM-plus-devcontainer framing is the architectural target, not the current default runtime.

Pod lifecycle

A pod's durable state lives in the compute_pod row (migration 0128_compute_pod.sql plus the quota columns in 0224_compute_pod_quota_columns.sql). Its status column CHECK is the exact PodStatus surface:

export type PodStatus =
  'pending' | 'provisioning' | 'running' | 'paused' | 'terminating' | 'terminated' | 'orphaned';

Legal edges: pending → provisioning → running; running ⇄ paused; any live state → terminating → terminated; any live state → orphaned → terminated. orphaned is the reaper's marker when the substrate and the control plane disagree.

  • Auto-suspend. A running pod with no activity for 15 minutes is snapshotted and transitioned to paused; the user can resume within the retention window.
  • Auto-terminate. The reaper enforces the vCPU-time ceiling and lease_expires_at (see Budget); crossing either terminates the pod and records the terminal state.
  • Orphan reap (negative). The reaper reconciles the provider's live sandbox list against compute_pod — not last_heartbeat_at, which is guest-supplied and untrustworthy for a VM that is still alive. A pod stuck in terminating past a bounded interval is reaped even while heartbeating, so a crash mid-terminate cannot leak a live VM that burns vCPU, holds resolved secrets, and keeps egress open.
  • Idempotent terminate. Re-invoking terminate on a terminating/terminated pod is a no-op success, never a double-destroy and never an error.
  • Live handle. SandboxHandle.status is the runtime-level view ('running' | 'paused' | 'terminated'); the four RuntimeError codes (pod_not_found, already_terminated, invalid_state) are refusal shapes, not crash surfaces.

Configuration

KnobSourceDefaultNotes
Runtime modeenv ZEPHR_COMPUTE_RUNTIME | injectlocal-kvm fallbacklocal/local-kvm/firecracker/e2b; local always wins as the safe default
vCPU budgetcompute_pod.vcpu_seconds_budget (0224)7200 (2h)DEFAULT_VCPU_SECONDS_BUDGET = 2*60*60; NON-MONETARY
Lease expirycompute_pod.lease_expires_atunsetoptional wall-clock hard stop
Egress pin TTLDEFAULT_PIN_TTL_MS300000 (5 min)control-plane DNS pin freshness
Network allowlistNetworkScopeEntry[] on scope bindingtenant scope{ host, port? }; narrows the platform maximum, never widens
Pod templateprofile / worktreeId / templateId—materializes the profile's MCP servers, rules, hooks, model prefs

Every operation keyed by a caller-supplied podId first resolves the pod within the caller's installation_id; a pod belonging to another installation returns the same tenant-scoped not-found as a missing pod (no existence oracle).

Operations that mutate a pod or attach a surface — launch | pause | resume | terminate | exec | mcp | acp — additionally require a capability grant and pass a consent gate. mcp and acp are gated at the same strength as exec: attaching the pod's tool surface or driving its agent over ACP confers at least exec authority, so a gate that covers exec but omits mcp/acp would invert the threat model. A refused operation appends a trust evidence row at review_state = 'unreviewed' that an agent cannot self-confirm.

Egress (default-deny, platform-maximum)

Egress is default-deny at the microVM network layer, bounded by a platform-maximum allowlist that a tenant scope may only narrow, never widen. Enforcement is on the resolved destination address, not a presented name: DNS is resolved by the control plane, the answer is pinned for the connection, and the guest connects only to pinned addresses. EgressGuard returns either { allow: true, pinned } or { allow: false, reason, audit } with these block reasons:

egress_refused_unpinned_address; // destination never pinned by control-plane DNS
egress_refused_dns_rebinding; // resolved address != pinned address for the name
egress_refused_sni_not_allowlisted;
egress_refused_ip_literal; // no DNS/SNI (e.g. IP-literal connect)
egress_refused_stale_pin;
BLOCK_REASON_GUEST_TO_HOST_ESCAPE; // metadata service / host / peer-tenant / link-local

Name-only matching is explicitly insufficient: an IP-literal connect performs no DNS lookup and sends no SNI; ECH/ESNI encrypts the SNI; and a low-TTL rebind can return an attacker address for a legitimately allowlisted name. Link-local and RFC1918 destinations are denied outright — including the canonical metadata address (169.254.169.254), reached with no DNS, no SNI, no TLS. Every refused attempt is auditable: timestamp, source process, destination, and matching reason.

Compute budget and miner detection

Budget bounds vCPU-seconds only — it is not a bound on token-denominated work. An agent generating output tokens is waiting on a remote model API and burns near-zero CPU, so it can run far past any token ceiling without approaching this stop. Pod compute time is not metered: neither this change nor add-compute-plane-economics charges for vCPU-time or wall-clock residency (add-compute-plane-economics adds only the token kinds run_llm_agent_task and run_agent_lens_index). The vCPU budget is the only cost control and a standing gap, not a deferral.

The accrued ceiling is enforced host-side and is the primary miner control: the maximum overrun a pod can achieve is the sum of the budget-flush interval and the reaper-evaluation interval. Syscall-signature monitoring is best-effort and guest-defeatable, explicitly not a fail-closed gate: Firecracker's seccomp filters restrict the host VMM system calls, while the guest runs its own kernel behind KVM, so guest execve is not observable to a host filter. An in-guest agent may exist but is labelled best-effort and guest-defeatable — a hostile guest with root disables it in one syscall, so it is not counted as defence-in-depth. The false-positive ceiling is documented: cargo build, tsc --build, test suites, dev servers, and LLM inference are the intended workloads and must not trip the heuristic.

Secrets and residency

  • Runtime-only injection. secret:// references resolve only at runtime on the target installation; a saved/shared PodTemplate carries references, never resolved values. A template containing a resolved secret is rejected/stripped at save.
  • Memory snapshots are secret-bearing. A Firecracker full snapshot writes the mem_file_path as a complete copy of guest memory, so a claim that resolved secrets never reach the persisted snapshot is true for disk/template artifacts and false for memory snapshots. The memory file is encrypted at rest with a tenant-scoped key, retention-bounded, and deleted on terminate; on resume every secret:// reference is re-resolved rather than trusting the value captured in the snapshot.
  • Exec is the blast radius. A principal permitted to exec can read every secret:// value present in the guest environment by construction. Capability + consent gating (not sandboxing) is the control; exec runs unprivileged (uid 1000, never 0), bounded by exec_timeout_seconds and exec_max_output_bytes, and every exec is audited — including the allowed path — with principal, podId, command string, exit code, and duration.
  • Residency. A repo with a declared residency (lens_repo.residency ∈ ('global','eu','us')) must not be materialized into a non-matching region. The placement check runs on launch and independently on resume (which does not pass through provisionPod, so it re-checks against the restore region). Violations of a mislabelled repo mount the non-compliant repo, mark the pod terminating, and write a residency-incident audit row — both forward-looking remedies, because the crossing has already happened.

Managed environments (desired-state model)

A Managed Environment is the user-facing desired-state object; an EnvironmentRevision is an immutable, sealed desired-state row. Switching, rollback, and sync operate on revisions, never on a mutable live blob.

export interface ManagedEnvironmentRow {
  readonly id: string;
  readonly installationId: string;
  readonly name: string;
  readonly activeRevisionId: string | null;
  readonly createdAt: string;
  readonly updatedAt: string;
}
export interface ManagedEnvironmentRevisionRow {
  readonly id: string;
  readonly environmentId: string;
  readonly revision: number; // 1,2,3... linear, no gaps
  readonly snapshot: Record<string, unknown>; // sealed; secrets are secret:// refs only
  readonly sealedAt: string;
}
  • managed_environment (migration 0312_managed_environment.sql): active_revision_id FKs managed_environment_revision(id) ON DELETE SET NULL; next_revision is an atomic counter claimed by row-locking SELECT ... FOR UPDATE (no colliding revisions, enforced by UNIQUE (environment_id, revision)). Secrets never live in the sealed snapshot — the same CHECK that protects mcp_fabric_catalog.env_refs refutes sk-, AKIA…, and Bearer … literals.
  • managed_environment_revision: immutable; the historical row is never mutated, so a rollback creates a new deployment targeting the prior revision instead of rewriting history (apps/web/src/lib/cloud/managed-environments/data-access.ts exposes listManagedEnvironments, getManagedEnvironment, and rollbackEnvironmentRevision).

Availability and limits

  • Cold start. A pod reaches ready via warm-pool snapshot-resume (preferred) or cold snapshot-restore; the target cold-start bound is stated in the compute-plane design, not asserted here.
  • Compute horizon. Bounded by vcpu_seconds_budget (default 7200s) and lease_expires_at; auto-suspend on 15 min idle; auto-terminate on budget/lease exhaustion.
  • Scale. compute_pod_installation_status_idx indexes live (running/provisioning) pods for the reaper; compute_pod_profile_idx is partial on non-null profile_id.
  • No duration metering. Pod compute is a non-monetary quota surface; charges for vCPU-time are a tracked gap (see Billing).

Failure and recovery

  • Termination crash mid-flight. A lifecycle worker that sets status = 'terminating' and then crashes before destroying the guest VM leaves a live machine still heartbeating. The reaper force-reaps by reconciling the provider's live sandbox list (not the guest's last_heartbeat_at), so the leaked VM is reclaimed and its resolved secret:// values stop being live.
  • Revocation crosses instances. Capability/consent revocation is evaluated at invocation and swept into running pods within a bounded, stated interval; the revocation watcher (packages/compute-plane/src/revocation-watcher.ts) terminates pods launched under a revoked grant and cuts in-flight exec/logs streams.
  • Tenant isolation is refuse-closed. Cross-tenant podId reference, unauthenticated exec, wrong-tenant pod exec, and leaked endpoint URLs all resolve to a structured not-found/refusal — never to another tenant's data.
  • Warm-pool hygiene. A pooled microVM previously serving installation A is either single-tenant for its whole lifetime or fully reset (memory zeroed, disk reverted to the pristine template) before reassignment, with no residual state readable by installation B.

Recovery summary

IncidentRecovery
Orphaned pod (substrate/control-plane split)Reaper reconciles provider live list, force-reaps to terminated
Terminated mid-shutdownSame reconciliation; idempotent terminate no-ops on re-invocation
Lease / vCPU budget exceededPod hard-stopped, terminal state recorded, user notified
Revoked capability while pod runsRevocation watcher pauses/terminates pods within a bounded interval; exec streams cut
Mislabelled residency after launchNon-compliant repo unmounted, pod marked terminating, incident audit row written
Rotated/revoked secret on a paused podOn resume, secret:// refs re-resolve; no persisted secret value is trusted