Zephr Docs

Retrieval

Concepts sketches how retrieval and trust meet at recall; this page walks the read path itself, stage by stage, with the shipped-versus- designed status of each.

Retrieval is one read path over both memory and code — with an honest caveat: the local recall service and hosted memory recall share the fusion and calibration machinery, while the code tools (zephr_search, zephr_grep, zephr_lsp) and zephr_graph_query are separate read paths today. A query that looks like an answer request is refused with retrieval_required if it tries to bypass the fused path, so the trust machinery cannot be routed around.

The flow, in order

  1. Dispatch (shipped). Lexical retrieval runs through SQLite FTS5 (belief_fts, BM25 ordering); semantic retrieval through sqlite-vec (scope-safe nearest-neighbor, manifest-verified native artifacts). Semantic runs per its configured mode and produces a typed abstention (semantic_unavailable, semantic_vector_required, semantic_capacity_exceeded) rather than a silent lexical answer when a required tier is unavailable.
  2. Query understanding (designed). Intent classification and rewrite planning are specified — none ships. Your query is dispatched as-is.
  3. Fusion (shipped). Candidates fuse by reciprocal rank — ranks only, never raw scores (fuseCandidates, RRF_K = 60 in packages/core/src/retrieval/policy.ts). Provider-native scores are kept for explanation, never compared. The fusion policy is versioned (z04-v1) and recorded on every hit, locally and on the hosted path.
  4. Scope resolution (shipped fence; designed lattice resolver). Results are fenced by the granted scope; fenced-out results are counted and disclosed, and an all-fenced query abstains with scope_denied. The full most-specific-wins lattice resolver is specified.
  5. Re-authorization (shipped). Every candidate is re-authorized against the authority store before disclosure — a vector hit is a candidate, never an answer. The recall path over-fetches a bounded multiple of the requested limit, drops rejected or degraded sources, and never admits content from a request-side cache alone.
  6. Trust-aware ranking (shipped). Review standing is a fusion tie-break: confirmed above needs-review above unreviewed. A contradiction at the top of the ranking forces an abstention by default (unresolved-contradiction, disclosing both belief ids); the per-worktree Trust Firewall — see Trust — can set the policy to demote instead, which downranks and discloses. Stale evidence is filtered and time-adjusted; freshness-based refusal is designed.
  7. Reranking (designed). A cross-encoder over a bounded candidate set is specified — none ships. Fusion order is served directly.
  8. Answer or abstain (shipped). Scores are calibrated to comparable probabilities before the threshold decision (packages/core/src/lens/calibration.ts, versioned calibration tables). At or above the threshold, results carry a per-hit explanation: ranks, adjustments, final score, calibrated probability, policy version. Below it, a structured abstention returns empty answer items with rejected candidates labeled belowThreshold.

Abstention

Abstention is a typed result, not an error. Shipped kinds include insufficient-confidence (naming the best probability and threshold), no-evidence (naming the tiers consulted), unresolved-contradiction, scope_denied, the semantic:* family, calibration_stale, calibration_refused, provenance_fail, and file_unparsed. Thresholds are configurable per caller with a floor (0.1) so abstention can never be configured away, and clamping is disclosed. Abstention publishes coverage — what was consulted, what was skipped and why — and never degrades into a plausible guess downstream: rejected candidates cannot re-enter as answers.

Guarantees

  • Deterministic — identical inputs produce identical results; the fusion sort is a total order ending in stable id tie-breaks.
  • Parity-gated — local and hosted ranking agree within a declared tolerance, enforced by a CI-blocking benchmark. The 0.9 top-k overlap tolerance is currently a declared placeholder pending burn-in measurement, and the benchmark fails closed until it is replaced.
  • Never metered — recall is throttled for fair use and never charged; quota exhaustion does not take retrieval away.
  • Reads the belief layer — recall never returns L0 observations; they are reachable only by explicit handle lookup.

Sources

  • openspec/specs/retrieval/spec.md — the read-path contract with per- requirement build states.
  • packages/core/src/retrieval/ — policy.ts, service.ts, types.ts, trust-revalidation.ts, budget.ts.
  • packages/core/src/lens/calibration.ts — calibration tables and abstentions.
  • packages/mcp-server/src/tools/recall.ts — the MCP surface and tier-bypass refusal.