Retrieval
Concepts sketches how retrieval and trust meet at recall; this page walks the read path itself, stage by stage, with the shipped-versus- designed status of each.
Retrieval is one read path over both memory and code — with an honest
caveat: the local recall service and hosted memory recall share the fusion and
calibration machinery, while the code tools (zephr_search, zephr_grep,
zephr_lsp) and zephr_graph_query are separate read paths today. A query
that looks like an answer request is refused with retrieval_required if
it tries to bypass the fused path, so the trust machinery cannot be routed
around.
The flow, in order
- Dispatch (shipped). Lexical retrieval runs through SQLite FTS5
(
belief_fts, BM25 ordering); semantic retrieval through sqlite-vec (scope-safe nearest-neighbor, manifest-verified native artifacts). Semantic runs per its configured mode and produces a typed abstention (semantic_unavailable,semantic_vector_required,semantic_capacity_exceeded) rather than a silent lexical answer when a required tier is unavailable. - Query understanding (designed). Intent classification and rewrite planning are specified — none ships. Your query is dispatched as-is.
- Fusion (shipped). Candidates fuse by reciprocal rank — ranks only,
never raw scores (
fuseCandidates,RRF_K = 60inpackages/core/src/retrieval/policy.ts). Provider-native scores are kept for explanation, never compared. The fusion policy is versioned (z04-v1) and recorded on every hit, locally and on the hosted path. - Scope resolution (shipped fence; designed lattice resolver). Results are
fenced by the granted scope; fenced-out results are counted and disclosed,
and an all-fenced query abstains with
scope_denied. The full most-specific-wins lattice resolver is specified. - Re-authorization (shipped). Every candidate is re-authorized against the authority store before disclosure — a vector hit is a candidate, never an answer. The recall path over-fetches a bounded multiple of the requested limit, drops rejected or degraded sources, and never admits content from a request-side cache alone.
- Trust-aware ranking (shipped). Review standing is a fusion tie-break:
confirmed above needs-review above unreviewed. A contradiction at the top
of the ranking forces an abstention by default (
unresolved-contradiction, disclosing both belief ids); the per-worktree Trust Firewall — see Trust — can set the policy todemoteinstead, which downranks and discloses. Stale evidence is filtered and time-adjusted; freshness-based refusal is designed. - Reranking (designed). A cross-encoder over a bounded candidate set is specified — none ships. Fusion order is served directly.
- Answer or abstain (shipped). Scores are calibrated to comparable
probabilities before the threshold decision
(
packages/core/src/lens/calibration.ts, versioned calibration tables). At or above the threshold, results carry a per-hit explanation: ranks, adjustments, final score, calibrated probability, policy version. Below it, a structured abstention returns empty answer items with rejected candidates labeledbelowThreshold.
Abstention
Abstention is a typed result, not an error. Shipped kinds include
insufficient-confidence (naming the best probability and threshold),
no-evidence (naming the tiers consulted), unresolved-contradiction,
scope_denied, the semantic:* family, calibration_stale,
calibration_refused, provenance_fail, and file_unparsed. Thresholds are
configurable per caller with a floor (0.1) so abstention can never be
configured away, and clamping is disclosed. Abstention publishes coverage —
what was consulted, what was skipped and why — and never degrades into a
plausible guess downstream: rejected candidates cannot re-enter as answers.
Guarantees
- Deterministic — identical inputs produce identical results; the fusion sort is a total order ending in stable id tie-breaks.
- Parity-gated — local and hosted ranking agree within a declared tolerance, enforced by a CI-blocking benchmark. The 0.9 top-k overlap tolerance is currently a declared placeholder pending burn-in measurement, and the benchmark fails closed until it is replaced.
- Never metered — recall is throttled for fair use and never charged; quota exhaustion does not take retrieval away.
- Reads the belief layer — recall never returns L0 observations; they are reachable only by explicit handle lookup.
Sources
openspec/specs/retrieval/spec.md— the read-path contract with per- requirement build states.packages/core/src/retrieval/—policy.ts,service.ts,types.ts,trust-revalidation.ts,budget.ts.packages/core/src/lens/calibration.ts— calibration tables and abstentions.packages/mcp-server/src/tools/recall.ts— the MCP surface and tier-bypass refusal.