The Evidence Facility

Evidence is file-shaped output the runtime keeps: a report an application wrote, a recording a sidecar made, a log something outside this process is still appending to. This is the design of that facility — what it adds, which decisions are closed, and which keel scars each decision exists to avoid.

The facility is delivered end to end: the facade and its storage port in harmos, the EvidenceDirectory adapter in examples/jsonl-adapter, a scripted lifecycle in the simulation harness, and the guide chapter that teaches both worked examples. Harmos ships the port; the adapter is a worked implementation an application copies. Every behaviour of the port on this page is a named test in crates/harmos/tests/evidence.rs.

Objective

Give applications one honest home for the three shapes above, which are the same concept at different stages: scoped, digest-verified, journal-known, boundedly readable. Nothing here is a second durable registry, and nothing here is a directory a reader is asked to trust.

The One Structural Decision

Finalize couples bytes and fact. Keel kept evidence truth in its store index while applications journaled their own copy in a second step. Dokime's upload_artifact_bytes does store-finalize and then a separate journal commit, and a crash between them strands a finalized, digest-verified evidence that application state never learns about — unreachable through every application API, never garbage collected, with no reconciliation pass anywhere that would find it.

Here, finalization is the recorded fact, and the order is the invariant:

1. the staged bytes are made durable                     Vault::verify
2. the digest and length are recomputed by streaming     Vault::verify
   those bytes back — never taken from the producer
3. one SystemFact carrying them is recorded              Journal::record_system
4. and awaited durable                                   Journal::wait_until_stored
5. the evidence enters the finalized index               visible before removed
6. the verified draft takes its finished name            Vault::claim
7. the draft leaves the active index

Each step earns the next, and each crash window has one honest outcome:

A crash falls	What survives	What the next boot sees
before 3	staged bytes, no fact	an unfinished draft — resume it or abandon it
between 3 and 6	a fact, bytes still staged	the evidence; the first read completes the claim
after 6	a fact and named evidence	the evidence

Recovery reads facts, never a directory listing. A draft with no fact is not a visible evidence under any circumstances, so a partial capture is never mistaken for evidence. list folds from the journal like every other read in this runtime — at boot, from the same replayed order the state folds from, and from the facility's own finalizations after that.

One consequence is worth stating plainly rather than discovering: an evidence fact is an ordinary entry in the one order. An application that prunes history beneath a snapshot prunes its evidence facts with it. The bytes remain, and the next boot meets them as an unfinished draft — which is the honest answer for a runtime whose truth is its journal, but it means an application with long-lived evidence should not prune beneath it.

The Second Decision: There Is No Evidence Id

The triple an application names — kind, scope, key — is the identity, and it does not change when a draft becomes evidence.

That single choice removes three separate pains keel's consumers paid. Keel minted an opaque artifact-{n} per draft and left the caller-supplied logical key write-only — there was no lookup by it — so every consumer read degenerated into listing a whole scope and linear-scanning decoded metadata, once over active drafts and again over finalized ones. It also gave one conceptual file two ids, before and after finish, which the consumer had to stitch back together by hand.

Here, one read serves both stages through the same key, so a capture can be tailed straight through its own finalization; list is one entry per key with neither a gap nor a duplicate; and there is nothing to stitch.

The Third Decision: A Kind Declares What Its Fact Carries

The finalize fact carries the application's own account of the evidence, and that account is typed, declared by the kind, and nothing else. EvidenceKind is the declaration — Identified for the name a kind has always had, plus one associated Details type — and EvidenceRecord::details::<K>() is the only way back out.

This is symmetry rather than a new idea: every declared fact in this runtime says what its payload is at the site that authors it, and an evidence fact was the last one whose payload was the facility's alone. Dokime's request is what made the gap concrete — a finalized upload has to carry the producer's filename, its content type, the application's own id for it, and when it was created, and without somewhere to put them the application ran a second transaction after finalize. That is exactly the crash window and the second registry this facility exists to remove, rebuilt one layer up.

Two properties are load-bearing, and both are tested rather than asserted:

• The declaration is the decode. details::<K>() refuses EvidenceError::Mismatched when K is not the kind the evidence was opened under, so a consumer cannot name a shape the declaration never sanctioned. Opaque bytes with an agreed decoder at the far end would have been the same thing one honesty short.
• The recorded bytes are never re-encoded. The account is serialized once, at open, and travels as raw JSON text inside the fact — written once, folded back verbatim, and handed to serde only when a caller asks. A replay re-derives nothing, so the wire canon a history was written under is the one it is read under.

The account is given at open rather than at finalize because open is the one call that names the kind. That is what keeps finalize and finalize_scope free of a payload argument no caller could type across several kinds, and it makes an unfinished draft describe itself already: list answers details for a draft and finalized evidence alike.

A kind with nothing of its own to say declares type Details = (), which serializes to null — which is precisely what a finalize fact written before this declaration existed already carries, so history written by an earlier build folds without a migration.

Vocabulary

• Evidence<A> — the facade, held on Runtime and reachable from inside a resident service. Cheap and cloneable.
• EvidenceKind — the kind declaration: a permanent name through Identified, and the Details type that kind's finalized fact carries.
• EvidenceRecord — one evidence at whichever stage it has reached. finalized is Option<At<u64>>: the verified digest stamped with where its fact landed, and None while it is still a draft. Both or neither, structurally.
• Draft — evidence still being written. A cheap addressed handle, not an owner: Clone, and the runtime's index is the authority.
• Excerpt — one bounded read: requested, the clamped offset, the bytes, the length observed, and next().
• EvidenceError — why one operation could not be carried out.
• Vault — the storage port. Declared in harmos beside Source and Series; every concrete implementation lives in an application, or in the reference adapter an application copies.

The Declaration Surface

impl EvidenceKind for Recording {
    type Details = Upload;
}

builder.evidence(
    EvidenceDirectory::at("/var/lib/acme/evidence")?,
    [Recording::definition(), Report::definition()],
)

One call, one frozen catalog, exactly as register and streams are. The catalog is still (DefinitionId, u32) pairs — Identified::definition() answers with exactly what a frozen catalog needs, so boot holds no decoder and the facility still adds no authoring attribute. What a kind declares beside that is its Details type, which lives in the type system rather than in the catalog: the verbs that name a kind reach it, and a fold that meets an unknown kind at boot never has to. Boot refuses a catalog that declares a kind twice; open refuses a kind outside it.

Without the declaration the runtime keeps no evidence and every verb refuses EvidenceError::Unserved on the code path a declared vault takes — the same "absence is not a lesser mode" rule Storage follows.

Shape

The core module (crates/harmos/src/evidence/) owns the facade, the runtime-owned active-draft index keyed by scope and then by (kind, key), the frozen kind catalog, the in-memory fold of finalized facts, and the name encoding. Products never hand-roll a Mutex<BTreeMap> of open drafts; keel's first shape forced its one consumer to, and deleting that map was the whole of one upstream request.

The storage port is Vault: open, external, append, length, verify, claim, quarantine, read, read_draft, plus the two knobs a backend owns — ceiling (the most bytes one read answers with) and cadence (how often a foreign writer's draft is looked at again). The bounds live on the port because the backend is what pays for them: a large window is cheap over a local file and is not over an object store. verify is one method rather than two because its halves are an order — a digest folded before the bytes are durable describes a file that may not survive the fact about to name it.

Names reaching a vault are already safe. The facade escapes kind, scope, and key to a character set no filesystem argues with and joins them with a byte the escape never produces, so a backend never defends itself against a caller's separator or ... Keel left path safety to the application, and its one consumer wrote a path validator to survive it.

The adapter is EvidenceDirectory: drafts/, kept/, and quarantine/ under one root, where which directory a file sits in is its whole status. The durability sequence is the one every adapter beside it follows — write, sync the file, rename, sync the parent directory. Keel's evidence store had none of it: it copied a promoted draft straight over the visible blob path, so a crash mid-copy left a truncated evidence exactly where readers look. Nothing is deleted; an abandoned draft is moved into quarantine, because a partial capture is the only account of an interrupted write.

The adapter carries no codec. There is no format to leave out and no dependency it would drag in, which is why it is the one adapter an application can copy without inheriting a stored representation it then has to keep.

Reads and Watching

One Excerpt answers a live draft and a finalized evidence. Reading past the end is not an error: offset clamps, bytes comes back short or empty, and requested is how a caller tells that it did. Asking for more than the vault's ceiling is an error — EvidenceError::Bound — because a bound a caller got wrong is a bug worth hearing about, not something to quietly shorten.

watch is opt-in all the way down. A draft nobody watches costs nothing, and a draft a foreign process is writing is looked at again only while a receiver is alive; the last watcher leaving stops the looking, and the next one starts it afresh. The last value a watcher sees is the length the evidence ended at, and the channel closing after it is how a watcher learns there will be no more. Keel's own request rejected notify-by-convention — a changed() call every native producer must remember — and so does this.

Sidecar Integration

An external draft is what hands a Supervised child its output path, which closes the open "paths the embedder gave it" hole in the sidecar design. Service::evidence() is the reach: a recorder resident opens the draft, hands Draft::path() to the child before launching it, and finalizes on the completion notice. Writing through an external draft from the host refuses EvidenceError::Foreign — two writers appending to one capture interleave, and the runtime does not pretend otherwise.

Executable::at resolves and validates the program before supervision, so a relative, missing, or non-file target refuses as ExecutableError at the declaration boundary instead of becoming a delayed service crash.

Tailed Logs

No log-specific substrate. A tail is an ordinary Resident that appends into a draft through the facade and lets the watch machinery report growth, and it ships as the guide's second worked example over the local filesystem. Remote transports — ftp, sftp, s3 — are a recorded later layer: the draft-append seam is the port's, so they slot in as vaults feeding drafts, and none is built now.

Vocabulary Resolution

Earlier work retired three non-tenant uses of artifact: the JSONL adapter's private Artifact became Document; a series directory holds files; and the journal writer's pre-position Draft became Unsealed. That history stays true, including its stored generation field and every wire format.

The remaining collision was this facility. It is now Evidence. Artifact is reserved for an installable tenant file: an artifact is what you install; a tenant is what runs. harmos-artifact, direct guest artifacts, and sidecar artifacts keep that vocabulary. No persisted spelling changed: evidence facts continue to use harmos:work/artifact/finalized so existing history folds.

Refinements From the Frozen Contract

Four decisions moved while building, each for a reason the contract's own rules force:

• Kinds earned no attribute. The contract said kinds are "declared like stream rows (derive/attribute, catalog-frozen)". Identified::definition() already returns exactly the (DefinitionId, u32) pair a kind needs, so the catalog is frozen without an authoring attribute of its own — the evidence facility left harmos-macros exactly as it found it, and still does. What the freeze buys is kept: a kind declared twice refuses the boot, and an undeclared kind refuses at open. The one census name a kind costs arrived later and deliberately: EvidenceKind is what names the payload the finalize fact carries, and that is a declaration no identity could make.
• The store trait is in harmos, the implementation is not. The contract lists "store trait defined in the facade crate instead of the storage crate" as a scar and then says "ours goes in the storage crate", which cannot mean the trait — anything implementing these ports depends on harmos, so a port declared out there would be unnameable by the facade. Read as intended: the concrete backend must not live in the facade crate. Keel's FsStore did, asymmetrically with its journal and stream backends one crate out. Vault sits beside Source and Series; EvidenceDirectory sits beside Jsonl and Snapshot, outside the facade entirely.
• The per-read ceiling and the watch cadence are the vault's, not a policy struct's. Both are things only the backend can answer for, and locating them on the port keeps the census smaller and makes a future remote adapter able to state its own. Neither is a hardcoded limit: EvidenceDirectory declares defaults and reading and watching name them.
• A finalized key stays claimed. Opening a draft for a key evidence already holds refuses EvidenceError::Conflict rather than superseding it. Evidence is not overwritten, and a re-run that wants a second capture names a second key.

What This Build Does Not Do

• The digest is FNV-1a 64, and it is not tamper evidence. It answers the question a finalize actually has — are these the bytes that were written — while claiming nothing about an adversary who can rewrite a journal beside a file. A cryptographic digest is a dependency decision, and this facility does not take one on its own.
• A bounded read verifies nothing. The digest is recomputed at finalize and recorded; a later windowed read trusts the bytes it finds. Verifying a window needs chunk-level or Merkle digests recorded at finalize, which is a recorded later layer rather than something built here. Keel has the same gap and does not name it; this does.
• A same-length rewrite is invisible to a watch. Growth is length, so a foreign writer that overwrites a fixed-size record in place produces no wakeup. Appends and truncations are seen.
• finalize_scope has no rollback. A fact is truth the moment it lands, so a refusal partway leaves the evidence already finalized finalized and visible through list, and the rest still open.
• Verification and reads are blocking inside an async method, as every other adapter beside it is. Reads are bounded by the ceiling; the one unbounded pass is the verify at finalize.
• Nothing bounds a kind's account. The requesting consumer asked for bounded application metadata and this build bounds none: Details is whatever the kind declares, and it rides an ordinary journal entry, so a large account is paid for on every replay of that history — the same bulk-bytes-in-the-order cost this facility exists to keep out. Kinds here describe evidence in a handful of fields, and a declared ceiling is a follow-up for the first consumer that needs one rather than a number guessed now.
• An account is fixed when the draft opens. Details is given at open, so a value only the end of a capture knows — a recording's duration, a final page count — is not what this carries. Every field the requesting consumer named is known when the upload begins, and a value known only at the end is an application record beside the evidence rather than a second write into its fact. EvidenceRecord::version is the seam if a kind's account has to change shape: it is the kind version the draft was opened under, recorded in the same fact.
• Nothing sweeps the quarantine. Retention beyond draft-abandon quarantine is a follow-up once real consumers press.

Non-Goals

Bulk bytes in the journal or on the sidecar wire (facts and telemetry only); evidence bytes as stream rows; a second durable registry beside the journal; retention beyond quarantine; evidence declarations for directly installed tenants; ftp, sftp, and s3 transports.

Validation Bar

mise run check, mise run test, and mise run build:docs:release green, with the census updated in the same commit as the surface change.

Behaviour, each a named test: a written evidence verified before any fact names it; an external draft written by a real child process, finalized, and folded back; a draft no fact names never becoming visible across a reboot, and resuming with its staged bytes; a fact recorded before its claim still reading after a reboot; a bounded read clamping at the end of either stage and refusing a bound past the vault's ceiling; a scope finalize visible before it is removed; a watch coalescing and closing on its final revision; an unwatched foreign draft never looked at; nothing verified meaning nothing recorded; an abandoned draft kept where nothing reads it; a claimed key staying claimed; and a build with no vault refusing on the same path.

On disk: a claim that moves bytes and survives the process that wrote them, a digest that answers for the stored bytes and nothing else, an interrupted claim completed by the next read, a quarantine that never overwrites, a read bounded by the directory's own ceiling, an external path that exists before the process that writes it, and a resumed draft that carries on.

In simulation: a scripted draft lifecycle that survives its reboot, inheriting exactly the evidence a fact finalized and never the draft no fact names, and the same trace under two identical seeds.