Evolve

The first time a customer opens a file your new build did not write, the version = 1 sitting on every definition since chapter 2 starts doing work. This chapter is how a definition changes shape without invalidating history: the flat history(...) list, what dispatch does with a stored (id, version) pair, what a build does with entries it has never heard of, and the one condition under which an old version may finally be deleted.

The Day You Ship v2

Everything so far has assumed one version of your application. That assumption expires the first time a customer opens a file your new build did not write. Their .schematic was produced by last year's editor. Its sidecar holds entries committed by a build you have since deleted from your laptop. Recovery is going to replay those entries — chapter 9 promised it would — and the transaction types they were written from have changed shape since.

This is the chapter where version = 1, sitting dormant on every definition since chapter 2, starts doing work.

A Transaction Changes Shape

Say the editor learns about buses. In v1, renaming a net wrote whatever text the user typed and applied it verbatim. In v2, a name like DATA[3] is parsed into a bus and a member index, and renaming a bus member renames its siblings too. Same operation, same wire name, different payload and different behavior.

You bump the version and declare, on the current definition, every historical version you still support:

transactions/net/rename_net.rs

#[harmos::transaction(id = "net.rename", version = 2, history(history::RenameNetV1))]
#[derive(Clone, Debug, Deserialize, PartialEq, Eq, Serialize)]
pub struct RenameNet {
    /// Which net is renamed.
    pub net: NetId,
    /// What it is called from now on. v2: parsed — "DATA[3]" is bus DATA,
    /// member 3.
    pub to: NetName,
}

impl Transaction<Schematic> for RenameNet {
    type Error = SchematicError;

    fn check(&self, schematic: &Schematic) -> Result<(), Self::Error> {
        schematic.require_net(self.net)?;
        schematic.require_name_free(&self.to)
    }

    fn apply(&self, schematic: &mut Schematic) {
        schematic.net_mut(self.net).name = self.to.clone();
    }
}

impl From<history::RenameNetV1> for RenameNet {
    /// Version 1 stored raw text and applied it verbatim, so carrying one of
    /// its entries forward carries the text as the name it *was*. Parsing it
    /// under version 2's rules here would silently change what a settled entry
    /// meant, which is the one thing an upgrade may never do.
    fn from(old: history::RenameNetV1) -> Self {
        Self {
            net: old.net,
            to: NetName::verbatim(old.to),
        }
    }
}

mod history {
    use harmos::Transaction;
    use serde::Deserialize;

    use super::{NetId, NetName, Schematic, SchematicError};

    /// The version-1 shape, kept complete so an old history still means what it
    /// meant: the new name was raw text, applied exactly as written.
    #[harmos::transaction(version = 1)]
    #[derive(Clone, Debug, Deserialize, PartialEq, Eq)]
    pub(super) struct RenameNetV1 {
        pub(super) net: NetId,
        pub(super) to: String,
    }

    impl Transaction<Schematic> for RenameNetV1 {
        type Error = SchematicError;

        /// Version 1 never asked whether the name was free. It is recorded here
        /// because the shape of a historical implementation is the whole
        /// contract — replay itself never calls it, since history is settled.
        fn check(&self, schematic: &Schematic) -> Result<(), Self::Error> {
            schematic.require_net(self.net)
        }

        fn apply(&self, schematic: &mut Schematic) {
            schematic.net_mut(self.net).name = NetName::verbatim(&self.to);
        }
    }
}

Four things in that listing are the whole convention.

Every type named in history(...) is a complete Transaction implementation — its own payload, its own check, its own apply. It is not a decoding shim and not a description of a schema. It is the transaction as it was, kept alive. When a v1 entry replays, RenameNetV1::apply runs and writes the raw text, exactly as it did the day it was committed. The bus logic never touches it, because in that user's history, buses had not been invented.

A historical change also converts into the current shape, with From. Two things come out of a stored v1 entry, and they answer different questions. Replay needs the behavior, so it gets RenameNetV1::apply. Everything above replay — the undo fold, a lagging consumer, a projection — needs a value in today's vocabulary, so it gets RenameNet::from(old). Writing that conversion is where you decide what an old entry means in current terms, and the answer is almost always "exactly what it meant then": the example's From wraps the raw text with NetName::verbatim rather than parsing it, because parsing would change a settled entry's meaning.

The derives follow from the same split. The current type is written as well as read, so it derives Serialize and Deserialize. A historical type is only ever read — no build writes v1 again — so Deserialize alone is what it needs.

The historical type declares a version and no id. An attribute with id creates the current family root; a version-only attribute marks a historical type and generates its version metadata but no separately registered definition. RenameNetV1 is reachable only through RenameNet's history(...) list, and your catalog enum from chapter 3 still has exactly one RenameNet variant. Versions are not new definitions; they are the same definition, remembered.

It all lives in one file. The current type, its private history module, the historical implementations, and their focused tests stay together vertically. The complete compatibility promise for "net.rename" is one file you can read top to bottom.

That is also why the list is flat rather than a chain of predecessors. A chain — v3 knows v2, v2 knows v1 — puts the compatibility promise in three places and quietly shortens it the day someone omits a link. A flat catalog states the whole promise on the definition that is current, where a reviewer can see it.

Records and Artifacts Convert, They Do Not Re-Run

A transaction is behavior, so an old version keeps its behavior. A record is a fact and an artifact is data, and neither has behavior to preserve. They evolve with the ordinary Rust conversion trait, straight into the current type:

#[harmos::record(id = "export.completed", version = 2, history(history::ExportCompletedV1))]
#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq, Serialize)]
pub struct ExportCompleted {
    /// v2: a closed enum, was free text.
    pub format: ExportFormat,
    /// How many symbols it covered.
    pub symbols: u32,
}

mod history {
    #[harmos::record(version = 1)]
    #[derive(Clone, Debug, Deserialize, PartialEq, Eq)]
    pub(super) struct ExportCompletedV1 {
        pub(super) format: String,
        pub(super) symbols: u32,
    }
}

impl TryFrom<history::ExportCompletedV1> for ExportCompleted {
    /// Whatever this is, it must implement `Display`: a conversion that refuses
    /// becomes a load error, and the message it carries is what the person
    /// holding the unreadable file gets to read.
    type Error = UnknownExportFormat;

    fn try_from(old: history::ExportCompletedV1) -> Result<Self, Self::Error> {
        Ok(Self {
            format: old.format.parse()?,
            symbols: old.symbols,
        })
    }
}

(The shipped example's ExportCompleted is at version 1 and has no history yet — this is what its second version would look like.)

Every historical version converts directly into the current type. When v3 arrives, it gets a TryFrom<ExportCompletedV1> of its own, not a hop through v2. Chained upgrades look economical and are not: each link becomes a conversion nobody exercises alone, and a bug in the middle of the chain corrupts everything downstream of it. Direct conversions are independently testable, and there is no harmos-specific conversion trait to learn — this is TryFrom, the one Rust already has.

There is no downgrade path in either direction of the system. Loading upgrades to the current in-memory type; saving and exporting always write the current version. A v2 build does not produce v1 files.

Under the Hood: Dispatch by (id, version)

Chapter 2 said the macro generates durable identity and nothing else. Here is what that identity is for. Every stored envelope carries the DefinitionId and the version stamped at encode time, so recovery reads them off the entry before it has any opinion about what the entry means:

stored entry:  id = "net.rename", version = 1
               |
               v
       (id, version) dispatch  ->  history::RenameNetV1
               |
               v
              apply            ->  and only apply

Four negatives define the mechanism, and each one is a class of bug that cannot happen:

• It never calls check. Not the current one, not the historical one. The entry was accepted when it was committed; validation governs the future only. This is chapter 2's rule, and it is the reason a stricter v2 cannot brick a v1 file.
• It never upcasts a transaction. A v1 entry is not converted into a v2 value and handed to v2's apply. It is decoded as RenameNetV1 and given to RenameNetV1::apply.
• It never reuses current behavior implicitly. If v1 and v2 genuinely do the same thing, you make v1's apply call a shared private function. That is ordinary Rust delegation, written down, visible in review — not a default that silently changes v1's meaning the next time someone edits v2.
• It never guesses. A version that is not in history(...) is a loud unsupported-version load error, not a best-effort attempt with the nearest implementation.

The historical check implementations exist even though replay will not call them. They are there for contract completeness — RenameNetV1 is a Transaction, not half of one — and because they make a focused test possible: you can assert what v1 accepted and refused, years after v1 shipped, without booting anything.

Entries This Build Does Not Recognize

A newer build wrote entries your build has never heard of — a colleague's branch, a rolled-back deployment, a file from a customer on the beta channel. Harmos decides what to do without consulting any catalog, using the Kind stamped on the envelope (chapter 8):

The entry is	Decision	Because
an unknown record	skipped	a record mutates nothing, so the state you rebuild is still correct
an unknown transaction	fatal refusal	later state depends on its mutation, so skipping it silently builds a wrong document

That table is why Kind is a stored field rather than something derived from the catalog. A runtime must be able to classify an entry it cannot decode, and "transaction or record" is exactly the one bit needed to choose between skipping and refusing.

The same principle governs versions of definitions you do know. A stored "net.rename" at version 5, in a build whose current version is 2, is refused loudly. An older runtime never skips a higher version and never tries to interpret it: the entry means something this build has no implementation for, and the only honest answers are "I applied it" or "I cannot open this."

Common Mistakes

Assuming a tightened rule is retroactive. v2's check forbids two symbols sharing a reference designator. A v1 file contains a duplicate. That file opens fine — replay does not call check — and the rebuilt state contains the duplicate. So v2's check must be written to tolerate finding what it refuses to create. Checks stop new violations; they do not erase old ones. Cleaning up existing data is a new transaction the user or a job commits, appended to history like any other change, and never a rewrite of what is already there.

Reaching for a migration script. There is no journal migration step in harmos, and adding one by hand is the most expensive mistake available in this chapter. See question 2.

Versioning something that is not durable. Projections, snapshots that are still caches, undo stacks, replica states in clients — all of these are folds, rebuilt from the entry order on demand. A schema change there is not a migration; it is a rebuild. The versioned-evolution obligation applies to durable truth: stored entries, exported artifacts, and a snapshot that has outlived the entries beneath it.

Retiring a Version

history(...) grows over the life of an application, and nothing about that is free — every historical implementation is code you keep compiling and testing. So versions do retire, but only under one condition:

A historical version may be retired only when no stored entry of that version can still appear. In practice that means below the snapshot horizon: the position at which retention has pruned the entries beneath a snapshot, leaving the snapshot as the only record of that prefix.

The reasoning comes straight from chapter 9's two lives of a snapshot. While the entries under a snapshot are retained, replay may reach any of them, so every version they contain must still have an implementation. Once retention prunes beneath the snapshot, those entries are gone; nothing above the horizon carries the retired version, and the implementation has no possible caller.

The test to apply, in one line: could a stored entry of this version still appear above the horizon? If yes, it stays in history(...) — no matter how old it is, and no matter how confident anyone is that "nobody has v1 files anymore." One customer's archived project is a counterexample that arrives as a support ticket.

Note what the snapshot inherits at that same moment. Below the horizon it was a disposable cache; above it, it is the only surviving record of the prefix, and it acquires the full versioned obligation of durable data — the same TryFrom grammar the records above use.

Why This Matters

A file written by your first release still opens in the build you ship five years from now, and you can point at the machinery instead of hoping. Every stored entry carries the id and version it was written with; dispatch runs the implementation that was current then, kept in the same file as the current one; and replay never calls check, so no rule you add later can retroactively invalidate a customer's history. Journals and documents from every version of your app keep opening, and the compiler tells you when you are about to break that.

Test Your Knowledge

1. Reviewing your v2 branch, a colleague spots that RenameNetV1 still has a check nobody calls. “Either replay should run it — that is what it is for — or delete it. Dead code is dead code.”

Both halves are wrong, and for different reasons.

Replay must not run it. The v1 entry was already accepted, by v1's own check, against the state that existed then. Running any check during replay reintroduces the failure chapter 2 ruled out: a validation that refuses mid-replay leaves only two options, abort the load or build state without that mutation, and both are corruption of a file that was never invalid. History is settled. check governs what may be committed now.

Nor is it dead code. It is the other half of a trait implementation — RenameNetV1 is a Transaction, and a Transaction that had no check would be a different kind of thing. More usefully, it is testable: you can assert exactly what v1 accepted and refused, in a unit test, years after v1 shipped and without a runtime. When a customer asks how a particular entry ever got into their file, that test is the answer.

The kernel of it: keeping the historical check costs a few lines and documents the rules of a past release; calling it during replay would make old files fail to open.

2. A teammate proposes a load-time migration: “walk the stored entries once, rewrite every v1 net.rename into the v2 shape, write the journal back, and delete RenameNetV1. One pass and the history problem is gone forever.”

This converts a solved problem into an unsolvable one.

Start with what the rewrite claims. A v1 entry says the user typed DATA[3] and the editor applied it as literal text. The v2 shape says DATA[3] is bus DATA, member 3, and renaming it renames the siblings. Rewriting the entry asserts the user did the second thing. They did not — that feature did not exist. This is exactly the upcast that dispatch refuses to perform: (id, version) selects the implementation that was current when the entry was written, precisely so an old entry keeps its old meaning.

It also breaks things downstream that read stored entries rather than state. Undo inverses are never persisted; they are re-derived by folding the entry order and calling each declared change's own invert against the pre-state (chapter 7). Change the stored payload and you have silently changed derived undo history. Any projection or per-file history folded from the order is affected the same way.

And the migration has no failure story. Half-rewritten journals, a crash mid-pass, a bug found after the pass has run on ten thousand customer machines — none of it is recoverable, because the original entries are gone. Compare the mechanism harmos ships: the old entries stay exactly as written, RenameNetV1::apply is thirty lines in the same file as v2, and a bug in it is a normal fix in a normal release.

Journals append. The only correct way to change what state says is a new transaction on the end of history — visible, undoable, attributable to whoever ran it.

3. A customer on the beta channel sends a bug report with their project attached. Your stable build opens it. The file contains: a drc.completed record your build has never heard of, and net.merge transaction entries it has never heard of, and net.rename entries at version 5 when your current version is 2. Walk through each.

Three different answers, and the runtime reaches all of them without consulting the catalog for the first two.

The unknown record is skipped. Kind on the envelope says Record, and a record mutates nothing. Skipping it leaves the rebuilt state exactly correct — the only loss is that a consumer which would have reacted to drc.completed does not, and your build has no such consumer by definition. The entry itself is untouched on disk.

The unknown transaction is a fatal refusal. Kind says Transaction, so later state depends on its mutation. There is no safe way to continue: skipping it produces a document that is silently wrong, and wrong-but-open is worse than refused, because the user will save on top of it. The file does not open.

The known id at version 5 is also refused, loudly. Your build knows "net.rename" but has no implementation for what version 5 means. An older runtime never guesses from the nearest version and never skips forward. There is no downgrade path — a v5 entry cannot be expressed in v2's terms, and pretending otherwise would fabricate the user's history.

The rule underneath all three: harmos would rather refuse a file than open a version of it that never existed.

4. history(...) for net.rename has carried RenameNetV1 for three years. Telemetry says no v1 entry has been loaded in eighteen months. Delete it?

Not on that evidence. Telemetry reports what has been loaded, and the question is what could still be loaded.

The condition for retiring a historical version is that no stored entry of it can still appear — in practice, that it sits below the snapshot horizon, where retention has pruned the entries beneath a snapshot and left the snapshot as the only record of that prefix. Above that horizon, stored entries are still reachable by replay, and any one of them may be a v1 net.rename.

Eighteen quiet months do not establish that. An archived project on a customer's NAS, a file in a compliance vault, a sidecar in a VM image somebody suspends and resumes next quarter — each is a v1 entry that has not been loaded yet. Delete the implementation and the next one of them turns into an unsupported-version load error: a file that opened yesterday and refuses today, which is the exact failure this whole chapter exists to prevent.

The test is structural, not statistical: could a stored entry of this version still appear above the horizon? If yes, it stays. And note what changes at the moment the answer becomes no — the snapshot that outlived those entries has stopped being a disposable cache and become truth, so it now carries the versioned-evolution obligation that the pruned entries used to carry for it.