Isolate a Sidecar

The Program You Do Not Trust

Chapter 11 gave work two shapes. A Job is finite and answers; a Service is resident and is supervised. Both run inside your process, which is exactly the problem when the work is a screen recorder, a codec, a vendor SDK, or anything else whose bad afternoon takes the whole application with it.

A sidecar is that program moved out of your address space and kept under the same supervision it would have had inside it. The author embeds a restart policy in #[harmos::sidecar]; the host validates, caps, and installs it into the same resident supervision used for in-process services.

Everything below is one facility. The design behind it — the wire, the exits, and the decisions that were closed — is The Sidecar Facility.

Declaring One

Two halves are yours: how the child is reached, and what to do with what it says unasked.

use std::time::Duration;

use harmos::{Executable, Notice, Service, Supervised};

let recorder = Supervised::new(
    Executable::at("/opt/acme/bin/recorder")?
        .argument("--codec=h264")
        .variable("PATH", "/usr/bin:/bin"),
    |context: &Service<Editor>, notice: Notice| {
        if let Some(captured) = notice.take::<Captured>() {
            context.streams().publish(Frames { count: captured.frames });
        }
    },
);

let runtime = builder.runtime().await?;
let calls = recorder.install(&runtime.work)?;

Executable::at takes an absolute path and refuses right there — at the line that declares it — if nothing is at it. The refusal is Incompatible, the same one start answers with, so the ? above carries it straight out of your assembly function. A typo is a build that does not stand up, not a service that deactivates four restarts later. The environment is cleared, the three streams are private pipes, and the child dies with the attempt that started it.

A Supervised is both the resident the supervisor owns and the handle you call it with, and it is Clone — a clone addresses whichever attempt is currently live and refuses when none is. That is what decides where a bridge is declared.

Declare it into the assembly, and keep a clone of the handle in Resources, whenever the caller is a job:

// resources.rs — `connect` builds the handle and keeps a clone of it.
let sidecar = services::probe::attach(config)?;
let resources = Resources { probe: Some(sidecar.clone()), .. };

// app.rs — `launch` hands the original to the assembly.
builder((), resources)
    .service_with(services::probe::NAME, supervision, sidecar)
    .runtime()
    .await?

Boot then hands back a runtime whose every attempt reaches the routes through scope.resources::<Bench>(). Nobody passes a handle around, no shell holds one, and a restart changes nothing: the clone in Resources addresses the new attempt the moment the supervisor opens it.

Install it after boot — recorder.install(&runtime.work)?, as above — only when the caller is a shell rather than a job, because then the shell is the one thing that must physically hold the handle.

minibench-app is the first shape. minibench-api's probe/ declares the edge's payloads — the settings an attempt is opened with, and the request and answer one sample exchanges — and that crate is all the supervised executable compiles, so the two halves cannot come to disagree about them. minibench-app's services/probe.rs is the host half: the name, the route descriptor, the notice handler, and the attach that builds the declaration launch hands to service_with. Its shell calls app::launch(&config) and then submits jobs::Sample, and never sees a route handle at all.

A publication is the one thing the api crate does not carry. A stream row's window and grouping are the host's retention policy, so Reading is declared in minibench-app, and the bridge declares its own row in its own crate — which is what examples/standalone-sidecar does too, because a sidecar with no application to share a row with must still be able to declare one. What the two sides share is the id and the schema, and ingest refuses a publication whose schema does not match the row it is decoding into. The example asserts the agreement twice over rather than describing it: the walkthrough checks the inspected executable's declared (id, schema, window) against the application's own Reading constants before boot, and probe-sidecar/tests/metadata.rs checks the same triple from the other side.

The file's restart(...) declaration is the single source of truth either way, and the host caps excessive credit and timing values before turning it into a service policy. install does that conversion for you from the declaration it already read; a build-time attachment does it where it reads the declaration — minibench-app's services::probe::attach hands launch the capped ServicePolicy beside the sidecar. Use restart = "never" for a process that must not be relaunched.

Routes Are Declarations, Not Strings

One type is one operation, and it has two halves. Call is the transport-neutral half — the method both ends speak, the answer type, and the bound a call of it is held to — and it is all a caller can have, because the handler lives in another executable. Route is the served half, and it lives in the sidecar beside the sidecar it answers for. The payload is protobuf, the same wire format the framing around it already uses. Authors write #[harmos::message]; Harmos derives tags from field names at compile time:

use harmos::sidecar::Call;

#[harmos::message]
#[derive(Clone, PartialEq)]
pub struct StartRecording {
    pub output: String,
}

impl Call for StartRecording {
    type Request = Self;
    const METHOD: &'static str = "recorder.start";
    const BOUND: Duration = Duration::from_secs(2);

    type Reply = ();
}

#[harmos::route] over the sidecar's own impl block writes both halves at once; the hand-written form above is what a host-side caller writes when the sidecar it calls is a binary it did not build.

Calling it is one line, and the answer comes back typed:

calls.call::<StartRecording>(&StartRecording { output: capture_path }).await?;

There is no route table to register it in and no manifest to list it in. The declaration site and the serving site both name StartRecording, so the two cannot drift — which is the failure this shape exists to make unrepresentable.

A call refuses four ways, and each is a different next move. Running means the attempt is executing, not that the child has finished its handshake, so the first call after a restart may refuse once — retry it:

SidecarError	What happened	Your move
Unavailable	no attempt is live, or one has not opened yet	retry
Deadline	the route's deadline elapsed	the child is alive and stuck
Refused { code, message }	the child or transport said no	branch on the stable code
Malformed(_)	a payload left the contract	fix the types on one side

Budget the Complete Route Frame

Frame::request_fits and Frame::reply_fits size a typed route payload inside the complete Harmos protobuf frame. Their corresponding request_encoded_len and reply_encoded_len methods return the exact budgeted length. Route code cannot know the runtime call id, so the public calculation uses u64::MAX, the exact widest id the envelope can carry:

use harmos::sidecar::Frame;

let next = page_with_one_more_item();
if Frame::reply_fits::<PollOutput>(&next) {
    reply = next;
}

This matters even when every application item has already been chunked. Sixteen 1 MiB chunks batched into one reply are still one route answer, and their protobuf fields plus the Harmos answer and outer envelopes exceed a 16 MiB frame. Page or batch by the complete typed reply budget, not only by item count or each item's encoded length.

An actual oversized request is refused locally before it enters Tonic. An oversized answer is replaced by a small refusal so the exchange stays live. Both use the stable Frame::OVERSIZED code. The 16 MiB limit is inclusive: a complete frame whose encoded length equals Frame::LIMIT fits.

Writing the Other Half

harmos::sidecar is what the child program names, through the same harmos dependency the host declares with features = ["sidecar"]. It is seven names, because the wire is the runtime's:

use harmos::sidecar::{Refusal, Routes, Supervisor, serve};

#[tokio::main]
async fn main() -> std::io::Result<()> {
    serve(
        Recorder::default(),
        Routes::new()
            .on::<StartRecording, _>(|recorder, host, start| {
                recorder.open(&start.output).map_err(Refusal::of)?;
                recorder.report_on(host.clone());
                Ok(())
            })
            .on::<StopRecording, _>(|recorder, _host, _stop| {
                recorder.close().map_err(Refusal::of)
            }),
    )
    .await
}

The declared form is one trait and catalogs of types.

#[harmos::sidecar(
    id = "recorder",
    streams(Captured),
    routes(routes::Routes),
    restart(credit = 3, sustained_health_ms = 30_000, backoff_base_ms = 250, backoff_cap_ms = 10_000),
)]
struct Recorder { path: PathBuf }

impl harmos::sidecar::Sidecar for Recorder {
    type Config = Settings;
    type Error = Refusal;
    type Resources = Resources;

    async fn initialize(config: Settings, _context: &Context) -> Result<Self, Refusal> {
        Ok(Self { path: config.path })
    }

    fn resources(&self) -> Resources { Resources { path: self.path.clone() } }
}

#[harmos::sidecar] on the struct owns identity, restart policy, the rows the sidecar publishes by hand, and the catalogs naming what it contributes — all literal data a host reads before anything runs. The trait owns everything typed: the configuration, the error every contribution answers, and the bundle they read. Its binary entry point is only harmos::sidecar::run::<Recorder>().await; the wire underneath is the one above.

Static registration belongs to a separate harmos::sidecar::Plugin implementation. Its register hook defaults to empty and runs before initialization. Embedded metadata inspection never executes that hook.

Routes live in their own files, one per route, and the catalog lists them:

src/routes/mod.rs

#[harmos::routes]
pub enum Routes { Echo }

// src/routes/echo.rs
pub struct Echo;

#[harmos::route(id = "recorder.echo")]
impl Echo {
    async fn handle(recorder: &mut Recorder, request: String) -> Result<String, Refusal> {
        let _ = recorder;
        Ok(request)
    }
}

Splitting them across files is free, because the catalog lists types rather than functions. The route type is what the caller names: calls.call::<routes::Echo>(&request).await. The payload can belong to another SDK, and two routes may share one — the declaration is the route rather than the payload. Synchronous guest route functions are still free functions taking only the request, and compose with echo::serve(table): a guest has no live state to hand a handler and no catalog of types to list one in.

Sample a Stream

A stream's producer is a Sampler: an instance with a lifecycle, not a function.

#[harmos::sampler(interval_ms = 250)]
impl Level {
    async fn start(resources: &Resources) -> Result<Self, Refusal> {
        Ok(Self { meter: resources.meter.clone() })
    }

    async fn sample(&mut self, tick: &Tick) -> Result<LevelRow, Refusal> {
        Ok(LevelRow { elapsed_s: tick.elapsed().as_secs_f64(), db: self.meter.read().await? })
    }
}

It starts on the first lease of its stream and stops after the last release, so nothing measures while nothing is reading. State lives on the struct, which is what a poller plus a watch channel used to stand in for; a stateless sampler is a unit struct, and that is the whole cost of the shape. The row sample answers declares the stream, so the sampler names neither an id nor a label, and samplers(...) on the sidecar attribute carries its publication grant.

The host takes the lease off its own consumer count: a row it ingests with one watcher is a leased stream, and a row it never ingests is never leased. Cadence belongs to the sampler — every leaseholder shares one instance and one tick, so their rows agree — and a sample that overruns skips only its own ticks.

Every instance is cancelled and joined before cleanup, so nothing a sampler reads is released while it is still reading it.

Standard output is the protocol. Anything you want a human to read goes to standard error, and arrives at your host handler as Notice::Logged — route it into your own logging, which the runtime deliberately knows nothing about.

One call is served at a time, so the recorder's actual capture loop lives on its own task and reports through Supervisor:

host.push::<Captured>(&Captured { frames: 120 });

A push waits for nothing and is answered by nothing. It crosses the same serialized stream an answer does, so telemetry never has to wait for the loop to be between calls — which is the whole reason start can return immediately.

The host end of that trade is the other way round: it takes arrivals before it takes your queued calls, so under a flood of telemetry a call waits its turn. That is the right priority — the reverse would starve every answer — but it is worth knowing when a route's deadline is tight.

Bulk Output Does Not Ride the Wire

The recorder writes video to the path you handed it. The protocol carries control and telemetry; the file is a file. When the capture finishes, what crosses is a push saying so — not the megabytes.

The Three Ways It Ends

A sidecar has exactly three endings, and it chooses between two of them by saying something or saying nothing:

host.exiting("the recording is complete");  // a decision
host.abandon();                             // an outage

The ending	The regime	What the runtime does
the host asked, and it went	Ok(())	ServiceStatus::Stopped
it announced its exit	Fault::Application	stop, and record why
registration or initialization was refused	Fault::Application	stop without retrying
it stopped saying anything	Fault::Environmental	spend a credit, start a fresh one

Nothing here reads an exit code. A process killed by a signal and a process that returned zero both simply stop, and only the program knows which it meant — so it says, or it does not, and an ending nobody announced is an outage. That is what makes a crashed recorder restart and a finished one deactivate, with the same three lines of supervision chapter 11 already gave you.

Stopping One That Will Not Stop

runtime.stop() asks each service to finish and waits. For a sidecar the asks are harmos.interrupt followed by harmos.cleanup — which quiesces every running sampler before the sidecar's own cleanup — and the waiting has a deadline that lives inside the attempt:

Supervised::new(attach, notices).grace(Duration::from_millis(500))

Within the grace the child may finish its file, flush, and exit on its own. Past it, it is ended. The deadline has to live here rather than in stop, because stop has none of its own — a child that ignores cleanup would otherwise hold your whole exit open for as long as it liked.

Testing It Without a Process

The transport is injectable, and the double is not a mock: above the byte halves it runs the identical framing, handshake, demultiplexer, and exit classification the real thing does.

use harmos::sidecar::{Local, Routes};

let attach = Local::new(|| (Recorder::fake(), recorder_routes()));
let recorder = Supervised::new(attach, notices);

A fresh sidecar is built per attempt, so a test of a restart is a test of a restart. And because the resident is Clone, it declares into a simulation like any other resident — sim.fault("recorder", Fault::Environmental("the device went away")) scripts a sidecar failure through the simulation seam of chapter 17, deterministically, with no process anywhere.

What the double cannot stand in for is exactly what a process adds: a spawn that fails, a diagnostic stream, and an ending that needs a signal. Keep one test against a real program for those.

What You Have Now

Chapter 11's work layer supervises a program in another address space, with one declaration, one handle, one route contract, and no second lifecycle vocabulary.

Chapter 18 counts the surface. This facility adds eight names to it — Supervised, Attach, Attached, Executable, Call, Frame, Notice, and SidecarError — and not one of them is a supervision verb, because the supervision was already there.

Declare Publications

A reusable #[harmos::stream] row owns its identity, schema, and window. Publication permission belongs to the executable that explicitly composes it:

#[harmos::sidecar(id = "probe", restart = "never", streams(Reading))]
struct Probe;

A sampled stream needs no entry there: the sampler that owns its row is already its declaration, and samplers(Catalog) carries the grant.

Guest declarations use the same streams(Reading, ...) argument. Omitting it means no publications. Registering a row in an application stream catalog or linking its library grants no executable permission. Inert inspection selects exactly one complete publication inventory for the executable's facility and owner. Native ingress rejects an ungranted stream or wrong schema before delivering bytes to typed ingestion or the notice callback; guest publication capabilities check the same owner-qualified list.

An in-process Local transport declares its grant with .publishing(...); Declared::local uses the generated declaration. Handwritten SDK compositions can use inventory::publications to encode existing row constants, without running registration or factories during inspection.