The ensemble is meant to behave as one computer distributed across many nodes. That abstraction holds inside its own boundary, where every merge strategy is commutative and nothing needs a coordinator, and breaks the moment the ensemble reaches outward: one computer implies one HTTP request, many leaderless nodes implies N. Leader election is disqualified rather than merely complex. The ensemble must be fully functional with exactly one node awake — that is the common case and the product — and no quorum-based mechanism can make progress there. Coordination therefore may only ever suppress redundant work; if it is wrong, disconnected, or disabled, the outcome must still be correct, just more expensive. That leaves capability (which nodes may do this at all — static, declarative, missing from the codebase entirely) and selection (which of them does it this time). Selection lands as a pure local function over a synced NodeProfile document and rendezvous hashing, needing no new protocol: gossip already re-announces every 60s and writes last_seen_at to PeerNodeStore, a liveness signal the module comment anticipates and nothing currently uses. Answers docs/rss-reader.md §10. Proposed, not built; step 3 of §7 puts it before Feeds phase 3, not before Feeds itself, since deterministic item ids mean coordination can arrive later with no migration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WEEHetSTuEH5amjXVKPVn
19 KiB
External effects: how one computer made of many nodes talks to the outside world
Status
Proposed, and deliberately ahead of its first consumer. docs/rss-reader.md
§10 raised "where should refresh actually run?" as an open question; this
document is the answer to the general form of it, which every agent that
touches the network will hit.
The problem
The ensemble is meant to behave as one computer distributed across many nodes. Inside its own boundary that abstraction holds up well: the document model is convergent, every merge strategy is commutative, and any node can write anything at any time. Nothing in the sync plane needs a coordinator.
The abstraction breaks the moment the ensemble reaches outward. An HTTP GET to a feed, a post to Mastodon, an OAuth refresh — these happen on exactly one node, consume that node's bandwidth and battery, and are visible to a third party who has no idea they're talking to a distributed system. "One computer" implies one request. "Many nodes, none in charge" implies N.
The instinct is to reach for leader election. That instinct is wrong here, and it's worth being precise about why before anything else, because it constrains every option below.
The constraint that rules out consensus
The ensemble must be fully functional when exactly one node is awake.
This is not a nice-to-have; it is the product. A personal ensemble is a phone, a laptop, maybe a VPS, maybe a desktop that's off most of the week. The common case is that one device is on. A user opening the reader on their phone at 7am, with everything else asleep, must get fresh articles.
Every quorum-based algorithm — Raft, Paxos, any lease system backed by a majority — cannot make progress without a quorum, and a 2-node ensemble with one node awake has no majority. Adopting one would mean the system stops working in its most common state. So:
Rule 1. No mechanism in this area may require more than one node to be reachable in order to make progress.
That single rule collapses the design space. What's left is not "elect a leader"; it's "let everyone act, and make acting redundantly cheap or harmless." Coordination becomes an optimization, never a correctness requirement. Which turns out to be a much nicer place to be.
Rule 2. Coordination may only ever suppress redundant work. If the coordination layer is wrong, disconnected, or disabled, the outcome must still be correct — just more expensive.
1. "Who does it" is the wrong first question
The question splits cleanly in two, and conflating them is what makes it look hard:
- (A) Which nodes can do this at all? Mostly static. Does this node hold the credential? Is it on a metered connection? Is it on the LAN the target lives on? Is it always-on? Has the user forbidden it?
- (B) Among those that can, which one does this time? Dynamic, and only interesting when (A) leaves more than one candidate.
(A) is missing from this codebase entirely, is purely declarative, needs no consensus, and — in practice — usually answers (B) by leaving exactly one candidate. It is both the easier half and the more valuable half. Build it first.
2. Not all external effects are the same
The right mechanism depends entirely on what happens when the effect runs twice. Four classes, in increasing order of difficulty:
Class 1 — Convergent reads
Polling a feed. Refreshing a podcast's episode list. Re-fetching a URL into a blob. Checking a watched folder.
Running twice is wasteful, never wrong, provided the write path is
convergent. docs/rss-reader.md §3.3 already makes it convergent by deriving
item document ids as uuidv5(feed_id, guid), so two nodes fetching the same
feed produce byte-identical documents that merge into one.
This is the class that matters right now, and it needs no coordination for correctness at all. Coordination here buys bandwidth, battery, and not hammering someone's server — real wins, but optimizations.
Class 2 — Reads with far-end side effects
Fetching mail from a POP3 server that deletes on read. Marking a notification seen as a side effect of listing it. Consuming from a queue.
Running twice may lose data at the far end. Sometimes an API offers an idempotent variant; when it does, this reduces to class 1, and taking that option is almost always worth a little extra client code.
Class 3 — External writes
Posting to Mastodon/BlueSky. Sending an email. Pushing playback position to a third-party service.
Running twice is wrong and user-visible. This is the only class that genuinely wants single-execution, and §6 argues it should be modeled as a durable intent rather than as an action, which changes the problem substantially.
Class 4 — Forced placement
The API key exists on one node. The target is only reachable from the home LAN. The node is the only one with a static IP.
There is no choice to make — (A) has exactly one answer, and the whole problem disappears into the capability model.
The taxonomy is the design. A single global "who does external I/O" answer would be wrong for most of these. What generalizes is the classification plus capability model, not one selection algorithm.
3. What the ensemble already has
More than expected, and one piece of it is built and entirely unused:
- A live liveness signal.
p2p/gossip.rsbroadcasts aNodeAnnouncementevery 60s (REANNOUNCE_INTERVAL), and every receipt upserts the peer with a freshlast_seen_atintoPeerNodeStore. The module's own comment says it "acts as a liveness signal for future presence-tracking features." That future is this document. No new protocol is needed to know who is awake. - A persistent peer registry.
PeerNodeStoresurvives restarts, so liveness has history, not just a live view. - A convergent, synced state plane. Documents/properties/artifacts, with
LastWriteWinsdoing the right thing for "when did this last happen." - Deterministic document identity.
uuidv5from a normalized key (music::artist_document_id, ADR 0018'sagent_settings_document_id) — the trick that lets independent nodes converge with zero coordination. - Node-to-node command.
p2p/node_admin.rs, whose doc is explicit that "any of your nodes may command any other, there is no hierarchy among moons." Worth honoring: this design introduces roles, not rank. - Local job dedup.
JobRunnerdedups by(JobKind, JobSubject)— but only within one node. It is the right shape to extend conceptually, not the right thing to make distributed.
What's missing: any notion of what a node is or what it's for. Settings
(per-node, never synced) has node_name, theme, log verbosity, sync policy.
PeerNode records what gossip announced. Nothing anywhere says "this is the
always-on one" or "this one is on cellular."
4. (A) The capability model: a synced NodeProfile document
Give every node a document describing itself, at a deterministic id derived from its pubkey — ADR 0018's recipe, generalized from per agent to per node:
/// uuidv5(NAMESPACE_URL, "syn:node-profile:v1")
const NODE_PROFILE_NS: Uuid = /* ... */;
pub fn node_profile_document_id(node: &NodePubkey) -> DocumentID {
DocumentID::from_uuid_bytes(
*Uuid::new_v5(&NODE_PROFILE_NS, node.as_bytes()).as_bytes()
)
}
NodeProfile {
properties:
title: String [LastWriteWins] -- Title kind; the node's display name
node_pubkey: String [Immutable] -- NodeProfilePubkey; hex, so it's queryable
always_on: bool [LastWriteWins] -- NodeAlwaysOn; "expected to be reachable"
metered: bool [LastWriteWins] -- NodeMetered; cellular/quota'd
external_fetch: bool [LastWriteWins] -- NodeExternalFetch; may reach the public internet
on the ensemble's behalf. Default true.
}
DocumentType::NodeProfile, is_derived() == true — here that flag is
genuinely right, for the same three reasons ADR 0018 gives for
AgentSettings: hidden from document listings, never purged (a tombstone on a
deterministic id would poison that node's identity ensemble-wide), and
reproducible.
Design notes:
- Synced, not gossiped. Gossip announcements are ephemeral and explicitly unverified display metadata; a node that is asleep has no announcement, but its profile is exactly what you need while it's asleep. The document plane gives persistence, history, conflict resolution, and a UI for free.
- Any node may edit any profile, per node_admin's stated trust model. Being
able to mark the VPS always-on from the phone is the point — it is the same
"configure the VPS from the phone" workflow
docs/sync-management.mdalready calls the flagship. - Facts and policy in one place, deliberately.
meteredis a fact about the node;external_fetchis a policy the user sets. Both are LWW booleans on the same document, and separating them would buy nothing. - Start with three booleans, resist more. Per-agent capability flags
(
fetch_feeds,send_mail, …) are the obvious next request and should be refused until a second consumer exists —AgentSettingsis the right home for them if they're ever needed, keyed by agent rather than bloating the node profile. - Credentials are not in here. A
NodeProfileis plaintext synced metadata. "Which node holds credential X" is class-4 placement and belongs with whatever vault-scoped mechanism eventually stores the credential.
The Ensemble agent's existing node list is where this gets edited — it already lists nodes and already has a detail surface.
5. (B) Selection: rendezvous hashing over the live, eligible set
For class 1, with (A) narrowing the field, the selection rule can be entirely local computation — no messages, no claims, no protocol:
/// Whether this node should perform `task_key` right now.
///
/// Pure function of local state: the synced NodeProfiles, PeerNodeStore's
/// last_seen_at, and the task's own synced "last completed" timestamp.
/// Never blocks, never does I/O beyond the local store.
pub fn should_run_here(
store: &Store,
me: NodePubkey,
task_key: &str,
last_completed: Option<Timestamp>,
interval: Duration,
now: Timestamp,
) -> bool {
let eligible = live_eligible_nodes(store, now); // profile.external_fetch && seen recently
// Nobody else can — including "I'm the only one awake". Rule 1.
if eligible.is_empty() { return true; }
// Rendezvous (highest-random-weight): every node computes the same owner
// from the same set, with no communication. Adding or removing a node
// reassigns only 1/N of tasks.
let owner = eligible.iter()
.max_by_key(|n| blake3::hash(&[task_key.as_bytes(), n.as_bytes()].concat()))
.copied();
if owner == Some(me) { return true; }
if !eligible.contains(&me) { return false; }
// Overdue fallback: the owner isn't doing its job (asleep, throttled,
// crashed, or we simply disagree about who it is). Anyone eligible may
// step in. This is what makes a wrong `owner` cost latency, not
// correctness — Rule 2.
match last_completed {
None => true,
Some(t) => now - t > interval * OVERDUE_GRACE,
}
}
Why this shape:
- Zero new state and zero new messages. Liveness comes from
PeerNodeStore.last_seen_at(§3). Eligibility comes from syncedNodeProfiles. Andlast_completedis already there for real tasks — Feeds'FeedLastFetchedAtis exactly it, an LWW synced property the design needed anyway. - Per-task rather than global ownership. Hashing the task key spreads 200 feeds across the awake nodes instead of dumping all of them on one. A single global owner would have marginally better per-host HTTP cache behavior; that does not outweigh handing one device the whole load. (Global ownership is the degenerate case — hash a constant — if that ever proves wrong.)
- Disagreement is safe by construction. Two nodes with different views of
liveness may both think they own a feed (both fetch: wasteful, convergent,
fine) or both think a third node owns it (neither fetches: caught by the
overdue fallback within
OVERDUE_GRACE).
The one real tuning hazard
last_completed propagates by document sync, not instantly. If OVERDUE_GRACE
is shorter than the worst-case sync latency, every node independently concludes
the owner is delinquent and they all fetch — the coordination silently degrades
to "everyone does it." Not incorrect, but it quietly buys nothing.
So OVERDUE_GRACE * interval must comfortably exceed the sync period. With a
30-minute feed interval and a grace of 3, that's a 90-minute window against a
sync scheduler running far more often than that — a wide margin, which is the
right posture for a heuristic.
Clock skew
Everything here uses wall-clock Timestamps, as LastWriteWins already does.
Skew shifts when the fallback fires; it never makes an outcome wrong. That's
the property to preserve deliberately:
Rule 3. Clock skew, liveness error, and sync lag may only affect how often an external effect happens — never whether the ensemble ends up in the right state.
What was considered and rejected
- Claim-then-act soft leases (write a claim document, act if yours wins). Sync latency vastly exceeds the operation's duration, so a claim usually arrives after the work is done. LWW on unsynchronized wall clocks picks an arbitrary winner. Strictly more machinery than §5 for strictly worse behavior.
- Raft/Paxos/quorum leases. Rule 1.
- A user-designated fixed node. Genuinely defensible — it's simple, and it
is what §5 degenerates to when only one node has
external_fetch. It is rejected as the primary mechanism only because its failure mode (designated node offline ⇒ silently nothing happens) requires an overdue fallback anyway, and once that exists, rendezvous hashing is one additional line and strictly better. The user-designated case remains fully expressible.
6. Class 3: model outbound writes as intents, not actions
For "post this", "send this", "pay this", §5's overdue fallback is exactly wrong — duplicate execution is the failure being avoided.
The right move is to stop treating the external write as an action taken by whichever node the user happened to touch, and treat it as a durable intent that syncs, with execution deliberately decoupled from creation:
- Composing creates a document — an intent — with its content and target. It syncs like everything else. The user's phone can go into a tunnel immediately afterward with nothing lost.
- Execution is gated by
should_run_herewith no overdue fallback, or with a very long one. The default is to wait for the right node. - The intent's document id is passed to the external service as an idempotency key whenever the API supports one (most modern ones do). When it does, duplicate execution stops being harmful and the whole thing collapses back to class 1 — worth actively checking for on any new integration.
- Completion is recorded as an artifact on the intent, which syncs and suppresses re-execution everywhere.
This is an email outbox, and it has the same honest tradeoff: if the executing node is offline for a week, the post goes out a week late. Nothing can fix that without either running the effect from a node that is awake (which is what eligibility already expresses) or accepting duplicates.
It is also the version of "one computer" that actually delivers the promise: compose on the phone, send from the always-on node, see the result everywhere. And it needs no new infrastructure — an intent is a document, and this codebase is already very good at documents.
Nothing in class 3 is being built now. It is written down so the first agent that needs it doesn't invent a worse thing under deadline, and so §4/§5 are built general enough to serve it.
7. What to build, and when
The whole coordination layer is roughly: one DocumentType, four
PropertyKinds, one pure function, and an Ensemble-screen editor. Small. But
it should not block the reader.
| Step | Content | Blocks |
|---|---|---|
| 1 | This document reviewed and settled. | Nothing |
| 2 | Feeds phases 0–2 (docs/rss-reader.md): parse, data model, GUI. Single-node-useful, no scheduling, so "who fetches" never arises. |
Nothing |
| 3 | NodeProfile (§4) + should_run_here (§5) + Ensemble-screen editing, with unit tests over a synthetic peer set. |
— |
| 4 | Feeds phase 3 wires refresh_due_feeds through should_run_here, keyed feeds:refresh:<feed_id>, with FeedLastFetchedAt as last_completed. |
Needs 3 |
Step 3 before step 4, not before step 2. Building the coordination layer against a real consumer is how it stays the right size; building it first, in the abstract, is how it grows a per-agent capability matrix nobody needed.
The one thing that genuinely must be settled before writing Feeds code is
already settled: item identity (docs/rss-reader.md §3.3). Because item ids
are deterministic, "everyone fetches" and "one node fetches" produce identical
data. Coordination can therefore be added later without a migration, which is
precisely why it's safe to defer it to step 3.
8. Testing
should_run_here is a pure function of local state, which makes it unusually
testable for something in this area — no network, no async, no timing:
- One eligible node ⇒ true, regardless of whether it's the hash winner.
- Zero eligible nodes ⇒ true (Rule 1: the last node awake always acts).
- Node not eligible, task badly overdue ⇒ still false (an ineligible node never becomes eligible through neglect — the metered phone stays off the hook).
- All nodes agree on the owner given the same peer set — asserted over many random sets, since this is the property the whole scheme rests on.
- Removing one node from an N-node set reassigns ≈1/N of tasks, not most of them (the rendezvous property; catches a regression to naive modulo hashing).
- Overdue fallback fires after
OVERDUE_GRACE * intervaland not before.
9. Open questions
- Should liveness be sharper than 60s gossip re-announcement? A node that
crashes looks alive for up to
LIVENESS_WINDOW. For 30-minute feed polling that's irrelevant; for anything interactive it wouldn't be. Deliberately not solved until something needs it. - Does
external_fetchwant to be per-agent? §4 argues no until a second consumer exists. The second consumer is what should settle it, not speculation. - Is there a role for
node_adminhere? "Refresh feeds now, on that node" as anAdminRequestis a natural fit with the existing protocol and would make the Ensemble screen a genuine control panel. Additive; not needed for any of the above.