docs: how the ensemble performs external I/O without a leader
Build Debug APK / build (push) Successful in 22m54s

The ensemble is meant to behave as one computer distributed across many
nodes. That abstraction holds inside its own boundary, where every merge
strategy is commutative and nothing needs a coordinator, and breaks the
moment the ensemble reaches outward: one computer implies one HTTP
request, many leaderless nodes implies N.

Leader election is disqualified rather than merely complex. The ensemble
must be fully functional with exactly one node awake — that is the
common case and the product — and no quorum-based mechanism can make
progress there. Coordination therefore may only ever suppress redundant
work; if it is wrong, disconnected, or disabled, the outcome must still
be correct, just more expensive.

That leaves capability (which nodes may do this at all — static,
declarative, missing from the codebase entirely) and selection (which of
them does it this time). Selection lands as a pure local function over a
synced NodeProfile document and rendezvous hashing, needing no new
protocol: gossip already re-announces every 60s and writes last_seen_at
to PeerNodeStore, a liveness signal the module comment anticipates and
nothing currently uses.

Answers docs/rss-reader.md §10. Proposed, not built; step 3 of §7 puts
it before Feeds phase 3, not before Feeds itself, since deterministic
item ids mean coordination can arrive later with no migration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WEEHetSTuEH5amjXVKPVn
This commit is contained in:
Greg ShuflinandClaude Opus 5 committed 2026-08-01 23:10:46 -07:00
1 parent d084ace601
commit 81e0c011ff
2 files changed
+414 -6

No files matched your search

+404
View File
@@ -0,0 +1,404 @@
# External effects: how one computer made of many nodes talks to the outside world
## Status
Proposed, and deliberately ahead of its first consumer. `docs/rss-reader.md`
§10 raised "where should refresh actually run?" as an open question; this
document is the answer to the general form of it, which every agent that
touches the network will hit.
## The problem
The ensemble is meant to behave as **one computer distributed across many
nodes**. Inside its own boundary that abstraction holds up well: the document
model is convergent, every merge strategy is commutative, and any node can
write anything at any time. Nothing in the sync plane needs a coordinator.
The abstraction breaks the moment the ensemble reaches *outward*. An HTTP GET
to a feed, a post to Mastodon, an OAuth refresh — these happen on exactly one
node, consume that node's bandwidth and battery, and are visible to a third
party who has no idea they're talking to a distributed system. "One computer"
implies one request. "Many nodes, none in charge" implies N.
The instinct is to reach for leader election. That instinct is wrong here, and
it's worth being precise about why before anything else, because it constrains
every option below.
### The constraint that rules out consensus
**The ensemble must be fully functional when exactly one node is awake.**
This is not a nice-to-have; it is the product. A personal ensemble is a phone,
a laptop, maybe a VPS, maybe a desktop that's off most of the week. The common
case is that one device is on. A user opening the reader on their phone at
7am, with everything else asleep, must get fresh articles.
Every quorum-based algorithm — Raft, Paxos, any lease system backed by a
majority — cannot make progress without a quorum, and a 2-node ensemble with
one node awake has no majority. Adopting one would mean the system stops
working in its most common state. So:
> **Rule 1.** No mechanism in this area may require more than one node to be
> reachable in order to make progress.
That single rule collapses the design space. What's left is not "elect a
leader"; it's "let everyone act, and make acting redundantly cheap or
harmless." Coordination becomes an *optimization*, never a *correctness*
requirement. Which turns out to be a much nicer place to be.
> **Rule 2.** Coordination may only ever suppress redundant work. If the
> coordination layer is wrong, disconnected, or disabled, the outcome must
> still be correct — just more expensive.
---
## 1. "Who does it" is the wrong first question
The question splits cleanly in two, and conflating them is what makes it look
hard:
- **(A) Which nodes *can* do this at all?** Mostly static. Does this node hold
the credential? Is it on a metered connection? Is it on the LAN the target
lives on? Is it always-on? Has the user forbidden it?
- **(B) Among those that can, which one *does* this time?** Dynamic, and only
interesting when (A) leaves more than one candidate.
(A) is missing from this codebase entirely, is purely declarative, needs no
consensus, and — in practice — usually answers (B) by leaving exactly one
candidate. It is both the easier half and the more valuable half. Build it
first.
## 2. Not all external effects are the same
The right mechanism depends entirely on what happens when the effect runs
twice. Four classes, in increasing order of difficulty:
### Class 1 — Convergent reads
Polling a feed. Refreshing a podcast's episode list. Re-fetching a URL into a
blob. Checking a watched folder.
Running twice is **wasteful, never wrong**, provided the write path is
convergent. `docs/rss-reader.md` §3.3 already makes it convergent by deriving
item document ids as `uuidv5(feed_id, guid)`, so two nodes fetching the same
feed produce byte-identical documents that merge into one.
**This is the class that matters right now, and it needs no coordination for
correctness at all.** Coordination here buys bandwidth, battery, and not
hammering someone's server — real wins, but optimizations.
### Class 2 — Reads with far-end side effects
Fetching mail from a POP3 server that deletes on read. Marking a notification
seen as a side effect of listing it. Consuming from a queue.
Running twice may lose data at the far end. Sometimes an API offers an
idempotent variant; when it does, this reduces to class 1, and taking that
option is almost always worth a little extra client code.
### Class 3 — External writes
Posting to Mastodon/BlueSky. Sending an email. Pushing playback position to a
third-party service.
Running twice is **wrong and user-visible**. This is the only class that
genuinely wants single-execution, and §6 argues it should be modeled as a
durable intent rather than as an action, which changes the problem
substantially.
### Class 4 — Forced placement
The API key exists on one node. The target is only reachable from the home LAN.
The node is the only one with a static IP.
There is no choice to make — (A) has exactly one answer, and the whole problem
disappears into the capability model.
**The taxonomy is the design.** A single global "who does external I/O" answer
would be wrong for most of these. What generalizes is the *classification plus
capability model*, not one selection algorithm.
---
## 3. What the ensemble already has
More than expected, and one piece of it is built and entirely unused:
- **A live liveness signal.** `p2p/gossip.rs` broadcasts a `NodeAnnouncement`
every 60s (`REANNOUNCE_INTERVAL`), and every receipt upserts the peer with a
fresh `last_seen_at` into `PeerNodeStore`. The module's own comment says it
"acts as a liveness signal for future presence-tracking features." That
future is this document. **No new protocol is needed to know who is awake.**
- **A persistent peer registry.** `PeerNodeStore` survives restarts, so
liveness has history, not just a live view.
- **A convergent, synced state plane.** Documents/properties/artifacts, with
`LastWriteWins` doing the right thing for "when did this last happen."
- **Deterministic document identity.** `uuidv5` from a normalized key
(`music::artist_document_id`, ADR 0018's `agent_settings_document_id`) —
the trick that lets independent nodes converge with zero coordination.
- **Node-to-node command.** `p2p/node_admin.rs`, whose doc is explicit that
"any of *your* nodes may command any other, there is no hierarchy among
moons." Worth honoring: this design introduces roles, not rank.
- **Local job dedup.** `JobRunner` dedups by `(JobKind, JobSubject)` — but
only within one node. It is the right shape to extend conceptually, not the
right thing to make distributed.
What's missing: **any notion of what a node is or what it's for.** `Settings`
(per-node, never synced) has `node_name`, theme, log verbosity, sync policy.
`PeerNode` records what gossip announced. Nothing anywhere says "this is the
always-on one" or "this one is on cellular."
---
## 4. (A) The capability model: a synced `NodeProfile` document
Give every node a document describing itself, at a deterministic id derived
from its pubkey — ADR 0018's recipe, generalized from *per agent* to *per
node*:
```rust
/// uuidv5(NAMESPACE_URL, "syn:node-profile:v1")
const NODE_PROFILE_NS: Uuid = /* ... */;
pub fn node_profile_document_id(node: &NodePubkey) -> DocumentID {
DocumentID::from_uuid_bytes(
*Uuid::new_v5(&NODE_PROFILE_NS, node.as_bytes()).as_bytes()
)
}
```
```
NodeProfile {
properties:
title: String [LastWriteWins] -- Title kind; the node's display name
node_pubkey: String [Immutable] -- NodeProfilePubkey; hex, so it's queryable
always_on: bool [LastWriteWins] -- NodeAlwaysOn; "expected to be reachable"
metered: bool [LastWriteWins] -- NodeMetered; cellular/quota'd
external_fetch: bool [LastWriteWins] -- NodeExternalFetch; may reach the public internet
on the ensemble's behalf. Default true.
}
```
`DocumentType::NodeProfile`, `is_derived() == true` — here that flag is
genuinely right, for the same three reasons ADR 0018 gives for
`AgentSettings`: hidden from document listings, never purged (a tombstone on a
deterministic id would poison that node's identity ensemble-wide), and
reproducible.
Design notes:
- **Synced, not gossiped.** Gossip announcements are ephemeral and explicitly
unverified display metadata; a node that is asleep has no announcement, but
its profile is exactly what you need while it's asleep. The document plane
gives persistence, history, conflict resolution, and a UI for free.
- **Any node may edit any profile**, per node_admin's stated trust model. Being
able to mark the VPS always-on from the phone is the point — it is the same
"configure the VPS from the phone" workflow `docs/sync-management.md` already
calls the flagship.
- **Facts and policy in one place, deliberately.** `metered` is a fact about
the node; `external_fetch` is a policy the user sets. Both are LWW booleans
on the same document, and separating them would buy nothing.
- **Start with three booleans, resist more.** Per-agent capability flags
(`fetch_feeds`, `send_mail`, …) are the obvious next request and should be
refused until a second consumer exists — `AgentSettings` is the right home
for them if they're ever needed, keyed by agent rather than bloating the node
profile.
- **Credentials are not in here.** A `NodeProfile` is plaintext synced
metadata. "Which node holds credential X" is class-4 placement and belongs
with whatever vault-scoped mechanism eventually stores the credential.
The Ensemble agent's existing node list is where this gets edited — it already
lists nodes and already has a detail surface.
---
## 5. (B) Selection: rendezvous hashing over the live, eligible set
For class 1, with (A) narrowing the field, the selection rule can be *entirely
local computation* — no messages, no claims, no protocol:
```rust
/// Whether this node should perform `task_key` right now.
///
/// Pure function of local state: the synced NodeProfiles, PeerNodeStore's
/// last_seen_at, and the task's own synced "last completed" timestamp.
/// Never blocks, never does I/O beyond the local store.
pub fn should_run_here(
store: &Store,
me: NodePubkey,
task_key: &str,
last_completed: Option<Timestamp>,
interval: Duration,
now: Timestamp,
) -> bool {
let eligible = live_eligible_nodes(store, now); // profile.external_fetch && seen recently
// Nobody else can — including "I'm the only one awake". Rule 1.
if eligible.is_empty() { return true; }
// Rendezvous (highest-random-weight): every node computes the same owner
// from the same set, with no communication. Adding or removing a node
// reassigns only 1/N of tasks.
let owner = eligible.iter()
.max_by_key(|n| blake3::hash(&[task_key.as_bytes(), n.as_bytes()].concat()))
.copied();
if owner == Some(me) { return true; }
if !eligible.contains(&me) { return false; }
// Overdue fallback: the owner isn't doing its job (asleep, throttled,
// crashed, or we simply disagree about who it is). Anyone eligible may
// step in. This is what makes a wrong `owner` cost latency, not
// correctness — Rule 2.
match last_completed {
None => true,
Some(t) => now - t > interval * OVERDUE_GRACE,
}
}
```
Why this shape:
- **Zero new state and zero new messages.** Liveness comes from
`PeerNodeStore.last_seen_at` (§3). Eligibility comes from synced
`NodeProfile`s. And `last_completed` is already there for real tasks —
Feeds' `FeedLastFetchedAt` is exactly it, an LWW synced property the design
needed anyway.
- **Per-task rather than global ownership.** Hashing the task key spreads 200
feeds across the awake nodes instead of dumping all of them on one. A single
global owner would have marginally better per-host HTTP cache behavior; that
does not outweigh handing one device the whole load. (Global ownership is the
degenerate case — hash a constant — if that ever proves wrong.)
- **Disagreement is safe by construction.** Two nodes with different views of
liveness may both think they own a feed (both fetch: wasteful, convergent,
fine) or both think a third node owns it (neither fetches: caught by the
overdue fallback within `OVERDUE_GRACE`).
### The one real tuning hazard
`last_completed` propagates by document sync, not instantly. If `OVERDUE_GRACE`
is shorter than the worst-case sync latency, every node independently concludes
the owner is delinquent and they all fetch — the coordination silently degrades
to "everyone does it." Not incorrect, but it quietly buys nothing.
So `OVERDUE_GRACE * interval` must comfortably exceed the sync period. With a
30-minute feed interval and a grace of 3, that's a 90-minute window against a
sync scheduler running far more often than that — a wide margin, which is the
right posture for a heuristic.
### Clock skew
Everything here uses wall-clock `Timestamp`s, as `LastWriteWins` already does.
Skew shifts when the fallback fires; it never makes an outcome wrong. That's
the property to preserve deliberately:
> **Rule 3.** Clock skew, liveness error, and sync lag may only affect *how
> often* an external effect happens — never *whether the ensemble ends up in
> the right state*.
### What was considered and rejected
- **Claim-then-act soft leases** (write a claim document, act if yours wins).
Sync latency vastly exceeds the operation's duration, so a claim usually
arrives after the work is done. LWW on unsynchronized wall clocks picks an
arbitrary winner. Strictly more machinery than §5 for strictly worse
behavior.
- **Raft/Paxos/quorum leases.** Rule 1.
- **A user-designated fixed node.** Genuinely defensible — it's simple, and it
is what §5 degenerates to when only one node has `external_fetch`. It is
rejected as the *primary* mechanism only because its failure mode (designated
node offline ⇒ silently nothing happens) requires an overdue fallback anyway,
and once that exists, rendezvous hashing is one additional line and strictly
better. The user-designated case remains fully expressible.
---
## 6. Class 3: model outbound writes as intents, not actions
For "post this", "send this", "pay this", §5's overdue fallback is exactly
wrong — duplicate execution is the failure being avoided.
The right move is to stop treating the external write as an action taken by
whichever node the user happened to touch, and treat it as a **durable intent
that syncs, with execution deliberately decoupled from creation**:
1. Composing creates a document — an intent — with its content and target. It
syncs like everything else. The user's phone can go into a tunnel
immediately afterward with nothing lost.
2. Execution is gated by `should_run_here` **with no overdue fallback**, or
with a very long one. The default is to wait for the right node.
3. The intent's document id is passed to the external service as an
**idempotency key** whenever the API supports one (most modern ones do). When
it does, duplicate execution stops being harmful and the whole thing
collapses back to class 1 — worth actively checking for on any new
integration.
4. Completion is recorded as an artifact on the intent, which syncs and
suppresses re-execution everywhere.
This is an email outbox, and it has the same honest tradeoff: if the executing
node is offline for a week, the post goes out a week late. Nothing can fix that
without either running the effect from a node that is awake (which is what
eligibility already expresses) or accepting duplicates.
It is also the version of "one computer" that actually delivers the promise:
compose on the phone, send from the always-on node, see the result everywhere.
And it needs no new infrastructure — an intent is a document, and this codebase
is already very good at documents.
**Nothing in class 3 is being built now.** It is written down so the first
agent that needs it doesn't invent a worse thing under deadline, and so §4/§5
are built general enough to serve it.
---
## 7. What to build, and when
The whole coordination layer is roughly: one `DocumentType`, four
`PropertyKind`s, one pure function, and an Ensemble-screen editor. Small. But
it should not block the reader.
| Step | Content | Blocks |
|---|---|---|
| 1 | This document reviewed and settled. | Nothing |
| 2 | Feeds phases 0–2 (`docs/rss-reader.md`): parse, data model, GUI. Single-node-useful, no scheduling, so "who fetches" never arises. | Nothing |
| 3 | `NodeProfile` (§4) + `should_run_here` (§5) + Ensemble-screen editing, with unit tests over a synthetic peer set. | — |
| 4 | Feeds phase 3 wires `refresh_due_feeds` through `should_run_here`, keyed `feeds:refresh:<feed_id>`, with `FeedLastFetchedAt` as `last_completed`. | Needs 3 |
Step 3 before step 4, not before step 2. Building the coordination layer
against a real consumer is how it stays the right size; building it first, in
the abstract, is how it grows a per-agent capability matrix nobody needed.
The one thing that genuinely must be settled *before* writing Feeds code is
already settled: item identity (`docs/rss-reader.md` §3.3). Because item ids
are deterministic, "everyone fetches" and "one node fetches" produce identical
data. Coordination can therefore be added later without a migration, which is
precisely why it's safe to defer it to step 3.
## 8. Testing
`should_run_here` is a pure function of local state, which makes it unusually
testable for something in this area — no network, no async, no timing:
- One eligible node ⇒ true, regardless of whether it's the hash winner.
- Zero eligible nodes ⇒ true (Rule 1: the last node awake always acts).
- Node not eligible, task badly overdue ⇒ still false (an ineligible node never
becomes eligible through neglect — the metered phone stays off the hook).
- All nodes agree on the owner given the same peer set — asserted over many
random sets, since this is the property the whole scheme rests on.
- Removing one node from an N-node set reassigns ≈1/N of tasks, not most of
them (the rendezvous property; catches a regression to naive modulo hashing).
- Overdue fallback fires after `OVERDUE_GRACE * interval` and not before.
## 9. Open questions
- **Should liveness be sharper than 60s gossip re-announcement?** A node that
crashes looks alive for up to `LIVENESS_WINDOW`. For 30-minute feed polling
that's irrelevant; for anything interactive it wouldn't be. Deliberately not
solved until something needs it.
- **Does `external_fetch` want to be per-agent?** §4 argues no until a second
consumer exists. The second consumer is what should settle it, not
speculation.
- **Is there a role for `node_admin` here?** "Refresh feeds now, on that node"
as an `AdminRequest` is a natural fit with the existing protocol and would
make the Ensemble screen a genuine control panel. Additive; not needed for
any of the above.
+10 -6
View File
@@ -689,12 +689,16 @@ makes it pleasant, phase 4 what makes it an ensemble feature in practice.
§3.4 argues yes, but not via `is_derived`. Wants a small `SearchParams`
change; decide when the item count first gets uncomfortable rather than
pre-emptively.
- **Where should refresh actually run in a multi-node ensemble?** Every node
polling every feed is N× the requests for identical results, and §3.3 makes it
*correct* but not *efficient*. The natural answer is "the always-on node
does it and the others just sync", which needs a notion of node roles that
doesn't exist yet. Deliberately unresolved; the identity scheme means solving
it later changes no data.
- ~~**Where should refresh actually run in a multi-node ensemble?**~~ Answered
separately in `docs/external-effects.md`, which generalizes the question past
Feeds to every agent that touches the network. Short version: because §3.3
makes item identity deterministic, "every node fetches" and "one node
fetches" produce identical data, so coordination is a pure optimization that
can be added without migrating anything. Feeds phase 3 should route
`refresh_due_feeds` through that document's `should_run_here`, keyed
`feeds:refresh:<feed_id>`, with `FeedLastFetchedAt` as the "last completed"
input — it needs no new state because the property already exists. Phases
0–2 are unaffected and need not wait.
- **Full-text extraction for summary-only feeds.** Would arrive as a
`PrimaryContent` blob on `FeedItem`, which flips `has_blob_content()` to true
and gives the type a sync-policy row. Noted so the eventual change is a known