docs: how the ensemble performs external I/O without a leader
Build Debug APK / build (push) Successful in 22m54s
Build Debug APK / build (push) Successful in 22m54s
The ensemble is meant to behave as one computer distributed across many nodes. That abstraction holds inside its own boundary, where every merge strategy is commutative and nothing needs a coordinator, and breaks the moment the ensemble reaches outward: one computer implies one HTTP request, many leaderless nodes implies N. Leader election is disqualified rather than merely complex. The ensemble must be fully functional with exactly one node awake — that is the common case and the product — and no quorum-based mechanism can make progress there. Coordination therefore may only ever suppress redundant work; if it is wrong, disconnected, or disabled, the outcome must still be correct, just more expensive. That leaves capability (which nodes may do this at all — static, declarative, missing from the codebase entirely) and selection (which of them does it this time). Selection lands as a pure local function over a synced NodeProfile document and rendezvous hashing, needing no new protocol: gossip already re-announces every 60s and writes last_seen_at to PeerNodeStore, a liveness signal the module comment anticipates and nothing currently uses. Answers docs/rss-reader.md §10. Proposed, not built; step 3 of §7 puts it before Feeds phase 3, not before Feeds itself, since deterministic item ids mean coordination can arrive later with no migration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WEEHetSTuEH5amjXVKPVn
This commit is contained in:
1 parent
d084ace601
commit
81e0c011ff
2 files changed
+414
-6
No files matched your search
@@ -0,0 +1,404 @@
|
||||
# External effects: how one computer made of many nodes talks to the outside world
|
||||
|
||||
## Status
|
||||
|
||||
Proposed, and deliberately ahead of its first consumer. `docs/rss-reader.md`
|
||||
§10 raised "where should refresh actually run?" as an open question; this
|
||||
document is the answer to the general form of it, which every agent that
|
||||
touches the network will hit.
|
||||
|
||||
## The problem
|
||||
|
||||
The ensemble is meant to behave as **one computer distributed across many
|
||||
nodes**. Inside its own boundary that abstraction holds up well: the document
|
||||
model is convergent, every merge strategy is commutative, and any node can
|
||||
write anything at any time. Nothing in the sync plane needs a coordinator.
|
||||
|
||||
The abstraction breaks the moment the ensemble reaches *outward*. An HTTP GET
|
||||
to a feed, a post to Mastodon, an OAuth refresh — these happen on exactly one
|
||||
node, consume that node's bandwidth and battery, and are visible to a third
|
||||
party who has no idea they're talking to a distributed system. "One computer"
|
||||
implies one request. "Many nodes, none in charge" implies N.
|
||||
|
||||
The instinct is to reach for leader election. That instinct is wrong here, and
|
||||
it's worth being precise about why before anything else, because it constrains
|
||||
every option below.
|
||||
|
||||
### The constraint that rules out consensus
|
||||
|
||||
**The ensemble must be fully functional when exactly one node is awake.**
|
||||
|
||||
This is not a nice-to-have; it is the product. A personal ensemble is a phone,
|
||||
a laptop, maybe a VPS, maybe a desktop that's off most of the week. The common
|
||||
case is that one device is on. A user opening the reader on their phone at
|
||||
7am, with everything else asleep, must get fresh articles.
|
||||
|
||||
Every quorum-based algorithm — Raft, Paxos, any lease system backed by a
|
||||
majority — cannot make progress without a quorum, and a 2-node ensemble with
|
||||
one node awake has no majority. Adopting one would mean the system stops
|
||||
working in its most common state. So:
|
||||
|
||||
> **Rule 1.** No mechanism in this area may require more than one node to be
|
||||
> reachable in order to make progress.
|
||||
|
||||
That single rule collapses the design space. What's left is not "elect a
|
||||
leader"; it's "let everyone act, and make acting redundantly cheap or
|
||||
harmless." Coordination becomes an *optimization*, never a *correctness*
|
||||
requirement. Which turns out to be a much nicer place to be.
|
||||
|
||||
> **Rule 2.** Coordination may only ever suppress redundant work. If the
|
||||
> coordination layer is wrong, disconnected, or disabled, the outcome must
|
||||
> still be correct — just more expensive.
|
||||
|
||||
---
|
||||
|
||||
## 1. "Who does it" is the wrong first question
|
||||
|
||||
The question splits cleanly in two, and conflating them is what makes it look
|
||||
hard:
|
||||
|
||||
- **(A) Which nodes *can* do this at all?** Mostly static. Does this node hold
|
||||
the credential? Is it on a metered connection? Is it on the LAN the target
|
||||
lives on? Is it always-on? Has the user forbidden it?
|
||||
- **(B) Among those that can, which one *does* this time?** Dynamic, and only
|
||||
interesting when (A) leaves more than one candidate.
|
||||
|
||||
(A) is missing from this codebase entirely, is purely declarative, needs no
|
||||
consensus, and — in practice — usually answers (B) by leaving exactly one
|
||||
candidate. It is both the easier half and the more valuable half. Build it
|
||||
first.
|
||||
|
||||
## 2. Not all external effects are the same
|
||||
|
||||
The right mechanism depends entirely on what happens when the effect runs
|
||||
twice. Four classes, in increasing order of difficulty:
|
||||
|
||||
### Class 1 — Convergent reads
|
||||
|
||||
Polling a feed. Refreshing a podcast's episode list. Re-fetching a URL into a
|
||||
blob. Checking a watched folder.
|
||||
|
||||
Running twice is **wasteful, never wrong**, provided the write path is
|
||||
convergent. `docs/rss-reader.md` §3.3 already makes it convergent by deriving
|
||||
item document ids as `uuidv5(feed_id, guid)`, so two nodes fetching the same
|
||||
feed produce byte-identical documents that merge into one.
|
||||
|
||||
**This is the class that matters right now, and it needs no coordination for
|
||||
correctness at all.** Coordination here buys bandwidth, battery, and not
|
||||
hammering someone's server — real wins, but optimizations.
|
||||
|
||||
### Class 2 — Reads with far-end side effects
|
||||
|
||||
Fetching mail from a POP3 server that deletes on read. Marking a notification
|
||||
seen as a side effect of listing it. Consuming from a queue.
|
||||
|
||||
Running twice may lose data at the far end. Sometimes an API offers an
|
||||
idempotent variant; when it does, this reduces to class 1, and taking that
|
||||
option is almost always worth a little extra client code.
|
||||
|
||||
### Class 3 — External writes
|
||||
|
||||
Posting to Mastodon/BlueSky. Sending an email. Pushing playback position to a
|
||||
third-party service.
|
||||
|
||||
Running twice is **wrong and user-visible**. This is the only class that
|
||||
genuinely wants single-execution, and §6 argues it should be modeled as a
|
||||
durable intent rather than as an action, which changes the problem
|
||||
substantially.
|
||||
|
||||
### Class 4 — Forced placement
|
||||
|
||||
The API key exists on one node. The target is only reachable from the home LAN.
|
||||
The node is the only one with a static IP.
|
||||
|
||||
There is no choice to make — (A) has exactly one answer, and the whole problem
|
||||
disappears into the capability model.
|
||||
|
||||
**The taxonomy is the design.** A single global "who does external I/O" answer
|
||||
would be wrong for most of these. What generalizes is the *classification plus
|
||||
capability model*, not one selection algorithm.
|
||||
|
||||
---
|
||||
|
||||
## 3. What the ensemble already has
|
||||
|
||||
More than expected, and one piece of it is built and entirely unused:
|
||||
|
||||
- **A live liveness signal.** `p2p/gossip.rs` broadcasts a `NodeAnnouncement`
|
||||
every 60s (`REANNOUNCE_INTERVAL`), and every receipt upserts the peer with a
|
||||
fresh `last_seen_at` into `PeerNodeStore`. The module's own comment says it
|
||||
"acts as a liveness signal for future presence-tracking features." That
|
||||
future is this document. **No new protocol is needed to know who is awake.**
|
||||
- **A persistent peer registry.** `PeerNodeStore` survives restarts, so
|
||||
liveness has history, not just a live view.
|
||||
- **A convergent, synced state plane.** Documents/properties/artifacts, with
|
||||
`LastWriteWins` doing the right thing for "when did this last happen."
|
||||
- **Deterministic document identity.** `uuidv5` from a normalized key
|
||||
(`music::artist_document_id`, ADR 0018's `agent_settings_document_id`) —
|
||||
the trick that lets independent nodes converge with zero coordination.
|
||||
- **Node-to-node command.** `p2p/node_admin.rs`, whose doc is explicit that
|
||||
"any of *your* nodes may command any other, there is no hierarchy among
|
||||
moons." Worth honoring: this design introduces roles, not rank.
|
||||
- **Local job dedup.** `JobRunner` dedups by `(JobKind, JobSubject)` — but
|
||||
only within one node. It is the right shape to extend conceptually, not the
|
||||
right thing to make distributed.
|
||||
|
||||
What's missing: **any notion of what a node is or what it's for.** `Settings`
|
||||
(per-node, never synced) has `node_name`, theme, log verbosity, sync policy.
|
||||
`PeerNode` records what gossip announced. Nothing anywhere says "this is the
|
||||
always-on one" or "this one is on cellular."
|
||||
|
||||
---
|
||||
|
||||
## 4. (A) The capability model: a synced `NodeProfile` document
|
||||
|
||||
Give every node a document describing itself, at a deterministic id derived
|
||||
from its pubkey — ADR 0018's recipe, generalized from *per agent* to *per
|
||||
node*:
|
||||
|
||||
```rust
|
||||
/// uuidv5(NAMESPACE_URL, "syn:node-profile:v1")
|
||||
const NODE_PROFILE_NS: Uuid = /* ... */;
|
||||
|
||||
pub fn node_profile_document_id(node: &NodePubkey) -> DocumentID {
|
||||
DocumentID::from_uuid_bytes(
|
||||
*Uuid::new_v5(&NODE_PROFILE_NS, node.as_bytes()).as_bytes()
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
NodeProfile {
|
||||
properties:
|
||||
title: String [LastWriteWins] -- Title kind; the node's display name
|
||||
node_pubkey: String [Immutable] -- NodeProfilePubkey; hex, so it's queryable
|
||||
always_on: bool [LastWriteWins] -- NodeAlwaysOn; "expected to be reachable"
|
||||
metered: bool [LastWriteWins] -- NodeMetered; cellular/quota'd
|
||||
external_fetch: bool [LastWriteWins] -- NodeExternalFetch; may reach the public internet
|
||||
on the ensemble's behalf. Default true.
|
||||
}
|
||||
```
|
||||
|
||||
`DocumentType::NodeProfile`, `is_derived() == true` — here that flag is
|
||||
genuinely right, for the same three reasons ADR 0018 gives for
|
||||
`AgentSettings`: hidden from document listings, never purged (a tombstone on a
|
||||
deterministic id would poison that node's identity ensemble-wide), and
|
||||
reproducible.
|
||||
|
||||
Design notes:
|
||||
|
||||
- **Synced, not gossiped.** Gossip announcements are ephemeral and explicitly
|
||||
unverified display metadata; a node that is asleep has no announcement, but
|
||||
its profile is exactly what you need while it's asleep. The document plane
|
||||
gives persistence, history, conflict resolution, and a UI for free.
|
||||
- **Any node may edit any profile**, per node_admin's stated trust model. Being
|
||||
able to mark the VPS always-on from the phone is the point — it is the same
|
||||
"configure the VPS from the phone" workflow `docs/sync-management.md` already
|
||||
calls the flagship.
|
||||
- **Facts and policy in one place, deliberately.** `metered` is a fact about
|
||||
the node; `external_fetch` is a policy the user sets. Both are LWW booleans
|
||||
on the same document, and separating them would buy nothing.
|
||||
- **Start with three booleans, resist more.** Per-agent capability flags
|
||||
(`fetch_feeds`, `send_mail`, …) are the obvious next request and should be
|
||||
refused until a second consumer exists — `AgentSettings` is the right home
|
||||
for them if they're ever needed, keyed by agent rather than bloating the node
|
||||
profile.
|
||||
- **Credentials are not in here.** A `NodeProfile` is plaintext synced
|
||||
metadata. "Which node holds credential X" is class-4 placement and belongs
|
||||
with whatever vault-scoped mechanism eventually stores the credential.
|
||||
|
||||
The Ensemble agent's existing node list is where this gets edited — it already
|
||||
lists nodes and already has a detail surface.
|
||||
|
||||
---
|
||||
|
||||
## 5. (B) Selection: rendezvous hashing over the live, eligible set
|
||||
|
||||
For class 1, with (A) narrowing the field, the selection rule can be *entirely
|
||||
local computation* — no messages, no claims, no protocol:
|
||||
|
||||
```rust
|
||||
/// Whether this node should perform `task_key` right now.
|
||||
///
|
||||
/// Pure function of local state: the synced NodeProfiles, PeerNodeStore's
|
||||
/// last_seen_at, and the task's own synced "last completed" timestamp.
|
||||
/// Never blocks, never does I/O beyond the local store.
|
||||
pub fn should_run_here(
|
||||
store: &Store,
|
||||
me: NodePubkey,
|
||||
task_key: &str,
|
||||
last_completed: Option<Timestamp>,
|
||||
interval: Duration,
|
||||
now: Timestamp,
|
||||
) -> bool {
|
||||
let eligible = live_eligible_nodes(store, now); // profile.external_fetch && seen recently
|
||||
|
||||
// Nobody else can — including "I'm the only one awake". Rule 1.
|
||||
if eligible.is_empty() { return true; }
|
||||
|
||||
// Rendezvous (highest-random-weight): every node computes the same owner
|
||||
// from the same set, with no communication. Adding or removing a node
|
||||
// reassigns only 1/N of tasks.
|
||||
let owner = eligible.iter()
|
||||
.max_by_key(|n| blake3::hash(&[task_key.as_bytes(), n.as_bytes()].concat()))
|
||||
.copied();
|
||||
|
||||
if owner == Some(me) { return true; }
|
||||
if !eligible.contains(&me) { return false; }
|
||||
|
||||
// Overdue fallback: the owner isn't doing its job (asleep, throttled,
|
||||
// crashed, or we simply disagree about who it is). Anyone eligible may
|
||||
// step in. This is what makes a wrong `owner` cost latency, not
|
||||
// correctness — Rule 2.
|
||||
match last_completed {
|
||||
None => true,
|
||||
Some(t) => now - t > interval * OVERDUE_GRACE,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Why this shape:
|
||||
|
||||
- **Zero new state and zero new messages.** Liveness comes from
|
||||
`PeerNodeStore.last_seen_at` (§3). Eligibility comes from synced
|
||||
`NodeProfile`s. And `last_completed` is already there for real tasks —
|
||||
Feeds' `FeedLastFetchedAt` is exactly it, an LWW synced property the design
|
||||
needed anyway.
|
||||
- **Per-task rather than global ownership.** Hashing the task key spreads 200
|
||||
feeds across the awake nodes instead of dumping all of them on one. A single
|
||||
global owner would have marginally better per-host HTTP cache behavior; that
|
||||
does not outweigh handing one device the whole load. (Global ownership is the
|
||||
degenerate case — hash a constant — if that ever proves wrong.)
|
||||
- **Disagreement is safe by construction.** Two nodes with different views of
|
||||
liveness may both think they own a feed (both fetch: wasteful, convergent,
|
||||
fine) or both think a third node owns it (neither fetches: caught by the
|
||||
overdue fallback within `OVERDUE_GRACE`).
|
||||
|
||||
### The one real tuning hazard
|
||||
|
||||
`last_completed` propagates by document sync, not instantly. If `OVERDUE_GRACE`
|
||||
is shorter than the worst-case sync latency, every node independently concludes
|
||||
the owner is delinquent and they all fetch — the coordination silently degrades
|
||||
to "everyone does it." Not incorrect, but it quietly buys nothing.
|
||||
|
||||
So `OVERDUE_GRACE * interval` must comfortably exceed the sync period. With a
|
||||
30-minute feed interval and a grace of 3, that's a 90-minute window against a
|
||||
sync scheduler running far more often than that — a wide margin, which is the
|
||||
right posture for a heuristic.
|
||||
|
||||
### Clock skew
|
||||
|
||||
Everything here uses wall-clock `Timestamp`s, as `LastWriteWins` already does.
|
||||
Skew shifts when the fallback fires; it never makes an outcome wrong. That's
|
||||
the property to preserve deliberately:
|
||||
|
||||
> **Rule 3.** Clock skew, liveness error, and sync lag may only affect *how
|
||||
> often* an external effect happens — never *whether the ensemble ends up in
|
||||
> the right state*.
|
||||
|
||||
### What was considered and rejected
|
||||
|
||||
- **Claim-then-act soft leases** (write a claim document, act if yours wins).
|
||||
Sync latency vastly exceeds the operation's duration, so a claim usually
|
||||
arrives after the work is done. LWW on unsynchronized wall clocks picks an
|
||||
arbitrary winner. Strictly more machinery than §5 for strictly worse
|
||||
behavior.
|
||||
- **Raft/Paxos/quorum leases.** Rule 1.
|
||||
- **A user-designated fixed node.** Genuinely defensible — it's simple, and it
|
||||
is what §5 degenerates to when only one node has `external_fetch`. It is
|
||||
rejected as the *primary* mechanism only because its failure mode (designated
|
||||
node offline ⇒ silently nothing happens) requires an overdue fallback anyway,
|
||||
and once that exists, rendezvous hashing is one additional line and strictly
|
||||
better. The user-designated case remains fully expressible.
|
||||
|
||||
---
|
||||
|
||||
## 6. Class 3: model outbound writes as intents, not actions
|
||||
|
||||
For "post this", "send this", "pay this", §5's overdue fallback is exactly
|
||||
wrong — duplicate execution is the failure being avoided.
|
||||
|
||||
The right move is to stop treating the external write as an action taken by
|
||||
whichever node the user happened to touch, and treat it as a **durable intent
|
||||
that syncs, with execution deliberately decoupled from creation**:
|
||||
|
||||
1. Composing creates a document — an intent — with its content and target. It
|
||||
syncs like everything else. The user's phone can go into a tunnel
|
||||
immediately afterward with nothing lost.
|
||||
2. Execution is gated by `should_run_here` **with no overdue fallback**, or
|
||||
with a very long one. The default is to wait for the right node.
|
||||
3. The intent's document id is passed to the external service as an
|
||||
**idempotency key** whenever the API supports one (most modern ones do). When
|
||||
it does, duplicate execution stops being harmful and the whole thing
|
||||
collapses back to class 1 — worth actively checking for on any new
|
||||
integration.
|
||||
4. Completion is recorded as an artifact on the intent, which syncs and
|
||||
suppresses re-execution everywhere.
|
||||
|
||||
This is an email outbox, and it has the same honest tradeoff: if the executing
|
||||
node is offline for a week, the post goes out a week late. Nothing can fix that
|
||||
without either running the effect from a node that is awake (which is what
|
||||
eligibility already expresses) or accepting duplicates.
|
||||
|
||||
It is also the version of "one computer" that actually delivers the promise:
|
||||
compose on the phone, send from the always-on node, see the result everywhere.
|
||||
And it needs no new infrastructure — an intent is a document, and this codebase
|
||||
is already very good at documents.
|
||||
|
||||
**Nothing in class 3 is being built now.** It is written down so the first
|
||||
agent that needs it doesn't invent a worse thing under deadline, and so §4/§5
|
||||
are built general enough to serve it.
|
||||
|
||||
---
|
||||
|
||||
## 7. What to build, and when
|
||||
|
||||
The whole coordination layer is roughly: one `DocumentType`, four
|
||||
`PropertyKind`s, one pure function, and an Ensemble-screen editor. Small. But
|
||||
it should not block the reader.
|
||||
|
||||
| Step | Content | Blocks |
|
||||
|---|---|---|
|
||||
| 1 | This document reviewed and settled. | Nothing |
|
||||
| 2 | Feeds phases 0–2 (`docs/rss-reader.md`): parse, data model, GUI. Single-node-useful, no scheduling, so "who fetches" never arises. | Nothing |
|
||||
| 3 | `NodeProfile` (§4) + `should_run_here` (§5) + Ensemble-screen editing, with unit tests over a synthetic peer set. | — |
|
||||
| 4 | Feeds phase 3 wires `refresh_due_feeds` through `should_run_here`, keyed `feeds:refresh:<feed_id>`, with `FeedLastFetchedAt` as `last_completed`. | Needs 3 |
|
||||
|
||||
Step 3 before step 4, not before step 2. Building the coordination layer
|
||||
against a real consumer is how it stays the right size; building it first, in
|
||||
the abstract, is how it grows a per-agent capability matrix nobody needed.
|
||||
|
||||
The one thing that genuinely must be settled *before* writing Feeds code is
|
||||
already settled: item identity (`docs/rss-reader.md` §3.3). Because item ids
|
||||
are deterministic, "everyone fetches" and "one node fetches" produce identical
|
||||
data. Coordination can therefore be added later without a migration, which is
|
||||
precisely why it's safe to defer it to step 3.
|
||||
|
||||
## 8. Testing
|
||||
|
||||
`should_run_here` is a pure function of local state, which makes it unusually
|
||||
testable for something in this area — no network, no async, no timing:
|
||||
|
||||
- One eligible node ⇒ true, regardless of whether it's the hash winner.
|
||||
- Zero eligible nodes ⇒ true (Rule 1: the last node awake always acts).
|
||||
- Node not eligible, task badly overdue ⇒ still false (an ineligible node never
|
||||
becomes eligible through neglect — the metered phone stays off the hook).
|
||||
- All nodes agree on the owner given the same peer set — asserted over many
|
||||
random sets, since this is the property the whole scheme rests on.
|
||||
- Removing one node from an N-node set reassigns ≈1/N of tasks, not most of
|
||||
them (the rendezvous property; catches a regression to naive modulo hashing).
|
||||
- Overdue fallback fires after `OVERDUE_GRACE * interval` and not before.
|
||||
|
||||
## 9. Open questions
|
||||
|
||||
- **Should liveness be sharper than 60s gossip re-announcement?** A node that
|
||||
crashes looks alive for up to `LIVENESS_WINDOW`. For 30-minute feed polling
|
||||
that's irrelevant; for anything interactive it wouldn't be. Deliberately not
|
||||
solved until something needs it.
|
||||
- **Does `external_fetch` want to be per-agent?** §4 argues no until a second
|
||||
consumer exists. The second consumer is what should settle it, not
|
||||
speculation.
|
||||
- **Is there a role for `node_admin` here?** "Refresh feeds now, on that node"
|
||||
as an `AdminRequest` is a natural fit with the existing protocol and would
|
||||
make the Ensemble screen a genuine control panel. Additive; not needed for
|
||||
any of the above.
|
||||
+10
-6
@@ -689,12 +689,16 @@ makes it pleasant, phase 4 what makes it an ensemble feature in practice.
|
||||
§3.4 argues yes, but not via `is_derived`. Wants a small `SearchParams`
|
||||
change; decide when the item count first gets uncomfortable rather than
|
||||
pre-emptively.
|
||||
- **Where should refresh actually run in a multi-node ensemble?** Every node
|
||||
polling every feed is N× the requests for identical results, and §3.3 makes it
|
||||
*correct* but not *efficient*. The natural answer is "the always-on node
|
||||
does it and the others just sync", which needs a notion of node roles that
|
||||
doesn't exist yet. Deliberately unresolved; the identity scheme means solving
|
||||
it later changes no data.
|
||||
- ~~**Where should refresh actually run in a multi-node ensemble?**~~ Answered
|
||||
separately in `docs/external-effects.md`, which generalizes the question past
|
||||
Feeds to every agent that touches the network. Short version: because §3.3
|
||||
makes item identity deterministic, "every node fetches" and "one node
|
||||
fetches" produce identical data, so coordination is a pure optimization that
|
||||
can be added without migrating anything. Feeds phase 3 should route
|
||||
`refresh_due_feeds` through that document's `should_run_here`, keyed
|
||||
`feeds:refresh:<feed_id>`, with `FeedLastFetchedAt` as the "last completed"
|
||||
input — it needs no new state because the property already exists. Phases
|
||||
0–2 are unaffected and need not wait.
|
||||
- **Full-text extraction for summary-only feeds.** Would arrive as a
|
||||
`PrimaryContent` blob on `FeedItem`, which flips `has_blob_content()` to true
|
||||
and gives the type a sync-policy row. Noted so the eventual change is a known
|
||||
|
||||
Reference in new issue
Block a user