darkfi/openspec/changes/evgr-resume-bug/design.md

Design: evgr-resume-bug

Context

The rotating event graph (src/event_graph/) keeps one DAG per hourly slot (hours_rotation: 1) with a 24-slot retention window (max_dags: 24). Three mechanisms populate a node's DAG:

1. Initial sync — EventGraph::sync_impl (via dag_sync): quorum-agreed tips → HeaderReq(our_tips) → header_dag_insert → fetch_missing_events (batched EventReq bodies). Strict: fails the round if any requested body fails to commit.
2. Live gossip — ProtocolEventGraph::handle_event_put (src/event_graph/proto.rs): validates one arriving event, then resolves its unknown ancestry with fetch_parents.
3. Paginated range — RangeReq/fetch_page_with_blobs. Exists, is served by peers, and is unused by bin/app.

The mobile app (bin/app/src/plugin/darkirc.rs) drives these as follows:

• The dag_sync plugin task performs the initial sync once per process lifetime, then parks forever in a loop that only re-notifies the UI of peer-count changes.
• catch_up_sync walks the remaining retention-window slots once, then breaks. After it exits, no task ever syncs a slot again — including slots that rotate in later (dag_prune_task rotates current_genesis hourly, but nothing reconciles the new slot beyond gossip).
• Screen off/on toggles P2P_OUTBOUND_SLEEP/P2P_OUTBOUND_ACTIVE only.

The bug this change fixes: after a suspension, catch-up relies exclusively on fetch_parents walks, which are best-effort, single-peer, and cover only the lineages they descend. Anything not in a walked lineage is silently missing forever (no reconciliation exists), and the user sees permanent holes in conversation history.

This is not mobile-only. The same permanent-gap symptom has been observed with laptop users who close the lid and later return: the OS freezes the whole process (no mobile-style sleep hook, and desktop builds have no screen_changed signal at all), sockets die, and on resume the client is in exactly the state described above — current tips eventually arrive via gossip, but mid-history branches are never fetched. Consequence for this design: an explicit wake signal is a useful additional trigger, but repair correctness cannot depend on one existing or firing. The peer-recovery and periodic triggers below are therefore mandatory parts of the mechanism, not hardening.

Forensic evidence

All log excerpts below are from an Android logcat capture of the app (taken 2026-08-31), sanitized: nicknames, channel names, message bodies, and seed addresses are replaced with placeholders. Event ids are truncated blake3 prefixes retained where needed to show DAG topology. Local times are UTC+2; event timestamps are UTC.

A. The suspension and resume

08-31 16:42:03.942 [10685] I net::refinery: No connections for 747s. GreylistRefinery paused.
08-31 16:42:03.957 [10685] D net::seedsync_session: SeedSyncSession::start_seed() [START]
08-31 16:42:03.957 [10685] I net::connector::connect: [P2P] Connecting peer [<seed-0>] via route [<seed-0>]
08-31 16:42:03.957 [10685] I net::connector::connect: [P2P] Connecting peer [<seed-1>] via route [<seed-1>]
08-31 16:42:09.268 [10685] I net::seedsync_session: [P2P] Connected seed [<seed-0>]
08-31 16:42:09.273 [10685] I net::seedsync_session: [P2P] Connected seed [<seed-1>]
08-31 16:42:09.405 [10685] I net::seedsync_session: [P2P] Disconnecting from seed [<seed-0>]
08-31 16:42:09.429 [10685] I net::seedsync_session: [P2P] Disconnecting from seed [<seed-1>]

747s without connections ⇒ last traffic at ~14:29:36 UTC. Notably, the entire rest of the capture contains zero plugin::darkirc2 INFO lines — the plugin's dag_sync task logs "Syncing newest event DAG...", "Newest event DAG synced successfully", etc. at info level on every sync attempt. Their absence proves no sync path ran at resume; it had completed at cold start (before the capture window) and was parked.

B. The holey burst

At 16:42:16 (7 seconds after the seeds reconnected — consistent with ~100 sequential ancestry round-trips at ~70ms), ~110 events were committed in a 56ms burst and relayed to the UI. Received layers:

165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182
183 184 185 186 187 188 189 190 191 192
----------------------------------------- GAP (17 layers missing) -----
210 211 212
--- GAP (8) --- 221 --- 222-223 missing --- 224 --- 225 missing ---
226 227
----------------------------------- GAP (18 layers missing) ----------
246 --- 247 missing --- 248 249 250 ... 262

Message timestamps show the gap is real conversation time:

layer=192 t=1788186785188  ->  14:33:05 UTC   (received)
layer=210 t=1788186941479  ->  14:35:41 UTC   (received)
                                 ~156 seconds of conversation never arrived

From layer 263 onward, events arrive one at a time via live gossip (16:42:17 … 16:58), i.e. the network path itself was healthy after reconnect.

The burst shape (single 56ms commit, layer-ascending) matches fetch_parents exactly: it buffers fetched events in a BTreeMap<u64 /*layer*/, Vec<…>> and inserts them flattened in layer order after the walk completes. One completed walk = one burst.

C. Parent-chain analysis (why the holes are where they are)

Parsing ev_id + parents[0] from every relayed event and checking whether each parent was itself ever relayed:

t=1788186785060  ev=6210eaa4  parent=0bbb2fc6   ok (received)
t=1788186785187  ev=ea7bc14a  parent=6210eaa4   ok (received)
t=1788186785188  ev=550bdc68  parent=ea7bc14a   ok (received)   <- L192
t=1788186941479  ev=4713d021  parent=5143546a   <<< NEVER RELAYED
t=1788186941573  ev=bab13f11  parent=4713d021   ok
t=1788186996457  ev=6545d7ff  parent=45dae6ba   <<< NEVER RELAYED
t=1788187005626  ev=79f7506f  parent=b715e06f   <<< NEVER RELAYED
t=1788187024199  ev=31d0461b  parent=44810e02   <<< NEVER RELAYED
t=1788187180594  ev=21522ad9  parent=97b2d5d6   <<< NEVER RELAYED
t=1788187203763  ev=baa051fd  parent=9ef0a12c   <<< NEVER RELAYED
...

The received set is closed under parents only via events that were either received or already local; the "never relayed" bridging parents are events whose bodies must be in main_tree (otherwise the children could not have committed — see dag_insert_inner's parent-body closure below), but which produced no relay line. They are foreign-channel/DM traffic: darkirc uses one shared DAG across all channels, and the app's relay_events silently drops undecryptable privmsgs. The walked lineage therefore weaves through other channels' events, and the missing conversation messages are events on branches no completed walk ever descended.

D. Ruled-out alternatives

• Hourly rotation prune: boundary at 14:00 UTC; the whole incident (14:29–14:42) is inside the 14:00–15:00 slot; max_dags: 24 retention. Also handle_event_put's pre-genesis cut (event.header.timestamp < genesis_ts → continue) never triggers inside a slot.
• Publisher eviction: Publisher::notify uses force_send, which evicts the oldest queued notification on overflow — but per-subscriber capacity is 1024 (PUBLISHER_QUEUE_CAPACITY) vs a ~110-event burst, and the loop was draining concurrently. Cannot produce 17 consecutive mid-burst drops.
• UI-side dedup/deserialization drops: zero "Skipping duplicate seen message" and zero "Failed deserializing incoming Privmsg" lines in the capture. The target channel is plaintext, so a message present in main_tree would have relayed. Hence the missing messages are genuinely absent from the app's DAG.
• Local replay (rescan): rescan_channel_history iterates order_events() over the local main_tree only; it cannot fetch anything.

Code path walkthrough

1. The parked sync task (bin/app/src/plugin/darkirc.rs, dag_sync)

loop {
    // ... wait for peers, run static_sync, then:
    let latest_ts = self.event_graph.current_genesis.read().await.header.timestamp;
    i!("Syncing newest event DAG ({latest_ts}) (attempt #{sync_attempt})");
    let sync_result = self.sync_dag_slot(latest_ts, fast_mode).await;
    match sync_result {
        Ok(()) => { newest_synced = true; break }   // <-- leaves the loop forever
        Err(e) => { e!("Failed syncing newest DAG ({e}), retrying..."); }
    }
}
// ...
loop {
    // Parked forever: only notifies the UI of connection changes.
    if let Err(err) = channel_sub.receive().await { continue }
    let peers_count = self.p2p.peers_count();
    self.notify_connect(peers_count, self.event_graph.is_synced()).await;
}

2. Wake only toggles outbound slots (bin/app/src/plugin/darkirc.rs)

let screen_changed_task = ex.spawn(async move {
    while let Ok(data) = screen_changed_recv.recv().await {
        // ...
        if screen_on {
            self_.set_outbound_connections(P2P_OUTBOUND_ACTIVE).await;
        } else {
            self_.set_outbound_connections(P2P_OUTBOUND_SLEEP).await;
        }
    }
});

No sync call anywhere on this path.

3. catch_up_sync exits permanently (bin/app/src/plugin/darkirc.rs)

pending = still_pending;
if pending.is_empty() {
    i!("Background catch-up complete; all older DAGs synced");
    break          // <- nothing ever syncs the *next* rotated-in slot
}

4. The only post-sync catch-up: gossip ancestry walks

(src/event_graph/proto.rs, fetch_parents)

// Only ever asks the single peer that sent the EventPut:
if self.channel.send(&EventReq(requested.clone())).await.is_err() { return false }
let Ok(rep) = self.ev_rep_sub.receive_with_timeout(timeout).await else {
    self.channel.stop().await;
    return false          // <- whole walk discarded, triggering event dropped
};

Single-peer, drop-on-failure, and — per MAX_PARENT_FETCH_DEPTH's own doc comment — explicitly not the intended mechanism for a node that is far behind: "A node that's 1000+ layers behind should be using dag_sync rather than relying on EventPut catch-up." The app never does.

5. Why a naive re-run of dag_sync is insufficient (sync_impl)

let missing: HashSet<blake3::Hash> = accepted
    .iter()
    .filter(|h| !slot.main_tree.contains_key(h.as_bytes()).unwrap_or(true))
    .cloned()
    .collect();
if missing.is_empty() {
    return Ok(())          // <- early return when we already hold all tips
}

At resume-before-gossip this is fine (our tips are stale, peers' tips are missing). But if reconciliation fires after gossip has already delivered the current tips, the early return skips the header/body phases and the gaps are never repaired. Meanwhile the serving side is well suited to repair: fetch_headers_with_tips returns every header not reachable from the requester's tips (branch events included), layer-sorted (which is a valid topological order because a parent's layer is always strictly lower than its child's), capped at MAX_HEADER_REP_HEADERS (4096), and handle_header_req whitelists revealed ids into broadcasted_ids so the follow-up body EventReqs are admitted.

Goals / Non-Goals

Goals:

• Any event that existed at peers while the app was suspended becomes fetchable again after resume, without a process restart.
• Repair works even when triggered late (after gossip has delivered tips).
• Repair also covers slots that rotate in while the app is long-lived (covers the catch_up_sync-exits gap).
• Observable: field logs state when a repair round ran and what it committed.

Non-Goals:

• Scrollback/pagination UI via RangeReq (separate change).
• Changing fetch_parents multi-peer fallback / retry semantics.
• Rotation/retention config changes; any p2p wire-protocol change.
• Recovering events no reachable peer still serves (out of any node's control once retention expires).

Decisions

D1: Repair primitive = header-sync + body-fetch, without the tip-quorum gate

Add a lib-level EventGraph::dag_repair(dag_ts) (in src/event_graph/mod.rs) that reuses sync_impl's machinery but:

• Skips the "missing tips" early return (always issues HeaderReq(our_tips) to all peers and inserts returned headers), because gap events are by definition not reachable from our tips — fetch_headers_with_tips excludes our tips' ancestors, so the response is exactly the unreached set.
• Runs fetch_missing_events afterward for bodies.
• Treats per-event commit failures leniently: log and continue, return Ok with a count (a single peer missing an RLN blob for one event must not fail the whole repair round; the strict behavior stays for initial sync where completeness is the contract).

Alternative rejected: calling existing dag_sync from the app — blocked by the early return above. Alternative rejected: app-side RangeReq scan — would need a cursor policy over time_index and duplicate the header/body machinery; range sync is designed for ordered pagination, not arbitrary-branch reconciliation.

D2: Triggers — wake signal, peer recovery, periodic

In the app plugin:

• screen_changed(screen_on=true) and the darkirc_start slot schedule a repair round (debounced) — where such signals exist (mobile).
• A dedicated task watching p2p.hosts().subscribe_channel() schedules a repair round when peers transition 0 → ≥1. This is the primary trigger for suspend/resume on machines where no in-app wake signal exists or fires: laptop lid-close/resume freezes the process wholesale, and the first observable symptom inside the app is peers dropping to zero and later reconnecting.
• A periodic timer re-runs repair for the current slot every REPAIR_INTERVAL (default: 15 min, constant — matching the file's existing "TODO: these should be configurable" style) so a failed round is retried and rotated-in slots are reconciled without any event at all. This is also the backstop for resume shapes that produce no clean 0 → ≥1 edge (e.g. connections that die and return one at a time while others stay up, or a resume racing the channel subscription).

Debounce/coalescing: a single in-flight guard (see D3) collapses simultaneous triggers; triggers arriving during a round are latched, not dropped.

Alternative rejected: re-arming the parked dag_sync loop by feeding it channel events — the parked loop's contract is notify-only; overloading it mixes initial-sync retry semantics with repair semantics. A separate repair_sync task mirrors catch_up_sync's shape and keeps the state machines disjoint.

D3: Concurrency and synced-flag discipline

• dag_repair must not run concurrently with an initial dag_sync/ sync_selected for the same slot or with another repair round: guard with an app-side AtomicBool/mutex ("repair in flight"), since both paths terminate in dag_insert_with_blobs which is idempotent for known events but the strict variant's error contract differs from repair's lenient one.
• The synced flag is only ever set (never cleared) by the initial sync; repair observes it (must not run before initial sync completes — same gate handle_event_put uses) and must not flip it, so live gossip ingestion is never blocked by a repair round.

D4: Observability

• Info log on repair round start (slot id, trigger source) and completion (headers gained, bodies committed, events skipped+why).
• Existing warn/error logs on peer-side serving refusals ("declining to serve event ... - missing blob") remain the primary diagnostic for unrecoverable events.

Risks / Trade-offs

• src/event_graph/mod.rs is a security-critical subsystem. D1 adds a read-heavy path reusing existing insert primitives; no validation, RLN, or pruning logic is altered. Flagged for explicit review, and the apply phase must run make test (event_graph tests are proof-dependent).
• Wake traffic spike: a repair round after a long suspension transfers all unreached headers/bodies for the current slot. Bounded by protocol limits (MAX_HEADER_REP_HEADERS 4096, MAX_EVENT_REQ_IDS 128/batch, MAX_RANGE_PAGE_SIZE 100). On mobile this is the same volume the initial sync would have paid; debouncing prevents amplification from multiple triggers.
• Truncation: fetch_headers_with_tips keeps the lowest 4096 headers when overflowing — a node very far behind may need multiple repair rounds to converge (each round advances the frontier). Acceptable; periodic retry converges.
• Quorum caveat inherited from tips collection: repair itself doesn't depend on the 2/3 tip quorum (that gate only feeds the early return we skip), but branch events known to a minority of peers are only recoverable while at least one such peer is connected and serving.
• Known unrecoverable case: events whose blobs no serving peer retains are skipped and logged, not fatally failed. The UI may still show gaps for those; this change guarantees retry + visibility, not impossible recovery.