Skip to main content

Module committed_epoch

Module committed_epoch 

Source
Expand description

Per-PartialGraphId committed-epoch tracking for the iceberg pk-index sink.

Tracks, per partial graph, the latest checkpoint epoch whose iceberg commit has completed. A cursor entry exists exactly while a partial graph has a registered pk-index sink: PartialGraphCommittedEpochs::ensure is called from register_sink and PartialGraphCommittedEpochs::remove from unregister. Both PartialGraphCommittedEpochs::advance_all and PartialGraphCommittedEpochs::wait are non-creating, so a partial graph without a sink never accumulates an entry, and a wait that races an unregister resolves immediately (the caller then observes the coordinator is gone and errors out) rather than blocking on a resurrected entry that would never advance.

The key is a PartialGraphId rather than a DatabaseId so that a sink running in an independent partial graph (e.g. a batch-refresh job, to_partial_graph_id(database_id, Some(job_id))) is tracked by the very same mechanism: the barrier-completion path advances every completed partial graph, so no change here is needed to support such jobs. For a normal sink the key is its database’s main graph (to_partial_graph_id(database_id, None)).

The barrier-completion path advances the cursor on every checkpoint completion (even epochs where a sink reported nothing), so a merger’s seed wait converges even on idle tables. Waiters block on a tokio::sync::watch receiver and never hold any other lock.

Note: this is unrelated to RisingWave’s stream watermark abstraction; it is purely a per-partial-graph “iceberg commit has progressed to epoch N” cursor.

Structs§

PartialGraphCommittedEpochs
Per-PartialGraphId monotonic committed-epoch cursor, backed by one watch channel per tracked partial graph.