Module utils

Structs

• BackfillState
• BackfillStatePerVnode

Enums

• BackfillProgressPerVnode
  Used for tracking backfill state per vnode The OwnedRow only contains the pk of upstream, to track current_pos.

Constants

• METADATA_STATE_LEN
  vnode, is_finished, row_count, all occupy 1 column each.

Functions

• build_temporary_state 🔒
  We want to avoid allocating a row for every vnode. Instead we can just modify a single row, and dispatch it to state table to write. This builds the following segments of the row:
• compute_bounds 🔒
• construct_initial_finished_state 🔒
• create_builder
  Creates a data chunk builder for snapshot read. If the rate_limit is smaller than chunk_size, it will take precedence. This is so we can partition snapshot read into smaller chunks than chunk size.
• create_limiter
• flush_data 🔒
  Flush the data
• get_cdc_chunk_last_offset 🔒
• get_new_pos 🔒
  Get new backfill pos from the chunk. Since chunk should have ordered rows, we can just take the last row.
• get_progress_per_vnode 🔒
  Recovers progress per vnode, so we know which to backfill. See how it decodes the state with the inline comments.
• iter_chunks 🔒
• mapping_chunk 🔒
  Builds a new stream chunk with output_indices.
• mapping_message 🔒
• mapping_watermark 🔒
• mark_cdc_chunk 🔒
• mark_cdc_chunk_inner 🔒
• mark_chunk 🔒
• mark_chunk_inner 🔒
  Mark chunk: For each row of the chunk, forward it to downstream if its pk <= current_pos, otherwise ignore it. We implement it by changing the visibility bitmap.
• mark_chunk_ref_by_vnode 🔒
  Mark chunk: For each row of the chunk, forward it to downstream if its pk <= current_pos for the corresponding vnode, otherwise ignore it. We implement it by changing the visibility bitmap.
• owned_row_iter 🔒
• persist_state 🔒
  Schema | vnode | pk | backfill_finished | row_count |
• persist_state_per_vnode 🔒
  Schema | vnode | pk | backfill_finished | row_count | Persists the state per vnode based on BackfillState. We track the current committed state via committed_progress so we know whether we need to persist the state or not.
• update_pos_by_vnode 🔒
  Update backfill pos by vnode.

Type Aliases

• BackfillRateLimiter