Struct CreateMviewProgressTracker

pub(super) struct CreateMviewProgressTracker {
    progress_map: HashMap<TableId, (Progress, TrackingJob)>,
    actor_map: HashMap<ActorId, TableId>,
    pending_finished_jobs: Vec<TrackingJob>,
}

Tracking is done as follows:

1. We identify a StreamJob by its TableId of its Materialized table.
2. For each stream job, there are several actors which run its tasks.
3. With progress_map we can use the ID of the StreamJob to view its progress.
4. With actor_map we can use an actor’s ActorId to find the ID of the StreamJob.

Fields

progress_map: HashMap<TableId, (Progress, TrackingJob)>

Progress of the create-mview DDL indicated by the TableId.

actor_map: HashMap<ActorId, TableId>

pending_finished_jobs: Vec<TrackingJob>

Stash of finished jobs. They will be finally finished on checkpoint.

Implementations

impl CreateMviewProgressTracker

pub fn recover( mview_map: HashMap<TableId, (String, TableFragments)>, version_stats: &HummockVersionStats, ) -> Self

This step recovers state from the meta side:

1. Tables.
2. TableFragments.

Other state are persisted by the BackfillExecutor, such as:

1. CreateMviewProgress.
2. Backfill position.

fn recover_progress( states: HashMap<ActorId, BackfillState>, backfill_upstream_types: HashMap<ActorId, BackfillUpstreamType>, upstream_mv_count: HashMap<TableId, usize>, definition: String, version_stats: &HummockVersionStats, ) -> Progress

How recovery works

The progress (number of rows consumed) is persisted in state tables. During recovery, the backfill executor will restore the number of rows consumed, and then it will just report progress like newly created executors.

pub fn gen_ddl_progress(&self) -> HashMap<u32, DdlProgress>

pub(super) fn update_tracking_jobs<'a>( &mut self, info: Option<(&CreateStreamingJobCommandInfo, Option<&ReplaceTablePlan>)>, create_mview_progress: impl IntoIterator<Item = &'a CreateMviewProgress>, version_stats: &HummockVersionStats, )

pub(super) fn apply_collected_command( &mut self, command: Option<&Command>, barrier_info: &BarrierInfo, resps: impl IntoIterator<Item = &PbBarrierCompleteResponse>, version_stats: &HummockVersionStats, ) -> Vec<TrackingJob>

Apply a collected epoch node command to the tracker Return the finished jobs when the barrier kind is Checkpoint

pub(super) fn stash_command_to_finish(&mut self, finished_job: TrackingJob)

Stash a command to finish later.

pub(super) fn take_finished_jobs(&mut self) -> Vec<TrackingJob>

Finish stashed jobs on checkpoint.

pub(super) fn has_pending_finished_jobs(&self) -> bool

pub(super) fn cancel_command(&mut self, id: TableId)

pub fn abort_all(&mut self)

Notify all tracked commands that error encountered and clear them.

pub fn add( &mut self, info: &CreateStreamingJobCommandInfo, replace_table: Option<&ReplaceTablePlan>, version_stats: &HummockVersionStats, ) -> Option<TrackingJob>

Add a new create-mview DDL command to track.

If the actors to track is empty, return the given command as it can be finished immediately.

pub fn update( &mut self, progress: &CreateMviewProgress, version_stats: &HummockVersionStats, ) -> Option<TrackingJob>

Update the progress of actor according to the Pb struct.

If all actors in this MV have finished, returns the command.

Trait Implementations

impl Debug for CreateMviewProgressTracker

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CreateMviewProgressTracker

fn default() -> CreateMviewProgressTracker

Returns the “default value” for a type.