Struct Scheduler

pub(super) struct Scheduler {
    scheduled_worker_slots: Vec<WorkerSlotId>,
    default_hash_mapping: WorkerSlotMapping,
    default_singleton_worker_slot: WorkerSlotId,
}

Scheduler schedules the distribution of fragments in a stream graph.

Fields

scheduled_worker_slots: Vec<WorkerSlotId>

Worker slots to schedule. Use to generate mapping if a vnode count other than the default is required.

default_hash_mapping: WorkerSlotMapping

The default hash mapping for hash-distributed fragments, if there’s no requirement derived.

default_singleton_worker_slot: WorkerSlotId

The default worker slot for singleton fragments, if there’s no requirement derived.

Implementations

impl Scheduler

pub fn new( streaming_job_id: u32, workers: &HashMap<u32, WorkerNode>, default_parallelism: NonZeroUsize, expected_vnode_count: usize, ) -> MetaResult<Self>

Create a new Scheduler with the given worker slots and the default parallelism.

Each hash-distributed fragment will be scheduled to at most default_parallelism parallel units, in a round-robin fashion on all compute nodes. If the default_parallelism is None, all worker slots will be used.

For different streaming jobs, we even out possible scheduling skew by using the streaming job id as the salt for the scheduling algorithm.

pub fn schedule( &self, graph: &CompleteStreamFragmentGraph, ) -> MetaResult<HashMap<GlobalId<{ IdCategory::Fragment }>, Distribution>>

Schedule the given complete graph and returns the distribution of each building fragment.