Module source_manager 

Modules

split_assignment 🔒

worker 🔒

Structs

SourceManager

SourceManager keeps fetching the latest split metadata from the external source services (worker::ConnectorSourceWorker::tick), and sends a split assignment command if split changes detected (Self::tick).

SourceManagerCore

SourceManagerRunningInfo

SplitDiffOptions

SplitState

Enums

DiscoveredSplits

Discovered splits from the source enumerator.

ReplaceJobSplitPlan

Describes how splits should be resolved for a replace stream job.

SourceChange

Constants

DEFAULT_SOURCE_TICK_TIMEOUT 🔒

Functions

align_splits

Assign splits to a new set of actors, according to existing assignment.

build_actor_connector_splits

build_actor_split_impls

create_source_worker

Used to create a new ConnectorSourceWorkerHandle for a new source.

reassign_splits

Reassigns splits if there are new splits or dropped splits, i.e., actor_splits and discovered_splits differ, or actors are rescheduled.

Type Aliases

ConnectorPropsChange

SourceManagerRef

SourceSplitAssignment

Source-level split assignment: maps source ID to the discovered splits enum. Unlike SplitAssignment, this does not include fragment- or actor-level assignment. The DiscoveredSplits enum is preserved all the way through to the barrier command, and only resolved to concrete per-fragment, per-actor splits when the barrier worker performs the actual split assignment.

SplitAssignment

Actor-level split assignment, used inside the barrier worker after actor rendering.