Re-exportsΒ§
pub use source_manager::*;
ModulesΒ§
- cdc π
- refresh_
manager π - scale π
- sink π
- source_
manager - state_
match - This module contains the logic for matching the internal state tables of two streaming jobs,
used for replacing a streaming job (typically
ALTER MV
) while preserving the existing state. - stream_
graph π - stream_
manager π - test_
scale π
StructsΒ§
- Actor
Graph Build Result - The result of a built actor graph. Will be further embedded into the
Context
for building actors on the compute nodes. - Actor
Graph Builder ActorGraphBuilder
builds the actor graph for the given complete fragment graph, based on the current cluster info and the required parallelism.- Assigner
- Core assigner with configurable strategies.
- Assigner
Builder - Builder for
Assigner
. - Auto
Refresh Schema Sink Context - Complete
Stream Fragment Graph - A wrapper of
StreamFragmentGraph
that contains the additional information of pre-existing fragments, which are connected to the graphβs top-most or bottom-most fragments. - Create
Streaming JobContext CreateStreamingJobContext
carries one-time infos for creating a streaming job.- Create
Streaming JobOption - Custom
Actor Info - Custom
Fragment Info - Global
Stream Manager GlobalStreamManager
manages all the streams in the system.- JobReschedule
Plan - JobReschedule
Policy - JobReschedule
Post Updates - JobReschedule
Target - Locations
Locations
represents the locations of the actors.- Refresh
Manager - High level design for refresh table
- Replace
Stream JobContext ReplaceStreamJobContext
carries one-time infos for replacing the plan of an existing stream job.- Reschedule
Context - Reschedule
Options - Scale
Controller - Scale
Factor - A validated, non-negative, finite scale factor.
- Stream
Fragment Graph - In-memory representation of a Fragment Graph, built from the [
StreamFragmentGraphProto
] from the frontend. - Upstream
Sink Info - Worker
Reschedule
EnumsΒ§
- Balanced
By - Defines the vnode distribution strategy for hierarchical assignment.
- Capacity
Mode - Defines the capacity assignment strategy for containers.
- Creating
State - JobParallelism
Target - JobResource
Group Target - Vnode
Chunking Strategy - Defines the VNode chunking strategy for assignment.
FunctionsΒ§
- assign_
hierarchical - Hierarchically distributes virtual nodes to actors in two weighted stages with deterministic tie-breaking.
- assign_
items_ weighted_ with_ scale_ fn - Assign items to weighted containers with optional capacity scaling and deterministic tie-breaking.
- check_
sink_ fragments_ support_ refresh_ schema - fill_
snapshot_ backfill_ epoch - Fill snapshot epoch for
StreamScanNode
ofSnapshotBackfill
. Returntrue
when has change applied. - rebalance_
actor_ vnode - This function provides an simple balancing method The specific process is as follows
- rewrite_
refresh_ schema_ sink_ fragment - unbounded_
scale - A no-op capacity scaling function: always returns
None
. - validate_
sink - weighted_
scale - A unit capacity scaling function: always returns
Some(1.0)
.
Type AliasesΒ§
- Fragment
Backfill Order - Adjacency list (G) of backfill orders.
G[10] -> [1, 2, 11]
means for the backfill node infragment 10
should be backfilled before the backfill nodes infragment 1, 2 and 11
. - Global
Stream Manager Ref - Scale
Controller Ref