Module stream

Re-exports

pub use source_manager::*;

Modules

cdc 🔒

refresh_manager 🔒

scale 🔒

sink 🔒

source_manager

state_match

This module contains the logic for matching the internal state tables of two streaming jobs, used for replacing a streaming job (typically ALTER MV) while preserving the existing state.

stream_graph 🔒

stream_manager 🔒

test_scale 🔒

Structs

ActorGraphBuildResult

The result of a built actor graph. Will be further embedded into the Context for building actors on the compute nodes.

ActorGraphBuilder

ActorGraphBuilder builds the actor graph for the given complete fragment graph, based on the current cluster info and the required parallelism.

Assigner

Core assigner with configurable strategies.

AssignerBuilder

Builder for Assigner.

AutoRefreshSchemaSinkContext

CompleteStreamFragmentGraph

A wrapper of StreamFragmentGraph that contains the additional information of pre-existing fragments, which are connected to the graph’s top-most or bottom-most fragments.

CreateStreamingJobContext

CreateStreamingJobContext carries one-time infos for creating a streaming job.

CreateStreamingJobOption

CustomActorInfo

CustomFragmentInfo

FragmentGraphDownstreamContext

FragmentGraphUpstreamContext

GlobalStreamManager

GlobalStreamManager manages all the streams in the system.

JobReschedulePlan

JobReschedulePolicy

JobReschedulePostUpdates

JobRescheduleTarget

Locations

Locations represents the locations of the actors.

RefreshManager

High level design for refresh table

ReplaceStreamJobContext

ReplaceStreamJobContext carries one-time infos for replacing the plan of an existing stream job.

RescheduleContext

RescheduleOptions

ScaleController

ScaleFactor

A validated, non-negative, finite scale factor.

StreamFragmentGraph

In-memory representation of a Fragment Graph, built from the [StreamFragmentGraphProto] from the frontend.

UpstreamSinkInfo

WorkerReschedule

Enums

BalancedBy

Defines the vnode distribution strategy for hierarchical assignment.

CapacityMode

Defines the capacity assignment strategy for containers.

CreatingState

JobParallelismTarget

JobResourceGroupTarget

VnodeChunkingStrategy

Defines the VNode chunking strategy for assignment.

Functions

assign_hierarchical

Hierarchically distributes virtual nodes to actors in two weighted stages with deterministic tie-breaking.

assign_items_weighted_with_scale_fn

Assign items to weighted containers with optional capacity scaling and deterministic tie-breaking.

check_sink_fragments_support_refresh_schema

fill_snapshot_backfill_epoch

Fill snapshot epoch for StreamScanNode of SnapshotBackfill. Return true when has change applied.

rebalance_actor_vnode

This function provides an simple balancing method The specific process is as follows

rewrite_refresh_schema_sink_fragment

unbounded_scale

A no-op capacity scaling function: always returns None.

validate_sink

weighted_scale

A unit capacity scaling function: always returns Some(1.0).

Type Aliases

FragmentBackfillOrder

Adjacency list (G) of backfill orders. G[10] -> [1, 2, 11] means for the backfill node in fragment 10 should be backfilled before the backfill nodes in fragment 1, 2 and 11.

GlobalStreamManagerRef

ScaleControllerRef