Skip to main content

Module stream

Module stream 

Source

Re-exports§

pub use source_manager::*;

Modules§

cdc 🔒
refresh_manager 🔒
scale 🔒
sink 🔒
source_manager
state_match
This module contains the logic for matching the internal state tables of two streaming jobs, used for replacing a streaming job (typically ALTER MV) while preserving the existing state.
stream_graph 🔒
stream_manager 🔒

Structs§

ActorGraphBuildResult
The result of a built actor graph. Will be further embedded into the Context for building actors on the compute nodes.
ActorGraphBuilder
ActorGraphBuilder builds the actor graph for the given complete fragment graph, based on the current cluster info and the required parallelism.
Assigner
Core assigner with configurable strategies.
AssignerBuilder
Builder for Assigner.
AutoRefreshSchemaSinkContext
CompleteStreamFragmentGraph
A wrapper of StreamFragmentGraph that contains the additional information of pre-existing fragments, which are connected to the graph’s top-most or bottom-most fragments.
CreateStreamingJobContext
CreateStreamingJobContext carries one-time infos for creating a streaming job.
CreateStreamingJobOption
FragmentGraphDownstreamContext
FragmentGraphUpstreamContext
GlobalRefreshManager
Owns the refresh cycles. The refresh_job row is the source of truth: a cycle is its last_trigger_time, and both transitions are the post-collect of a barrier (RefreshStart moves the job to Refreshing, FinishRefresh back to Idle). A recovery kills every cycle of the databases it re-initializes and moves their rows back to Idle before those databases accept commands again (abandon_cycles), so nothing has to infer an abandoned cycle afterwards. The in-memory tracker of a cycle is installed by the same barrier that started it, before the row moves.
GlobalStreamManager
GlobalStreamManager manages all the streams in the system.
Locations
Locations represents the locations of the actors.
ParallelismPolicy
RefreshCycleActors
The actors of a refreshable table’s job, grouped by the stage they report.
ReplaceStreamJobContext
ReplaceStreamJobContext carries one-time infos for replacing the plan of an existing stream job.
ResourceGroupPolicy
ScaleController
ScaleFactor
A validated, non-negative, finite scale factor.
StreamFragmentGraph
In-memory representation of a Fragment Graph, built from the StreamFragmentGraphProto from the frontend.
UpstreamSinkInfo
WorkerReschedule

Enums§

BalancedBy
Defines the vnode distribution strategy for hierarchical assignment.
CapacityMode
Defines the capacity assignment strategy for containers.
RefreshStage
ReschedulePolicy
VnodeChunkingStrategy
Defines the VNode chunking strategy for assignment.

Functions§

assign_hierarchical
Hierarchically distributes virtual nodes to actors in two weighted stages with deterministic tie-breaking.
assign_items_weighted_with_scale_fn
Assign items to weighted containers with optional capacity scaling and deterministic tie-breaking.
check_sink_fragments_support_refresh_schema
fill_snapshot_backfill_epoch
Fill snapshot epoch for StreamScanNode of SnapshotBackfill. Return true when has change applied.
first_variant_column
Returns the first column whose type contains VARIANT (nested included). Callers use this to reject connectors that cannot encode variant values.
rewrite_refresh_schema_sink_fragment
unbounded_scale
A no-op capacity scaling function: always returns None.
validate_sink
weighted_scale
A unit capacity scaling function: always returns Some(1.0).

Type Aliases§

CreateStreamingJobResult
The error carries whether the caller should explicitly cancel the creating job and an optional notifier for an awaited cancellation request.
ExtendedFragmentBackfillOrder
GlobalRefreshManagerRef
GlobalStreamManagerRef
ScaleControllerRef
UserDefinedFragmentBackfillOrder