Module stream

Modules

scale 🔒

sink 🔒

source_manager 🔒

stream_graph 🔒

stream_manager 🔒

test_scale 🔒

Structs

ActorGraphBuildResult

The result of a built actor graph. Will be further embedded into the Context for building actors on the compute nodes.

ActorGraphBuilder

ActorGraphBuilder builds the actor graph for the given complete fragment graph, based on the current cluster info and the required parallelism.

CompleteStreamFragmentGraph

A wrapper of StreamFragmentGraph that contains the additional information of pre-existing fragments, which are connected to the graph’s top-most or bottom-most fragments.

ConsistentHashRing

CreateStreamingJobContext

CreateStreamingJobContext carries one-time infos for creating a streaming job.

CreateStreamingJobOption

CustomActorInfo

CustomFragmentInfo

GlobalStreamManager

GlobalStreamManager manages all the streams in the system.

JobReschedulePlan

JobReschedulePolicy

JobReschedulePostUpdates

JobRescheduleTarget

Locations

Locations represents the worker slot and worker locations of the actors.

ReplaceStreamJobContext

ReplaceStreamJobContext carries one-time infos for replacing the plan of an existing stream job.

RescheduleContext

RescheduleOptions

ScaleController

SourceManager

SourceManager keeps fetching the latest split metadata from the external source services (worker::ConnectorSourceWorker::tick), and sends a split assignment command if split changes detected (Self::tick).

SourceManagerCore

SourceManagerRunningInfo

StreamFragmentGraph

In-memory representation of a Fragment Graph, built from the [StreamFragmentGraphProto] from the frontend.

WorkerReschedule

Enums

CreatingState

JobParallelismTarget

JobResourceGroupTarget

SourceChange

Functions

build_actor_connector_splits

build_actor_split_impls

create_source_worker

Used to create a new ConnectorSourceWorkerHandle for a new source.

fill_snapshot_backfill_epoch

Fill snapshot epoch for StreamScanNode of SnapshotBackfill. Return true when has change applied.

rebalance_actor_vnode

This function provides an simple balancing method The specific process is as follows

schedule_units_for_slots

validate_sink

Type Aliases

ConnectorPropsChange

GlobalStreamManagerRef

ScaleControllerRef

SourceManagerRef

SplitAssignment

ThrottleConfig