Struct StreamFragmentGraph

pub struct StreamFragmentGraph {
    fragments: HashMap<GlobalId<{ IdCategory::Fragment }>, BuildingFragment>,
    downstreams: HashMap<GlobalId<{ IdCategory::Fragment }>, HashMap<GlobalId<{ IdCategory::Fragment }>, StreamFragmentEdge>>,
    upstreams: HashMap<GlobalId<{ IdCategory::Fragment }>, HashMap<GlobalId<{ IdCategory::Fragment }>, StreamFragmentEdge>>,
    dependent_table_ids: HashSet<TableId>,
    specified_parallelism: Option<NonZeroUsize>,
    max_parallelism: usize,
}

In-memory representation of a Fragment Graph, built from the StreamFragmentGraphProto from the frontend.

This only includes nodes and edges of the current job itself. It will be converted to CompleteStreamFragmentGraph later, that contains the additional information of pre-existing fragments, which are connected to the graph’s top-most or bottom-most fragments.

Fields

fragments: HashMap<GlobalId<{ IdCategory::Fragment }>, BuildingFragment>

stores all the fragments in the graph.

downstreams: HashMap<GlobalId<{ IdCategory::Fragment }>, HashMap<GlobalId<{ IdCategory::Fragment }>, StreamFragmentEdge>>

stores edges between fragments: upstream => downstream.

upstreams: HashMap<GlobalId<{ IdCategory::Fragment }>, HashMap<GlobalId<{ IdCategory::Fragment }>, StreamFragmentEdge>>

stores edges between fragments: downstream -> upstream.

dependent_table_ids: HashSet<TableId>

Dependent relations of this job.

specified_parallelism: Option<NonZeroUsize>

The default parallelism of the job, specified by the STREAMING_PARALLELISM session variable. If not specified, all active worker slots will be used.

max_parallelism: usize

Specified max parallelism, i.e., expected vnode count for the graph.

The scheduler on the meta service will use this as a hint to decide the vnode count for each fragment.

Note that the actual vnode count may be different from this value. For example, a no-shuffle exchange between current fragment graph and an existing upstream fragment graph requires two fragments to be in the same distribution, thus the same vnode count.

Implementations

impl StreamFragmentGraph

pub fn new( env: &MetaSrvEnv, proto: StreamFragmentGraphProto, job: &StreamingJob, ) -> MetaResult<Self>

Create a new StreamFragmentGraph from the given StreamFragmentGraphProto, with all global IDs correctly filled.

pub fn incomplete_internal_tables(&self) -> BTreeMap<u32, Table>

Retrieve the incomplete internal tables map of the whole graph.

Note that some fields in the table catalogs are not filled during the current phase, e.g., fragment_id, vnode_count. They will be all filled after a TableFragments is built. Be careful when using the returned values.

See also crate::model::TableFragments::internal_tables.

pub fn refill_internal_table_ids(&mut self, table_id_map: HashMap<u32, u32>)

Refill the internal tables’ table_ids according to the given map, typically obtained from create_internal_table_catalog.

pub fn fit_internal_table_ids( &mut self, old_internal_tables: Vec<Table>, ) -> MetaResult<()>

Set internal tables’ table_ids according to a list of internal tables

pub fn table_fragment_id(&self) -> FragmentId

Returns the fragment id where the streaming job node located.

pub fn dml_fragment_id(&self) -> Option<FragmentId>

Returns the fragment id where the table dml is received.

pub fn dependent_table_ids(&self) -> &HashSet<TableId>

Get the dependent streaming job ids of this job.

pub fn specified_parallelism(&self) -> Option<NonZeroUsize>

Get the parallelism of the job, if specified by the user.

pub fn max_parallelism(&self) -> usize

Get the expected vnode count of the graph. See documentation of the field for more details.

fn get_downstreams( &self, fragment_id: GlobalId<{ IdCategory::Fragment }>, ) -> &HashMap<GlobalId<{ IdCategory::Fragment }>, StreamFragmentEdge>

Get downstreams of a fragment.

fn get_upstreams( &self, fragment_id: GlobalId<{ IdCategory::Fragment }>, ) -> &HashMap<GlobalId<{ IdCategory::Fragment }>, StreamFragmentEdge>

Get upstreams of a fragment.

pub fn collect_snapshot_backfill_info( &self, ) -> MetaResult<Option<SnapshotBackfillInfo>>

Trait Implementations

impl Default for StreamFragmentGraph

fn default() -> StreamFragmentGraph

Returns the “default value” for a type.