resolve_no_shuffle_ensembles

Function resolve_no_shuffle_ensembles 

Source
fn resolve_no_shuffle_ensembles(
    fragments: &StreamJobFragmentsToCreate,
    upstream_fragment_downstreams: &FragmentDownstreamRelation,
) -> MetaResult<Vec<NoShuffleEnsemble>>
Expand description

Derive NoShuffle edges from fragment downstream relations and resolve ensembles.

This scans both the internal downstream relations (fragments.downstreams) and the cross-boundary upstream-to-new-fragment relations (upstream_fragment_downstreams) to find all NoShuffle edges. It then runs BFS to find connected components (ensembles) and categorizes them into:

  • Ensembles whose entry fragments include existing (non-new) fragments
  • Ensembles whose entry fragments are all newly created