Module plan_node

Expand description

Defines all kinds of node in the plan tree, each node represent a relational expression.

We use a immutable style tree structure, every Node are immutable and cannot be modified after it has been created. If you want to modify the node, such as rewriting the expression in a ProjectNode or changing a node’s input node, you need to create a new node. We use Rc as the node’s reference, and a node just storage its inputs’ reference, so change a node just need create one new node but not the entire sub-tree.

So when you want to add a new node, make sure:

each field in the node struct are private
recommend to implement the construction of Node in a unified new() function, if have multi methods to construct, make they have a consistent behavior
all field should be valued in construction, so the properties’ derivation should be finished in the new() function.

Re-exports§

pub use generic::PlanAggCall;
pub use generic::PlanAggCallDisplay;

Modules§

batch
batch_delete 🔒
batch_exchange 🔒
batch_expand 🔒
batch_file_scan 🔒
batch_filter 🔒
batch_group_topn 🔒
batch_hash_agg 🔒
batch_hash_join 🔒
batch_hop_window 🔒
batch_iceberg_scan 🔒
batch_insert 🔒
batch_kafka_scan 🔒
batch_limit 🔒
batch_log_seq_scan 🔒
batch_lookup_join 🔒
batch_max_one_row 🔒
batch_mysql_query 🔒
batch_nested_loop_join 🔒
batch_over_window 🔒
batch_postgres_query 🔒
batch_project 🔒
batch_project_set 🔒
batch_seq_scan 🔒
batch_simple_agg 🔒
batch_sort 🔒
batch_sort_agg 🔒
batch_source 🔒
batch_sys_seq_scan 🔒
batch_table_function 🔒
batch_topn 🔒
batch_union 🔒
batch_update 🔒
batch_values 🔒
col_pruning 🔒
convert 🔒
derive 🔒
eq_join_predicate 🔒
expr_rewritable 🔒
expr_visitable 🔒
generic: This module contains the generic plan nodes that are shared by all the plan nodes. They are meant to reuse the common fields between logical, batch and stream nodes.
logical_agg 🔒
logical_apply 🔒
logical_cdc_scan 🔒
logical_changelog 🔒
logical_cte_ref 🔒
logical_dedup 🔒
logical_delete 🔒
logical_except 🔒
logical_expand 🔒
logical_file_scan 🔒
logical_filter 🔒
logical_hop_window 🔒
logical_iceberg_scan 🔒
logical_insert 🔒
logical_intersect 🔒
logical_join 🔒
logical_kafka_scan 🔒
logical_limit 🔒
logical_max_one_row 🔒
logical_multi_join 🔒
logical_mysql_query 🔒
logical_now 🔒
logical_over_window 🔒
logical_postgres_query 🔒
logical_project 🔒
logical_project_set 🔒
logical_recursive_union 🔒
logical_scan 🔒
logical_share 🔒
logical_source 🔒
logical_sys_scan 🔒
logical_table_function 🔒
logical_topn 🔒
logical_union 🔒
logical_update 🔒
logical_values 🔒
merge_eq_nodes 🔒
plan_base 🔒
plan_node_meta 🔒
plan_tree_node 🔒
predicate_pushdown 🔒
stream
stream_asof_join 🔒
stream_cdc_table_scan 🔒
stream_changelog 🔒
stream_dedup 🔒
stream_delta_join 🔒
stream_dml 🔒
stream_dynamic_filter 🔒
stream_eowc_over_window 🔒
stream_exchange 🔒
stream_expand 🔒
stream_filter 🔒
stream_fs_fetch 🔒
stream_global_approx_percentile 🔒
stream_group_topn 🔒
stream_hash_agg 🔒
stream_hash_join 🔒
stream_hop_window 🔒
stream_join_common 🔒
stream_local_approx_percentile 🔒
stream_materialize 🔒
stream_materialized_exprs 🔒
stream_now 🔒
stream_over_window 🔒
stream_project 🔒
stream_project_set 🔒
stream_row_id_gen 🔒
stream_row_merge 🔒
stream_share 🔒
stream_simple_agg 🔒
stream_sink 🔒
stream_sort 🔒
stream_source 🔒
stream_source_scan 🔒
stream_stateless_simple_agg 🔒
stream_sync_log_store 🔒
stream_table_scan 🔒
stream_temporal_join 🔒
stream_topn 🔒
stream_union 🔒
stream_values 🔒
stream_watermark_filter 🔒
to_prost 🔒
utils

Macros§

impl_down_cast_fn 🔒: impl plan node downcast fn for each node.
impl_plan_node 🔒
impl_plan_node_meta 🔒: impl PlanNodeType fn for each node.

Structs§

Batch: The marker for batch convention.
BatchDelete: BatchDelete implements super::LogicalDelete
BatchExchange: BatchExchange imposes a particular distribution on its input without changing its content.
BatchExpand
BatchExtra: Extra fields for batch plan nodes.
BatchFileScan
BatchFilter: BatchFilter implements super::LogicalFilter
BatchGroupTopN: BatchGroupTopN implements super::LogicalTopN to find the top N elements with a heap
BatchHashAgg
BatchHashJoin: BatchHashJoin implements super::LogicalJoin with hash table. It builds a hash table from inner (right-side) relation and then probes with data from outer (left-side) relation to get output rows.
BatchHopWindow: BatchHopWindow implements super::LogicalHopWindow to evaluate specified expressions on input rows
BatchIcebergScan
BatchInsert: BatchInsert implements super::LogicalInsert
BatchKafkaScan
BatchLimit: BatchLimit implements super::LogicalLimit to fetch specified rows from input
BatchLogSeqScan
BatchLookupJoin
BatchMaxOneRow: BatchMaxOneRow fetches up to one row from the input, returning an error if the input contains more than one row at runtime.
BatchMySqlQuery
BatchNestedLoopJoin: BatchNestedLoopJoin implements super::LogicalJoin by checking the join condition against all pairs of rows from inner & outer side within 2 layers of loops.
BatchOverWindow
BatchPostgresQuery
BatchProject: BatchProject implements super::LogicalProject to evaluate specified expressions on input rows
BatchProjectSet
BatchSeqScan: BatchSeqScan implements super::LogicalScan to scan from a row-oriented table
BatchSimpleAgg
BatchSort: BatchSort buffers all data from input and sort these rows by specified order, providing the collation required by user or parent plan node.
BatchSortAgg
BatchSource: BatchSource represents a table/connector source at the very beginning of the graph.
BatchSysSeqScan: BatchSysSeqScan implements super::LogicalSysScan to scan from a row-oriented table
BatchTableFunction
BatchTopN: BatchTopN implements super::LogicalTopN to find the top N elements with a heap
BatchUnion: BatchUnion implements super::LogicalUnion
BatchUpdate: BatchUpdate implements super::LogicalUpdate
BatchValues
ColumnPruningContext
EqJoinPredicate: The join predicate used in optimizer
EqJoinPredicateDisplay
IcebergPartitionInfo
Logical: The marker for logical convention.
LogicalAgg: LogicalAgg groups input data by their group key and computes aggregation functions.
LogicalApply: LogicalApply represents a correlated join, where the right side may refer to columns from the left side.
LogicalCdcScan: LogicalCdcScan reads rows of a table from an external upstream database
LogicalChangeLog
LogicalCteRef
LogicalDedup: LogicalDedup deduplicates data on specific columns. It is now used in DISTINCT ON without an ORDER BY.
LogicalDelete: LogicalDelete iterates on input relation and delete the data from specified table.
LogicalExcept: LogicalExcept returns the rows of its first input except any matching rows from its other inputs.
LogicalExpand: LogicalExpand expands one row multiple times according to column_subsets and also keeps original columns of input. It can be used to implement distinct aggregation and group set.
LogicalFileScan
LogicalFilter: LogicalFilter iterates over its input and returns elements for which predicate evaluates to true, filtering out the others.
LogicalHopWindow: LogicalHopWindow implements Hop Table Function.
LogicalIcebergScan: LogicalIcebergScan is only used by batch queries. At the beginning of the batch query optimization, LogicalSource with a iceberg property would be converted into a LogicalIcebergScan.
LogicalInsert: LogicalInsert iterates on input relation and insert the data into specified table.
LogicalIntersect: LogicalIntersect returns the intersect of the rows of its inputs. If all is false, it needs to eliminate duplicates.
LogicalJoin: LogicalJoin combines two relations according to some condition.
LogicalKafkaScan: LogicalKafkaScan is only used by batch queries. At the beginning of the batch query optimization, LogicalSource with a kafka property would be converted into a LogicalKafkaScan.
LogicalLimit: LogicalLimit fetches up to limit rows from offset
LogicalMaxOneRow: LogicalMaxOneRow fetches up to one row from the input, returning an error if the input contains more than one row at runtime. Only available in batch mode.
LogicalMultiJoin: LogicalMultiJoin combines two or more relations according to some condition.
LogicalMultiJoinBuilder
LogicalMySqlQuery
LogicalNow
LogicalOverWindow: LogicalOverWindow performs OVER window functions to its input.
LogicalPostgresQuery
LogicalProject: LogicalProject computes a set of expressions from its input relation.
LogicalProjectSet: LogicalProjectSet projects one row multiple times according to select_list.
LogicalRecursiveUnion: LogicalRecursiveUnion returns the union of the rows of its inputs. note: if all is false, it needs to eliminate duplicates.
LogicalScan: LogicalScan returns contents of a table or other equivalent object
LogicalShare: LogicalShare operator is used to represent reusing of existing operators. It is the key operator for DAG plan. It could have multiple parents which makes it different from other operators. Currently, it has been used to the following scenarios:
LogicalSource: LogicalSource returns contents of a table or other equivalent object
LogicalSysScan: LogicalSysScan returns contents of a table or other equivalent object
LogicalTableFunction: LogicalTableFunction is a scalar/table function used as a relation (in the FROM clause).
LogicalTopN: LogicalTopN sorts the input data and fetches up to limit rows from offset
LogicalUnion: LogicalUnion returns the union of the rows of its inputs. If all is false, it needs to eliminate duplicates.
LogicalUpdate: LogicalUpdate iterates on input relation, set some columns, and inject update records into specified table.
LogicalValues: LogicalValues builds rows according to a list of expressions
NoExtra: No extra fields for logical plan nodes.
PlanBase: The common fields of all plan nodes with different conventions.
PlanNodeId
PlanRef
PredicatePushdownContext
RewriteStreamContext
Stream: The marker for stream convention.
StreamAsOfJoin: StreamAsOfJoin implements super::LogicalJoin with hash tables.
StreamCdcTableScan: StreamCdcTableScan is a virtual plan node to represent a stream cdc table scan. It will be converted to cdc backfill + merge node (for upstream source)
StreamChangeLog
StreamDedup
StreamDeltaJoin: StreamDeltaJoin implements super::LogicalJoin with delta join. It requires its two inputs to be indexes.
StreamDml
StreamDynamicFilter
StreamEowcOverWindow
StreamEowcSort
StreamExchange: StreamExchange imposes a particular distribution on its input without changing its content.
StreamExpand
StreamExtra: Extra fields for stream plan nodes.
StreamFilter: StreamFilter implements super::LogicalFilter
StreamFsFetch: Fetch files from filesystem/s3/iceberg.
StreamGlobalApproxPercentile
StreamGroupTopN
StreamHashAgg
StreamHashJoin: StreamHashJoin implements super::LogicalJoin with hash table. It builds a hash table from inner (right-side) relation and probes with data from outer (left-side) relation to get output rows.
StreamHopWindow: StreamHopWindow represents a hop window table function.
StreamLocalApproxPercentile
StreamMaterialize: Materializes a stream.
StreamMaterializedExprs: StreamMaterializedExprs materializes the results of a set of expressions. The expressions are evaluated once and the results are stored in a state table, avoiding re-evaluation for delete operations. Particularly useful for expensive or non-deterministic expressions like UDF calls.
StreamNow
StreamOverWindow
StreamProject: StreamProject implements super::LogicalProject to evaluate specified expressions on input rows.
StreamProjectSet
StreamRowIdGen
StreamRowMerge: StreamRowMerge is used for merging two streams with the same stream key and distribution. It will buffer the outputs from its input streams until we receive a barrier. On receiving a barrier, it will Project their outputs according to the provided lhs_mapping and rhs_mapping.
StreamShare: StreamShare will be translated into an ExchangeNode based on its distribution finally.
StreamSimpleAgg
StreamSink: StreamSink represents a table/connector sink at the very end of the graph.
StreamSource: StreamSource represents a table/connector source at the very beginning of the graph.
StreamSourceScan: StreamSourceScan scans from a shared source. It forwards data from the upstream StreamSource, and also backfills data from the external source.
StreamStatelessSimpleAgg: Streaming stateless simple agg.
StreamSyncLogStore
StreamTableScan: StreamTableScan is a virtual plan node to represent a stream table scan. It will be converted to stream scan + merge node (for upstream materialize) + batch table scan when converting to MView creation request.
StreamTemporalJoin
StreamTopN: StreamTopN implements super::LogicalTopN to find the top N elements with a heap
StreamUnion: StreamUnion implements super::LogicalUnion
StreamValues: StreamValues implements LogicalValues.to_stream()
StreamWatermarkFilter
ToStreamContext

Enums§

Convention
PartitionComputeInfo: Why we need PartitionComputeInfo?
PlanBaseRef: Reference to PlanBase with erased conventions.
PlanNodeType: each enum value represent a PlanNode struct type, help us to dispatch and downcast

Constants§

PLAN_DEPTH_THRESHOLD: Recursion depth threshold for plan node visitor to send notice to user.
PLAN_TOO_DEEP_NOTICE: Notice message for plan node visitor to send to user when the depth threshold is reached.

Traits§

ColPrunable: The trait for column pruning, only logical plan node will use it, though all plan node impl it.
ConventionMarker: A marker trait for different conventions, used for enforcing type safety.
EndoPlan: A more sophisticated Endo taking into account of the DAG structure of PlanRef. In addition to Endo, one have to specify the cached function to persist transformed LogicalShare and their results, and the dag_apply function will take care to only transform every LogicalShare nodes once.
Explain
ExprRewritable: Rewrites expressions in a PlanRef. Due to Share operator, the ExprRewriter needs to be idempotent i.e., applying it more than once to the same ExprImpl will be a noop on subsequent applications. rewrite_exprs should only return a plan with the given node modified. To rewrite recursively, call rewrite_exprs_recursive on RewriteExprsRecursive.
PlanNode: The common trait over all plan nodes. Used by optimizer framework which will treat all node as dyn PlanNode
PlanNodeMeta: The trait for accessing the meta data and PlanBase for plan nodes.
PlanTreeNode: The trait PlanNode really need about tree structure and used by optimizer framework. every plan node should impl it.
PlanTreeNodeBinary: See PlanTreeNode
PlanTreeNodeLeaf: See PlanTreeNode
PlanTreeNodeUnary: See PlanTreeNode
PredicatePushdown: The trait for predicate pushdown, only logical plan node will use it, though all plan node impl it.
RewriteExprsRecursive 🔒
Semantics
StreamNode
ToBatch: ToBatch allows to convert a logical plan node to batch physical node with an optional required order.
ToBatchPb
ToDistributedBatch: ToDistributedBatch allows to convert a batch physical plan to distributed batch plan, by insert exchange node, with an optional required order and distributed.
ToLocalBatch: Converts a batch physical plan to local plan for local execution.
ToPb
ToStream: ToStream converts a logical plan node to streaming physical node with an optional required distribution.
TryToBatchPb
TryToStreamPb
VisitExprsRecursive 🔒
VisitPlan: A more sophisticated Visit taking into account of the DAG structure of PlanRef. In addition to Visit, one have to specify visited to store and report visited LogicalShare nodes, and the dag_visit function will take care to only visit every LogicalShare nodes once. See also EndoPlan.

Functions§

gen_filter_and_pushdown
pretty_config 🔒
reorganize_elements_id: In order to let expression display id started from 1 for explaining, hidden column names and other places. We will reset expression display id to 0 and clone the whole plan to reset the schema.
stream_enforce_eowc_requirement

Module plan_nodeCopy item path