Module executor Copy item path Source aggregation This module contains wrappers over Aggregator trait that implement order by, filter,
distinct and project. azblob_file_scan 🔒 delete 🔒 expand 🔒 filter 🔒 gcs_file_scan 🔒 generic_exchange 🔒 group_top_n 🔒 hash_agg 🔒 hash_join hop_window 🔒 iceberg_scan 🔒 insert 🔒 join 🔒 limit 🔒 local_lookup_join log_row_seq_scan 🔒 max_one_row 🔒 merge_sort 🔒 merge_sort_exchange 🔒 mysql_query 🔒 nested_loop_join order_by 🔒 postgres_query 🔒 project 🔒 project_set 🔒 row_seq_scan 🔒 s3_file_scan 🔒 sort_agg 🔒 sort_over_window 🔒 source 🔒 sys_row_seq_scan 🔒 table_function 🔒 test_utils top_n 🔒 union 🔒 update 🔒 utils 🔒 values 🔒 register_executor Register an executor builder so that it can be used to build the executor from protobuf. AggSpillManager AggSpillManager is used to manage how to write spill data file and read them back.
The spill data first need to be partitioned. Each partition contains 2 files: agg_state_file and input_chunks_file.
The spill file consume a data chunk and serialize the chunk into a protobuf bytes.
Finally, spill file content will look like the below.
The file write pattern is append-only and the read pattern is sequential scan.
This can maximize the disk IO performance.AsOf AsOfDesc BufferChunkExecutor DefaultCreateSource DeleteExecutor DeleteExecutorDistributedLookupJoinExecutor Distributed Lookup Join Executor.
High level Execution flow:
Repeat 1-3: DistributedLookupJoinExecutorBuilder DummyExecutor EquiJoinParams ExecutorBuilder ExecutorBuilderDescriptor Descriptor for executor builder. ExecutorInfo ExpandExecutor FastInsertExecutor A fast insert executor spacially designed for non-pgwire inserts such as websockets and webhooks. FilterExecutor GenericExchangeExecutor GenericExchangeExecutorBuilder GroupTopNExecutor Group Top-N Executor GroupTopNExecutorBuilder HashAggExecutor HashAggExecutor implements the hash aggregate algorithm.HashAggExecutorBuilder HashJoinExecutor Hash Join Executor HopWindowExecutor IcebergScanExecutor IcebergScanExecutorBuilder InsertExecutor InsertExecutorJoinSpillManager LimitExecutor Limit executor. LocalLookupJoinExecutor Local Lookup Join Executor.
High level Execution flow:
Repeat 1-3: LocalLookupJoinExecutorBuilder LookupJoinBase Lookup Join Base.
Used by LocalLookupJoinExecutor and DistributedLookupJoinExecutor. ManagedExecutor ManagedExecutor build on top of the underlying executor. For now, it does two things:MaxOneRowExecutor MergeSortExchangeExecutorBuilder MergeSortExchangeExecutorImpl MergeSortExchangeExecutor2 takes inputs from multiple sources and
The outputs of all the sources have been sorted in the same way.MergeSortExecutor MySqlQueryExecutor MySqlQuery executor. Runs a query against a MySql database.MySqlQueryExecutorBuilder NestedLoopJoinExecutor Nested loop join executor. PostgresQueryExecutor PostgresQuery executor. Runs a query against a Postgres database.PostgresQueryExecutorBuilder ProjectExecutor ProjectSetExecutor RowId Id of one row in chunked data. RowSeqScanExecutor Executor that scans data from row table RowSeqScanExecutorBuilder ScanRange Range for batch scan. SortAggExecutor SortAggExecutor implements the sort aggregate algorithm, which assumes
that the input chunks has already been sorted by group columns.
The aggregation will be applied to tuples within the same group.
And the output schema is [group columns, agg result].SortExecutor Sort Executor SortOverWindowExecutor SortOverWindowExecutorSourceExecutor TableFunctionExecutor TableFunctionExecutorBuilder TopNExecutor Top-N Executor UnionExecutor UpdateExecutor UpdateExecutorValuesExecutor ValuesExecutorWrapStreamExecutor AsOfInequalityType JoinType ProjectSetSelectItem Either a scalar expression or a set-returning function. BUILDER_DESCS All registered executor builders. BoxedExecutorBuilder Every Executor should impl this trait to provide a static method to build a BoxedExecutor
from proto and global environment. CreateSource CreateSource determines the right type of ExchangeSource to create.Executor Refactoring of Executor using Stream. LookupExecutorBuilder Used to build the executor for the inner side batch_read Read at least rows rows. build_scan_range_from_pb build_scan_ranges_from_pb mysql_row_to_owned_row postgres_row_to_owned_row unix_timestamp_sec_to_epoch BoxedDataChunkListStream BoxedDataChunkStream BoxedExecutor BoxedLookupExecutorBuilder ExchangeExecutor JoinHashMap In JoinHashMap, we only save the row id of the first build row that has the hash key.
In fact, in the build side there may be multiple rows with the same hash key. To handle this
case, we use ChunkedData to link them together. For example: MergeSortExchangeExecutor