Struct IcebergSplitEnumerator

pub struct IcebergSplitEnumerator {
    config: IcebergProperties,
}

Fields

config: IcebergProperties

Implementations

impl IcebergSplitEnumerator

pub fn new_inner( properties: IcebergProperties, _context: SourceEnumeratorContextRef, ) -> Self

impl IcebergSplitEnumerator

pub fn get_snapshot_id( table: &Table, time_travel_info: Option<IcebergTimeTravelInfo>, ) -> ConnectorResult<Option<i64>>

pub async fn list_splits_batch( &self, schema: Schema, time_traval_info: Option<IcebergTimeTravelInfo>, batch_parallelism: usize, iceberg_scan_type: IcebergScanType, predicate: IcebergPredicate, ) -> ConnectorResult<Vec<IcebergSplit>>

async fn list_splits_batch_scan( &self, table: &Table, snapshot_id: i64, schema: Schema, batch_parallelism: usize, iceberg_scan_type: IcebergScanType, predicate: IcebergPredicate, ) -> ConnectorResult<Vec<IcebergSplit>>

pub async fn list_splits_batch_count_star( &self, table: &Table, snapshot_id: i64, ) -> ConnectorResult<Vec<IcebergSplit>>

pub async fn all_delete_parameters( table: &Table, snapshot_id: i64, ) -> ConnectorResult<(Vec<String>, bool)>

List all files in the snapshot to check if there are deletes.

pub async fn get_delete_parameters( &self, time_travel_info: Option<IcebergTimeTravelInfo>, ) -> ConnectorResult<(Vec<String>, bool)>

fn split_n_vecs( file_scan_tasks: Vec<FileScanTask>, split_num: usize, ) -> Vec<Vec<FileScanTask>>

Uniformly distribute scan tasks to compute nodes. It’s deterministic so that it can best utilize the data locality.

Arguments

• file_scan_tasks: The file scan tasks to be split.
• split_num: The number of splits to be created.

This algorithm is based on a min-heap. It will push all groups into the heap, and then pop the smallest group and add the file scan task to it. Ensure that the total length of each group is as balanced as possible. The time complexity is O(n log k), where n is the number of file scan tasks and k is the number of splits. The space complexity is O(k), where k is the number of splits. The algorithm is stable, so the order of the file scan tasks will be preserved.

Trait Implementations

impl Clone for IcebergSplitEnumerator

fn clone(&self) -> IcebergSplitEnumerator

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for IcebergSplitEnumerator

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl SplitEnumerator for IcebergSplitEnumerator

type Properties = IcebergProperties

type Split = IcebergSplit

fn new<'async_trait>( properties: Self::Properties, context: SourceEnumeratorContextRef, ) -> Pin<Box<dyn Future<Output = ConnectorResult<Self>> + Send + 'async_trait>>

where Self: 'async_trait,

fn list_splits<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = ConnectorResult<Vec<Self::Split>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

fn on_drop_fragments<'life0, 'async_trait>( &'life0 mut self, _fragment_ids: Vec<u32>, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Do some cleanup work when a fragment is dropped, e.g., drop Kafka consumer group.

fn on_finish_backfill<'life0, 'async_trait>( &'life0 mut self, _fragment_ids: Vec<u32>, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Do some cleanup work when a backfill fragment is finished, e.g., drop Kafka consumer group.