Struct ParquetParser

pub struct ParquetParser {
    rw_columns: Vec<SourceColumnDesc>,
    file_name: String,
    offset: usize,
}

ParquetParser is responsible for converting the incoming record_batch_stream into a streamChunk.

Fields

rw_columns: Vec<SourceColumnDesc>

file_name: String

offset: usize

Implementations

impl ParquetParser

pub fn new( rw_columns: Vec<SourceColumnDesc>, file_name: String, offset: usize, ) -> ConnectorResult<Self>

pub fn into_stream( self, record_batch_stream: ParquetRecordBatchStream<Compat<FuturesAsyncReader>>, ) -> Pin<Box<dyn Stream<Item = Result<StreamChunk, ConnectorError>> + Send>>

fn inc_offset(&mut self)

fn convert_record_batch_to_stream_chunk( &mut self, record_batch: RecordBatch, ) -> Result<StreamChunk, ConnectorError>

The function convert_record_batch_to_stream_chunk is designed to transform the given RecordBatch into a StreamChunk.

For each column in the source column:

• If the column’s schema matches a column in the RecordBatch (both the data type and column name are the same), the corresponding records are converted into a column of the StreamChunk.
• If the column’s schema does not match, null values are inserted.
• Hidden columns are handled separately by filling in the appropriate fields to ensure the data chunk maintains the correct format.
• If a column in the Parquet file does not exist in the source schema, it is skipped.

Arguments

• record_batch - The RecordBatch to be converted into a StreamChunk.

Returns

A StreamChunk containing the converted data from the RecordBatch.

The hidden columns that must be included here are _rw_file and _rw_offset. Depending on whether the user specifies a primary key (pk), there may be an additional hidden column row_id. Therefore, the maximum number of hidden columns is three.

Trait Implementations

impl Debug for ParquetParser

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.