Struct SourceStreamChunkBuilder

pub struct SourceStreamChunkBuilder {
    column_descs: Vec<SourceColumnDesc>,
    source_ctrl_opts: SourceCtrlOpts,
    builders: Vec<ArrayBuilderImpl>,
    op_builder: Vec<Op>,
    vis_builder: BitmapBuilder,
    ongoing_txn: Option<Transaction>,
    ready_chunks: SmallVec<[StreamChunk; 1]>,
}

A builder for building a StreamChunk from SourceColumnDesc.

Output chunk size is controlled by source_ctrl_opts.chunk_size and source_ctrl_opts.split_txn. During building process, it’s possible that multiple chunks are built even without any explicit call to finish_current_chunk. This mainly happens when we find more than one records in one SourceMessage when parsing it. User of this builder should call consume_ready_chunks to consume the built chunks from time to time, to avoid the buffer from growing too large.

Fields

column_descs: Vec<SourceColumnDesc>

source_ctrl_opts: SourceCtrlOpts

builders: Vec<ArrayBuilderImpl>

op_builder: Vec<Op>

vis_builder: BitmapBuilder

ongoing_txn: Option<Transaction>

ready_chunks: SmallVec<[StreamChunk; 1]>

Implementations

impl SourceStreamChunkBuilder

pub fn new( column_descs: Vec<SourceColumnDesc>, source_ctrl_opts: SourceCtrlOpts, ) -> Self

fn create_builders( column_descs: &[SourceColumnDesc], chunk_size: usize, ) -> (Vec<ArrayBuilderImpl>, Vec<Op>, BitmapBuilder)

pub fn begin_transaction(&mut self, txn_id: Box<str>)

Begin a (CDC) transaction with the given txn_id.

pub fn commit_transaction(&mut self, txn_id: Box<str>)

Commit the ongoing transaction with the given txn_id.

pub fn is_in_transaction(&self) -> bool

Check if the builder is in an ongoing transaction.

pub fn row_writer(&mut self) -> SourceStreamChunkRowWriter<'_>

Get a row writer for parser to write records to the builder.

pub fn heartbeat(&mut self, meta: MessageMeta<'_>)

Write a heartbeat record to the builder. The builder will decide whether to finish the current chunk or not. Currently it ensures that heartbeats are always in separate chunks.

pub fn finish_current_chunk(&mut self)

Finish and build a StreamChunk from the current pending records in the builder, no matter whether the builder is in a transaction or not, split_txn or not. The built chunk will be appended to the ready_chunks and the builder will be reset.

pub fn consume_ready_chunks( &mut self, ) -> impl ExactSizeIterator<Item = StreamChunk> + '_

Consumes and returns the ready StreamChunks.

fn current_chunk_len(&self) -> usize

fn commit_record(&mut self, op: Op, vis: bool)

Commit a newly-written record by appending op and vis to the corresponding builders. This is supposed to be called via the row_writer only.