Struct DataChunk

pub struct DataChunk {
    columns: Arc<[ArrayRef]>,
    visibility: Bitmap,
}

DataChunk is a collection of Columns, a with visibility mask for each row. For instance, we could have a DataChunk of this format.

v1	v2	v3
1	a	t
2	b	f
3	c	t
4	d	f

Our columns are v1, v2, v3. Then, if the Visibility Mask hides rows 2 and 4, We will only have these rows visible:

v1	v2	v3
1	a	t
3	c	t

Fields

columns: Arc<[ArrayRef]>

visibility: Bitmap

Implementations

impl DataChunk

pub(crate) const PRETTY_TABLE_PRESET: &'static str = "||--+-++| ++++++"

pub fn new(columns: Vec<ArrayRef>, visibility: impl Into<Bitmap>) -> Self

Create a DataChunk with columns and visibility.

The visibility can either be a Bitmap or a simple cardinality number.

pub fn new_dummy(cardinality: usize) -> Self

new_dummy creates a data chunk without columns but only a cardinality.

pub fn from_rows(rows: &[impl Row], data_types: &[DataType]) -> Self

Build a DataChunk with rows.

Panics if the rows is empty.

Should prefer using DataChunkBuilder instead to avoid unnecessary allocation of rows.

pub fn next_visible_row_idx(&self, row_idx: usize) -> Option<usize>

Return the next visible row index on or after row_idx.

pub fn into_parts(self) -> (Vec<ArrayRef>, Bitmap)

pub fn into_parts_v2(self) -> (Arc<[ArrayRef]>, Bitmap)

pub fn from_parts(columns: Arc<[ArrayRef]>, visibilities: Bitmap) -> Self

pub fn dimension(&self) -> usize

pub fn cardinality(&self) -> usize

cardinality returns the number of visible tuples

pub fn capacity(&self) -> usize

capacity returns physical length of any chunk column

pub fn selectivity(&self) -> f64

pub fn with_visibility(&self, visibility: impl Into<Bitmap>) -> Self

pub fn visibility(&self) -> &Bitmap

pub fn set_visibility(&mut self, visibility: Bitmap)

pub fn is_compacted(&self) -> bool

pub fn column_at(&self, idx: usize) -> &ArrayRef

pub fn columns(&self) -> &[ArrayRef]

pub fn data_types(&self) -> Vec<DataType>

Returns the data types of all columns.

pub fn split_column_at(&self, idx: usize) -> (Self, Self)

Divides one chunk into two at an column index.

Panics

Panics if idx > columns.len().

pub fn to_protobuf(&self) -> PbDataChunk

pub fn compact(self) -> Self

compact will convert the chunk to compact format. Compacting removes the hidden rows, and returns a new visibility mask which indicates this.

compact has trade-offs:

Cost: It has to rebuild the each column, meaning it will incur cost of copying over bytes from the original column array to the new one.

Benefit: The main benefit is that the data chunk is smaller, taking up less memory. We can also save the cost of iterating over many hidden rows.

pub fn uncompact(self, vis: Bitmap) -> Self

Scatter a compacted chunk to a new chunk with the given visibility.

pub fn compact_cow(&self) -> Cow<'_, Self>

Convert the chunk to compact format.

If the chunk is not compacted, return a new compacted chunk, otherwise return a reference to self.

pub fn from_protobuf(proto: &PbDataChunk) -> ArrayResult<Self>

pub fn rechunk( chunks: &[DataChunk], each_size_limit: usize, ) -> ArrayResult<Vec<DataChunk>>

rechunk creates a new vector of data chunk whose size is each_size_limit. When the total cardinality of all the chunks is not evenly divided by the each_size_limit, the last new chunk will be the remainder.

pub fn get_hash_values<H: BuildHasher>( &self, column_idxes: &[usize], hasher_builder: H, ) -> Vec<HashCode<H>>

Compute hash values for each row. The number of the returning HashCodes is self.capacity(). When skip_invisible_row is true, the HashCode for the invisible rows is arbitrary.

pub fn row_at(&self, pos: usize) -> (RowRef<'_>, bool)

Random access a tuple in a data chunk. Return in a row format.

Arguments

• pos - Index of look up tuple
• RowRef - Reference of data tuple
• bool - whether this tuple is visible

pub fn row_at_unchecked_vis(&self, pos: usize) -> RowRef<'_>

Random access a tuple in a data chunk. Return in a row format. Note that this function do not return whether the row is visible.

Arguments

• pos - Index of look up tuple

pub fn to_pretty(&self) -> impl Display

Returns a table-like text representation of the DataChunk.

pub fn keep_columns(&self, column_indices: &[usize]) -> Self

Keep the specified columns and set the rest elements to null.

Example

i i i                            i i i
1 2 3  --> keep_columns([1]) --> . 2 .
4 5 6                            . 5 .

pub fn project(&self, indices: &[usize]) -> Self

Reorder (and possibly remove) columns.

e.g. if indices is [2, 1, 0], and the chunk contains column [a, b, c], then the output will be [c, b, a]. If indices is [2, 0], then the output will be [c, a]. If the input mapping is identity mapping, no reorder will be performed.

pub fn project_with_vis(&self, indices: &[usize], visibility: Bitmap) -> Self

Reorder columns and set visibility.

pub fn reorder_rows(&self, indexes: &[usize]) -> Self

Reorder rows by indexes.

fn partition_sizes(&self) -> (usize, Vec<&ArrayRef>)

Partition fixed size datums and variable length ones.

In some cases, we have fixed size for the entire column, when the datatypes are fixed size or the datums are constants. As such we can compute the size for it just once for the column.

Otherwise, for variable sized datatypes, such as varchar, we have to individually compute their sizes per row.

unsafe fn compute_size_of_variable_cols_in_row( variable_cols: &[&ArrayRef], row_idx: usize, ) -> usize

unsafe fn init_buffer( row_len_fixed: usize, variable_cols: &[&ArrayRef], row_idx: usize, ) -> Vec<u8>

pub fn serialize(&self) -> Vec<Bytes>

Serialize each row into value encoding bytes.

The returned vector’s size is self.capacity() and for the invisible row will give a empty bytes.

pub fn serialize_with(&self, serializer: &impl ValueRowSerializer) -> Vec<Bytes>

Serialize each row into bytes with given serializer.

This is similar to serialize but it uses a custom serializer. Prefer serialize if possible since it might be more efficient due to columnar operations.

pub fn estimate_value_encoding_size(&self, column_indices: &[usize]) -> usize

Estimate size of hash keys. Their indices in a row are indicated by column_indices. Size here refers to the number of u8s required to store the serialized datum.

impl DataChunk

pub fn rows(&self) -> DataChunkRefIter<'_>

Get an iterator for visible rows.

pub fn rows_in(&self, range: Range<usize>) -> DataChunkRefIter<'_>

Get an iterator for visible rows in range.

pub fn rows_with_holes(&self) -> DataChunkRefIterWithHoles<'_>

Get an iterator for all rows in the chunk, and a None represents an invisible row.

Trait Implementations

impl Clone for DataChunk

fn clone(&self) -> DataChunk

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl DataChunkTestExt for DataChunk

fn from_pretty(s: &str) -> Self

Parse a chunk from string.

fn with_invisible_holes(self) -> Self

where Self: Sized,

Insert one invisible hole after every record.

fn assert_valid(&self)

Panic if the chunk is invalid.

fn gen_data_chunk( chunk_offset: usize, chunk_size: usize, data_types: &[DataType], varchar_properties: &VarcharProperty, visibility_percent: f64, ) -> Self

Generate data chunk when supplied with chunk_size and column data types.

fn gen_data_chunks( num_of_chunks: usize, chunk_size: usize, data_types: &[DataType], varchar_properties: &VarcharProperty, visibility_percent: f64, ) -> Vec<Self>

Generate data chunks when supplied with chunk_size and column data types.

const SEED: u64 = 18_403_958_415_801_939_711u64

SEED for generating data chunk.

impl Debug for DataChunk

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl EstimateSize for DataChunk

fn estimated_heap_size(&self) -> usize

The estimated heap size of the current struct in bytes.

fn estimated_size(&self) -> usize

where Self: Sized,

The estimated total size of the current struct in bytes, including the estimated_heap_size and the size of Self.

impl<'a> From<&'a StructArray> for DataChunk

fn from(array: &'a StructArray) -> Self

Converts to this type from the input type.

impl From<DataChunk> for StreamChunk

StreamChunk can be created from DataChunk with all operations set to Insert.

fn from(data: DataChunk) -> Self

Converts to this type from the input type.

impl From<DataChunk> for StructArray

fn from(chunk: DataChunk) -> Self

Converts to this type from the input type.

impl PartialEq for DataChunk

fn eq(&self, other: &DataChunk) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl __ExtTrait9496273680842896069 for DataChunk

fn estimate_hash_key_sizes(&self, column_indices: &[usize]) -> Vec<usize>

impl StructuralPartialEq for DataChunk