Struct BatchPosixFsSplit

pub struct BatchPosixFsSplit {
    pub file_path: String,
    pub split_id: SplitId,
    pub finished: bool,
}

Batch Posix fs source for refreshable tables. (For testing only) Unlike regular posix_fs, this connector only lists files on demand (during refresh), not continuously. This makes it suitable for refreshable table functionality.

Split representing a single file to be read once

Fields

file_path: String

For batch posix fs, this is always the root directory. The reader will scan all files in this directory.

split_id: SplitId

A unique identifier for the split, typically including a timestamp to force refresh.

finished: bool

Whether this split has finished reading all data (used for batch sources) See BatchSourceSplit for details about recovery.

Implementations

impl BatchPosixFsSplit

pub fn new(file_path: String, split_id: SplitId) -> Self

pub fn mark_finished(&mut self)

Trait Implementations

impl BatchSourceSplit for BatchPosixFsSplit

fn finished(&self) -> bool

fn finish(&mut self)

Mark the source as finished. Called after the source is exhausted. Then SourceExecutor will report to meta to send a LoadFinish barrier, and the RefreshableMaterialize will begin to calculate the diff.

fn refresh(&mut self)

Refresh the source to make it ready for re-run. Called when receiving RefreshStart barrier.

impl Clone for BatchPosixFsSplit

fn clone(&self) -> BatchPosixFsSplit

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BatchPosixFsSplit

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for BatchPosixFsSplit

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<BatchPosixFsSplit> for SplitImpl

fn from(split: BatchPosixFsSplit) -> SplitImpl

Converts to this type from the input type.

impl Hash for BatchPosixFsSplit

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for BatchPosixFsSplit

fn eq(&self, other: &BatchPosixFsSplit) -> bool

Tests for self and other values to be equal, and is used by ==.

const fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for BatchPosixFsSplit

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl SplitMetaData for BatchPosixFsSplit

fn id(&self) -> SplitId

fn restore_from_json(value: JsonbVal) -> ConnectorResult<Self>

fn encode_to_json(&self) -> JsonbVal

Encode the whole split metadata to a JSON object

fn update_offset(&mut self, _last_seen_offset: String) -> ConnectorResult<()>

fn encode_to_bytes(&self) -> Bytes

fn restore_from_bytes(bytes: &[u8]) -> Result<Self>

impl TryFrom<SplitImpl> for BatchPosixFsSplit

type Error = ConnectorError

The type returned in the event of a conversion error.

fn try_from(split: SplitImpl) -> Result<Self, Self::Error>

Performs the conversion.

impl Eq for BatchPosixFsSplit

impl StructuralPartialEq for BatchPosixFsSplit