Module base

Structs

Column

CreateSplitReaderOpt

CreateSplitReaderResult

SourceContext

SourceCtrlOpts

SourceEnumeratorContext

SourceEnumeratorInfo

SourceMessage

The message pumped from the external source service. The third-party message structs will eventually be transformed into this struct.

SourceStruct

Enums

BackfillInfo

Information used to determine whether we should start and finish source backfill.

ConnectorProperties

SourceEncode

Refer to crate::parser::EncodingProperties

SourceFormat

SourceMeta

SplitImpl

Constants

MAX_CHUNK_SIZE

The max size of a chunk yielded by source stream.

SPLIT_INFO_FIELD 🔒

SPLIT_TYPE_FIELD 🔒

UPSTREAM_SOURCE_KEY

WEBHOOK_CONNECTOR

Traits

AnySplitEnumerator

Dyn-compatible SplitEnumerator.

SourceChunkStream

SourceProperties

Represents WITH options for sources.

SplitEnumerator

SplitEnumerator fetches the split metadata from the external source service. NOTE: It runs in the meta server, so probably it should be moved to the meta crate.

SplitMetaData

The metadata of a split.

SplitReader

SplitReader is a new abstraction of the external connector read interface which is responsible for parsing, it is used to read messages from the outside and transform them into a stream of parsed [StreamChunk]

TryFromBTreeMap

UnknownFields

Functions

build_connection

create_split_readers

enforce_secret_connection

extract_source_struct

Type Aliases

BoxSourceChunkStream

Stream of [StreamChunk]s parsed from the messages from the external source.

BoxSourceChunkWithStateStream

BoxSourceMessageStream

Stream of SourceMessage. Messages flow through the stream in the unit of a batch.

BoxStreamingFileSourceChunkStream

Stream of Option<StreamChunk>s parsed from the messages from the external source.

BoxTryStream

ConnectorState

ConnectorState maintains the consuming splits’ info. In specific split readers, ConnectorState cannot be None and contains one(for mq split readers) or many(for fs split readers) SplitImpl. If no split is assigned to source executor, ConnectorState is None and the created source stream will be a pending stream.

SourceContextRef

SourceEnumeratorContextRef

SplitId

Split id resides in every source message, use Arc to avoid copying.

StreamChunkWithState