Module base

Structs

• Column
• FsFilterCtrlCtx
• SourceContext
• SourceCtrlOpts
• SourceEnumeratorContext
• SourceEnumeratorInfo
• SourceMessage
  The message pumped from the external source service. The third-party message structs will eventually be transformed into this struct.
• SourceStruct

Enums

• BackfillInfo
  Information used to determine whether we should start and finish source backfill.
• ConnectorProperties
• SourceEncode
  Refer to crate::parser::EncodingProperties
• SourceFormat
• SourceMeta
• SplitImpl

Constants

• MAX_CHUNK_SIZE
  The max size of a chunk yielded by source stream.
• SPLIT_INFO_FIELD 🔒
• SPLIT_TYPE_FIELD 🔒
• UPSTREAM_SOURCE_KEY

Traits

• ChunkSourceStream
• FsListInner
• SourceProperties
  Represents WITH options for sources.
• SplitEnumerator
  SplitEnumerator fetches the split metadata from the external source service. NOTE: It runs in the meta server, so probably it should be moved to the meta crate.
• SplitMetaData
  The metadata of a split.
• SplitReader
  SplitReader is a new abstraction of the external connector read interface which is responsible for parsing, it is used to read messages from the outside and transform them into a stream of parsed StreamChunk
• TryFromBTreeMap
• UnknownFields

Functions

• create_split_reader
• extract_source_struct

Type Aliases

• BoxChunkSourceStream
• BoxSourceStream
  Stream of SourceMessage.
• BoxTryStream
• ConnectorState
  ConnectorState maintains the consuming splits’ info. In specific split readers, ConnectorState cannot be None and contains one(for mq split readers) or many(for fs split readers) SplitImpl. If no split is assigned to source executor, ConnectorState is None and the created source stream will be a pending stream.
• DataType
• FsFilterCtrlCtxRef
• SourceContextRef
• SourceEnumeratorContextRef
• SplitId
  Split id resides in every source message, use Arc to avoid copying.