Struct PubsubProperties

pub struct PubsubProperties {
    pub subscription: String,
    pub emulator_host: Option<String>,
    pub credentials: Option<String>,
    pub start_offset: Option<String>,
    pub start_snapshot: Option<String>,
    pub parallelism: Option<u32>,
    pub unknown_fields: HashMap<String, String>,
}

Implementation Notes

Pub/Sub does not rely on persisted state (SplitImpl) to start from a position. It rely on Pub/Sub to load-balance messages between all Readers. We ack received messages after checkpoint (see WaitCheckpointWorker) to achieve at-least-once delivery.

Fields

subscription: String

Pub/Sub subscription to consume messages from.

Note that we rely on Pub/Sub to load-balance messages between all Readers pulling from the same subscription. So one subscription (i.e., one Source) can only used for one MV (shared between the actors of its fragment). Otherwise, different MVs on the same Source will both receive part of the messages. TODO: check and enforce this on Meta.

emulator_host: Option<String>

use the connector with a pubsub emulator https://cloud.google.com/pubsub/docs/emulator

credentials: Option<String>

credentials is a JSON string containing the service account credentials. See the service-account credentials guide. The service account must have the pubsub.subscriber role.

start_offset: Option<String>

start_offset is a numeric timestamp, ideally the publish timestamp of a message in the subscription. If present, the connector will attempt to seek the subscription to the timestamp and start consuming from there. Note that the seek operation is subject to limitations around the message retention policy of the subscription. See Seeking to a timestamp for more details.

start_snapshot: Option<String>

start_snapshot is a named pub/sub snapshot. If present, the connector will first seek to the snapshot before starting consumption. Snapshots are the preferred seeking mechanism in pub/sub because they guarantee retention of:

• All unacknowledged messages at the time of their creation.
• All messages created after their creation. Besides retention guarantees, snapshots are also more precise than timestamp-based seeks. See Seeking to a snapshot for more details.

parallelism: Option<u32>

parallelism is the number of parallel consumers to run for the subscription. TODO: use system parallelism if not set

unknown_fields: HashMap<String, String>

Implementations

impl PubsubProperties

pub(crate) fn initialize_env(&self)

initialize_env sets environment variables read by the google-cloud-pubsub crate

pub(crate) async fn subscription_client(&self) -> ConnectorResult<Subscription>

Trait Implementations

impl Clone for PubsubProperties

fn clone(&self) -> PubsubProperties

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PubsubProperties

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for PubsubProperties

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<PubsubProperties> for ConnectorProperties

fn from(prop: PubsubProperties) -> ConnectorProperties

Converts to this type from the input type.

impl SourceProperties for PubsubProperties

const SOURCE_NAME: &'static str = GOOGLE_PUBSUB_CONNECTOR

type Split = PubsubSplit

type SplitEnumerator = PubsubSplitEnumerator

type SplitReader = PubsubSplitReader

fn init_from_pb_source(&mut self, _source: &PbSource)

Load additional info from PbSource. Currently only used by CDC.

fn init_from_pb_cdc_table_desc(&mut self, _table_desc: &ExternalTableDesc)

Load additional info from ExternalTableDesc. Currently only used by CDC.

impl UnknownFields for PubsubProperties

fn unknown_fields(&self) -> HashMap<String, String>

Unrecognized fields in the WITH clause.

impl WithOptions for PubsubProperties