Struct StreamingDeveloperConfig

pub struct StreamingDeveloperConfig {

    pub enable_executor_row_count: bool,
    pub connector_message_buffer_size: usize,
    pub unsafe_extreme_cache_size: usize,
    pub chunk_size: usize,
    pub exchange_initial_permits: usize,
    pub exchange_batched_permits: usize,
    pub exchange_concurrent_barriers: usize,
    pub exchange_concurrent_dispatchers: usize,
    pub dml_channel_initial_permits: usize,
    pub hash_agg_max_dirty_groups_heap_size: usize,
    pub memory_controller_threshold_aggressive: f64,
    pub memory_controller_threshold_graceful: f64,
    pub memory_controller_threshold_stable: f64,
    pub memory_controller_eviction_factor_aggressive: f64,
    pub memory_controller_eviction_factor_graceful: f64,
    pub memory_controller_eviction_factor_stable: f64,
    pub memory_controller_update_interval_ms: usize,
    pub memory_controller_sequence_tls_step: u64,
    pub memory_controller_sequence_tls_lag: u64,
    pub enable_arrangement_backfill: bool,
    pub high_join_amplification_threshold: usize,
    pub enable_actor_tokio_metrics: bool,
    pub exchange_connection_pool_size: Option<u16>,
    pub enable_auto_schema_change: bool,
    pub enable_shared_source: bool,
    pub switch_jdbc_pg_to_native: bool,
    pub max_barrier_batch_size: u32,
    pub hash_join_entry_state_max_rows: usize,
}

The subsections [streaming.developer].

It is put at StreamingConfig::developer.

Fields

enable_executor_row_count: bool

Set to true to enable per-executor row count metrics. This will produce a lot of timeseries and might affect the prometheus performance. If you only need actor input and output rows data, see stream_actor_in_record_cnt and stream_actor_out_record_cnt instead.

connector_message_buffer_size: usize

The capacity of the chunks in the channel that connects between ConnectorSource and SourceExecutor.

unsafe_extreme_cache_size: usize

Limit number of the cached entries in an extreme aggregation call.

chunk_size: usize

The maximum size of the chunk produced by executor at a time.

exchange_initial_permits: usize

The initial permits that a channel holds, i.e., the maximum row count can be buffered in the channel.

exchange_batched_permits: usize

The permits that are batched to add back, for reducing the backward AddPermits messages in remote exchange.

exchange_concurrent_barriers: usize

The maximum number of concurrent barriers in an exchange channel.

exchange_concurrent_dispatchers: usize

The concurrency for dispatching messages to different downstream jobs.

• 1 means no concurrency, i.e., dispatch messages to downstream jobs one by one.
• 0 means unlimited concurrency.

dml_channel_initial_permits: usize

The initial permits for a dml channel, i.e., the maximum row count can be buffered in the channel.

hash_agg_max_dirty_groups_heap_size: usize

The max heap size of dirty groups of HashAggExecutor.

memory_controller_threshold_aggressive: f64

memory_controller_threshold_graceful: f64

memory_controller_threshold_stable: f64

memory_controller_eviction_factor_aggressive: f64

memory_controller_eviction_factor_graceful: f64

memory_controller_eviction_factor_stable: f64

memory_controller_update_interval_ms: usize

memory_controller_sequence_tls_step: u64

memory_controller_sequence_tls_lag: u64

enable_arrangement_backfill: bool

Enable arrangement backfill If false, the arrangement backfill will be disabled, even if session variable set. If true, it’s decided by session variable streaming_use_arrangement_backfill (default true)

high_join_amplification_threshold: usize

If number of hash join matches exceeds this threshold number, it will be logged.

enable_actor_tokio_metrics: bool

Actor tokio metrics is enabled if enable_actor_tokio_metrics is set or metrics level >= Debug.

exchange_connection_pool_size: Option<u16>

The number of the connections for streaming remote exchange between two nodes. If not specified, the value of server.connection_pool_size will be used.

enable_auto_schema_change: bool

A flag to allow disabling the auto schema change handling

enable_shared_source: bool

Enable shared source If false, the shared source will be disabled, even if session variable set. If true, it’s decided by session variable streaming_use_shared_source (default true)

switch_jdbc_pg_to_native: bool

When true, all jdbc sinks with connector=‘jdbc’ and jdbc.url=“jdbc:postgresql://…” will be switched from jdbc postgresql sinks to rust native (connector=‘postgres’) sinks.

max_barrier_batch_size: u32

The maximum number of consecutive barriers allowed in a message when sent between actors.

hash_join_entry_state_max_rows: usize

Configure the system-wide cache row cardinality of hash join. For example, if this is set to 1000, it means we can have at most 1000 rows in cache.

Implementations

impl StreamingDeveloperConfig

pub fn config_docs( name: String, docs: &mut BTreeMap<String, Vec<(String, String)>>, )

Trait Implementations

impl Clone for StreamingDeveloperConfig

fn clone(&self) -> StreamingDeveloperConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for StreamingDeveloperConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for StreamingDeveloperConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for StreamingDeveloperConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for StreamingDeveloperConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.