Struct StreamingDeveloperConfig

pub struct StreamingDeveloperConfig {

    pub enable_executor_row_count: bool,
    pub connector_message_buffer_size: usize,
    pub unsafe_extreme_cache_size: usize,
    pub chunk_size: usize,
    pub exchange_initial_permits: usize,
    pub exchange_batched_permits: usize,
    pub exchange_concurrent_barriers: usize,
    pub exchange_concurrent_dispatchers: usize,
    pub dml_channel_initial_permits: usize,
    pub hash_agg_max_dirty_groups_heap_size: usize,
    pub memory_controller_threshold_aggressive: f64,
    pub memory_controller_threshold_graceful: f64,
    pub memory_controller_threshold_stable: f64,
    pub memory_controller_eviction_factor_aggressive: f64,
    pub memory_controller_eviction_factor_graceful: f64,
    pub memory_controller_eviction_factor_stable: f64,
    pub memory_controller_update_interval_ms: usize,
    pub memory_controller_sequence_tls_step: u64,
    pub memory_controller_sequence_tls_lag: u64,
    pub enable_arrangement_backfill: bool,
    pub high_join_amplification_threshold: usize,
    pub enable_actor_tokio_metrics: bool,
    pub exchange_connection_pool_size: Option<u16>,
    pub enable_auto_schema_change: bool,
    pub enable_shared_source: bool,
}

The subsections [streaming.developer].

It is put at StreamingConfig::developer.

Fields

enable_executor_row_count: bool

Set to true to enable per-executor row count metrics. This will produce a lot of timeseries and might affect the prometheus performance. If you only need actor input and output rows data, see stream_actor_in_record_cnt and stream_actor_out_record_cnt instead.

connector_message_buffer_size: usize

The capacity of the chunks in the channel that connects between ConnectorSource and SourceExecutor.

unsafe_extreme_cache_size: usize

Limit number of the cached entries in an extreme aggregation call.

chunk_size: usize

The maximum size of the chunk produced by executor at a time.

exchange_initial_permits: usize

The initial permits that a channel holds, i.e., the maximum row count can be buffered in the channel.

exchange_batched_permits: usize

The permits that are batched to add back, for reducing the backward AddPermits messages in remote exchange.

exchange_concurrent_barriers: usize

The maximum number of concurrent barriers in an exchange channel.

exchange_concurrent_dispatchers: usize

The concurrency for dispatching messages to different downstream jobs.

• 1 means no concurrency, i.e., dispatch messages to downstream jobs one by one.
• 0 means unlimited concurrency.

dml_channel_initial_permits: usize

The initial permits for a dml channel, i.e., the maximum row count can be buffered in the channel.

hash_agg_max_dirty_groups_heap_size: usize

The max heap size of dirty groups of HashAggExecutor.

memory_controller_threshold_aggressive: f64

memory_controller_threshold_graceful: f64

memory_controller_threshold_stable: f64

memory_controller_eviction_factor_aggressive: f64

memory_controller_eviction_factor_graceful: f64

memory_controller_eviction_factor_stable: f64

memory_controller_update_interval_ms: usize

memory_controller_sequence_tls_step: u64

memory_controller_sequence_tls_lag: u64

enable_arrangement_backfill: bool

Enable arrangement backfill If false, the arrangement backfill will be disabled, even if session variable set. If true, it’s decided by session variable streaming_use_arrangement_backfill (default true)

high_join_amplification_threshold: usize

If number of hash join matches exceeds this threshold number, it will be logged.

enable_actor_tokio_metrics: bool

Actor tokio metrics is enabled if enable_actor_tokio_metrics is set or metrics level >= Debug.

exchange_connection_pool_size: Option<u16>

The number of the connections for streaming remote exchange between two nodes. If not specified, the value of server.connection_pool_size will be used.

enable_auto_schema_change: bool

A flag to allow disabling the auto schema change handling

enable_shared_source: bool

Enable shared source If false, the shared source will be disabled, even if session variable set. If true, it’s decided by session variable streaming_use_shared_source (default true)

Implementations

impl StreamingDeveloperConfig

pub fn config_docs( name: String, docs: &mut BTreeMap<String, Vec<(String, String)>>, )

Trait Implementations

impl Clone for StreamingDeveloperConfig

fn clone(&self) -> StreamingDeveloperConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for StreamingDeveloperConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for StreamingDeveloperConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for StreamingDeveloperConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for StreamingDeveloperConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.