Struct StreamingDeveloperConfig 

pub struct StreamingDeveloperConfig {

    pub enable_executor_row_count: bool,
    pub connector_message_buffer_size: usize,
    pub unsafe_extreme_cache_size: usize,
    pub topn_cache_min_capacity: usize,
    pub chunk_size: usize,
    pub exchange_initial_permits: usize,
    pub exchange_batched_permits: usize,
    pub exchange_concurrent_barriers: usize,
    pub exchange_concurrent_dispatchers: usize,
    pub dml_channel_initial_permits: usize,
    pub hash_agg_max_dirty_groups_heap_size: usize,
    pub memory_controller_threshold_aggressive: f64,
    pub memory_controller_threshold_graceful: f64,
    pub memory_controller_threshold_stable: f64,
    pub memory_controller_eviction_factor_aggressive: f64,
    pub memory_controller_eviction_factor_graceful: f64,
    pub memory_controller_eviction_factor_stable: f64,
    pub memory_controller_update_interval_ms: usize,
    pub memory_controller_sequence_tls_step: u64,
    pub memory_controller_sequence_tls_lag: u64,
    pub enable_arrangement_backfill: bool,
    pub high_join_amplification_threshold: usize,
    pub enable_actor_tokio_metrics: bool,
    pub(super) exchange_connection_pool_size: Option<u16>,
    pub enable_auto_schema_change: bool,
    pub enable_shared_source: bool,
    pub switch_jdbc_pg_to_native: bool,
    pub max_barrier_batch_size: u32,
    pub hash_join_entry_state_max_rows: usize,
    pub now_progress_ratio: Option<f32>,
    pub enable_explain_analyze_stats: bool,
    pub compute_client_config: RpcClientConfig,
    pub iceberg_list_interval_sec: u64,
    pub iceberg_fetch_batch_size: u64,
    pub iceberg_sink_positional_delete_cache_size: usize,
    pub iceberg_sink_write_parquet_max_row_group_rows: usize,
    pub default_enable_mem_preload_state_table: bool,
    pub mem_preload_state_table_ids_whitelist: Vec<u32>,
    pub mem_preload_state_table_ids_blacklist: Vec<u32>,
    pub aggressive_noop_update_elimination: bool,
}

The subsections [streaming.developer].

It is put at StreamingConfig::developer.

Fields

enable_executor_row_count: bool

Set to true to enable per-executor row count metrics. This will produce a lot of timeseries and might affect the prometheus performance. If you only need actor input and output rows data, see stream_actor_in_record_cnt and stream_actor_out_record_cnt instead.

connector_message_buffer_size: usize

The capacity of the chunks in the channel that connects between ConnectorSource and SourceExecutor.

unsafe_extreme_cache_size: usize

Limit number of the cached entries in an extreme aggregation call.

topn_cache_min_capacity: usize

Minimum cache size for TopN cache per group key.

chunk_size: usize

The maximum size of the chunk produced by executor at a time.

exchange_initial_permits: usize

The initial permits that a channel holds, i.e., the maximum row count can be buffered in the channel.

exchange_batched_permits: usize

The permits that are batched to add back, for reducing the backward AddPermits messages in remote exchange.

exchange_concurrent_barriers: usize

The maximum number of concurrent barriers in an exchange channel.

exchange_concurrent_dispatchers: usize

The concurrency for dispatching messages to different downstream jobs.

• 1 means no concurrency, i.e., dispatch messages to downstream jobs one by one.
• 0 means unlimited concurrency.

dml_channel_initial_permits: usize

The initial permits for a dml channel, i.e., the maximum row count can be buffered in the channel.

hash_agg_max_dirty_groups_heap_size: usize

The max heap size of dirty groups of HashAggExecutor.

memory_controller_threshold_aggressive: f64

memory_controller_threshold_graceful: f64

memory_controller_threshold_stable: f64

memory_controller_eviction_factor_aggressive: f64

memory_controller_eviction_factor_graceful: f64

memory_controller_eviction_factor_stable: f64

memory_controller_update_interval_ms: usize

memory_controller_sequence_tls_step: u64

memory_controller_sequence_tls_lag: u64

enable_arrangement_backfill: bool

Enable arrangement backfill If false, the arrangement backfill will be disabled, even if session variable set. If true, it’s decided by session variable streaming_use_arrangement_backfill (default true)

high_join_amplification_threshold: usize

If number of hash join matches exceeds this threshold number, it will be logged.

enable_actor_tokio_metrics: bool

Actor tokio metrics is enabled if enable_actor_tokio_metrics is set or metrics level >= Debug.

exchange_connection_pool_size: Option<u16>

The number of the connections for streaming remote exchange between two nodes. If not specified, the value of server.connection_pool_size will be used.

enable_auto_schema_change: bool

A flag to allow disabling the auto schema change handling

enable_shared_source: bool

Enable shared source If false, the shared source will be disabled, even if session variable set. If true, it’s decided by session variable streaming_use_shared_source (default true)

switch_jdbc_pg_to_native: bool

When true, all jdbc sinks with connector=‘jdbc’ and jdbc.url=“jdbc:postgresql://…” will be switched from jdbc postgresql sinks to rust native (connector=‘postgres’) sinks.

max_barrier_batch_size: u32

The maximum number of consecutive barriers allowed in a message when sent between actors.

hash_join_entry_state_max_rows: usize

Configure the system-wide cache row cardinality of hash join. For example, if this is set to 1000, it means we can have at most 1000 rows in cache.

now_progress_ratio: Option<f32>

enable_explain_analyze_stats: bool

Enable / Disable profiling stats used by EXPLAIN ANALYZE

compute_client_config: RpcClientConfig

iceberg_list_interval_sec: u64

IcebergListExecutor: The interval in seconds for Iceberg source to list new files.

iceberg_fetch_batch_size: u64

IcebergFetchExecutor: The number of files the executor will fetch concurrently in a batch.

iceberg_sink_positional_delete_cache_size: usize

IcebergSink: The size of the cache for positional delete in the sink.

iceberg_sink_write_parquet_max_row_group_rows: usize

IcebergSink: The maximum number of rows in a row group when writing Parquet files.

default_enable_mem_preload_state_table: bool

Whether by default enable preloading all rows in memory for state table. If true, all capable state tables will preload its state to memory

mem_preload_state_table_ids_whitelist: Vec<u32>

The list of state table ids to enable preloading all rows in memory for state table. Only takes effect when default_enable_mem_preload_state_table is false.

mem_preload_state_table_ids_blacklist: Vec<u32>

The list of state table ids to disable preloading all rows in memory for state table. Only takes effect when default_enable_mem_preload_state_table is true.

aggressive_noop_update_elimination: bool

Eliminate unnecessary updates aggressively, even if it impacts performance. Enable this only if it’s confirmed that no-op updates are causing significant streaming amplification.

Implementations

impl StreamingDeveloperConfig

pub fn config_docs( name: String, docs: &mut BTreeMap<String, Vec<(String, String)>>, )

Trait Implementations

impl Clone for StreamingDeveloperConfig

fn clone(&self) -> StreamingDeveloperConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for StreamingDeveloperConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for StreamingDeveloperConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for StreamingDeveloperConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for StreamingDeveloperConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.