Struct StreamingDeveloperConfig 

pub struct StreamingDeveloperConfig {

    pub enable_executor_row_count: bool,
    pub connector_message_buffer_size: usize,
    pub unsafe_extreme_cache_size: usize,
    pub topn_cache_min_capacity: usize,
    pub chunk_size: usize,
    pub exchange_initial_permits: usize,
    pub exchange_batched_permits: usize,
    pub exchange_concurrent_barriers: usize,
    pub exchange_concurrent_dispatchers: usize,
    pub dml_channel_initial_permits: usize,
    pub hash_agg_max_dirty_groups_heap_size: usize,
    pub memory_controller_threshold_aggressive: f64,
    pub memory_controller_threshold_graceful: f64,
    pub memory_controller_threshold_stable: f64,
    pub memory_controller_eviction_factor_aggressive: f64,
    pub memory_controller_eviction_factor_graceful: f64,
    pub memory_controller_eviction_factor_stable: f64,
    pub memory_controller_update_interval_ms: usize,
    pub memory_controller_sequence_tls_step: u64,
    pub memory_controller_sequence_tls_lag: u64,
    pub enable_arrangement_backfill: bool,
    pub high_join_amplification_threshold: usize,
    pub enable_actor_tokio_metrics: bool,
    pub(super) exchange_connection_pool_size: Option<u16>,
    pub enable_auto_schema_change: bool,
    pub enable_shared_source: bool,
    pub switch_jdbc_pg_to_native: bool,
    pub max_barrier_batch_size: u32,
    pub hash_join_entry_state_max_rows: usize,
    pub now_progress_ratio: Option<f32>,
    pub enable_explain_analyze_stats: bool,
    pub compute_client_config: RpcClientConfig,
    pub iceberg_list_interval_sec: u64,
    pub iceberg_fetch_batch_size: u64,
    pub iceberg_sink_positional_delete_cache_size: usize,
    pub iceberg_sink_write_parquet_max_row_group_rows: usize,
    pub default_enable_mem_preload_state_table: bool,
    pub mem_preload_state_table_ids_whitelist: Vec<u32>,
    pub mem_preload_state_table_ids_blacklist: Vec<u32>,
    pub aggressive_noop_update_elimination: bool,
    pub refresh_scheduler_interval_sec: u64,
    pub join_encoding_type: JoinEncodingType,
    pub sync_log_store_pause_duration_ms: usize,
    pub sync_log_store_buffer_size: usize,
    pub over_window_cache_policy: OverWindowCachePolicy,
    pub unrecognized: Unrecognized<Self>,
}

The subsections [streaming.developer].

It is put at StreamingConfig::developer.

Fields

enable_executor_row_count: bool

Set to true to enable per-executor row count metrics. This will produce a lot of timeseries and might affect the prometheus performance. If you only need actor input and output rows data, see stream_actor_in_record_cnt and stream_actor_out_record_cnt instead.

connector_message_buffer_size: usize

The capacity of the chunks in the channel that connects between ConnectorSource and SourceExecutor.

unsafe_extreme_cache_size: usize

Limit number of the cached entries in an extreme aggregation call.

topn_cache_min_capacity: usize

Minimum cache size for TopN cache per group key.

chunk_size: usize

The maximum size of the chunk produced by executor at a time.

exchange_initial_permits: usize

The initial permits that a channel holds, i.e., the maximum row count can be buffered in the channel.

exchange_batched_permits: usize

The permits that are batched to add back, for reducing the backward AddPermits messages in remote exchange.

exchange_concurrent_barriers: usize

The maximum number of concurrent barriers in an exchange channel.

exchange_concurrent_dispatchers: usize

The concurrency for dispatching messages to different downstream jobs.

• 1 means no concurrency, i.e., dispatch messages to downstream jobs one by one.
• 0 means unlimited concurrency.

dml_channel_initial_permits: usize

The initial permits for a dml channel, i.e., the maximum row count can be buffered in the channel.

hash_agg_max_dirty_groups_heap_size: usize

The max heap size of dirty groups of HashAggExecutor.

memory_controller_threshold_aggressive: f64

memory_controller_threshold_graceful: f64

memory_controller_threshold_stable: f64

memory_controller_eviction_factor_aggressive: f64

memory_controller_eviction_factor_graceful: f64

memory_controller_eviction_factor_stable: f64

memory_controller_update_interval_ms: usize

memory_controller_sequence_tls_step: u64

memory_controller_sequence_tls_lag: u64

enable_arrangement_backfill: bool

Enable arrangement backfill If false, the arrangement backfill will be disabled, even if session variable set. If true, it’s decided by session variable streaming_use_arrangement_backfill (default true)

high_join_amplification_threshold: usize

If number of hash join matches exceeds this threshold number, it will be logged.

enable_actor_tokio_metrics: bool

Actor tokio metrics is enabled if enable_actor_tokio_metrics is set or metrics level >= Debug.

exchange_connection_pool_size: Option<u16>

The number of the connections for streaming remote exchange between two nodes. If not specified, the value of server.connection_pool_size will be used.

enable_auto_schema_change: bool

A flag to allow disabling the auto schema change handling

enable_shared_source: bool

Enable shared source If false, the shared source will be disabled, even if session variable set. If true, it’s decided by session variable streaming_use_shared_source (default true)

switch_jdbc_pg_to_native: bool

When true, all jdbc sinks with connector=‘jdbc’ and jdbc.url=“jdbc:postgresql://…” will be switched from jdbc postgresql sinks to rust native (connector=‘postgres’) sinks.

max_barrier_batch_size: u32

The maximum number of consecutive barriers allowed in a message when sent between actors.

hash_join_entry_state_max_rows: usize

Configure the system-wide cache row cardinality of hash join. For example, if this is set to 1000, it means we can have at most 1000 rows in cache.

now_progress_ratio: Option<f32>

enable_explain_analyze_stats: bool

Enable / Disable profiling stats used by EXPLAIN ANALYZE

compute_client_config: RpcClientConfig

iceberg_list_interval_sec: u64

IcebergListExecutor: The interval in seconds for Iceberg source to list new files.

iceberg_fetch_batch_size: u64

IcebergFetchExecutor: The number of files the executor will fetch concurrently in a batch.

iceberg_sink_positional_delete_cache_size: usize

IcebergSink: The size of the cache for positional delete in the sink.

iceberg_sink_write_parquet_max_row_group_rows: usize

IcebergSink: The maximum number of rows in a row group when writing Parquet files.

default_enable_mem_preload_state_table: bool

Whether by default enable preloading all rows in memory for state table. If true, all capable state tables will preload its state to memory

mem_preload_state_table_ids_whitelist: Vec<u32>

The list of state table ids to enable preloading all rows in memory for state table. Only takes effect when default_enable_mem_preload_state_table is false.

mem_preload_state_table_ids_blacklist: Vec<u32>

The list of state table ids to disable preloading all rows in memory for state table. Only takes effect when default_enable_mem_preload_state_table is true.

aggressive_noop_update_elimination: bool

Eliminate unnecessary updates aggressively, even if it impacts performance. Enable this only if it’s confirmed that no-op updates are causing significant streaming amplification.

refresh_scheduler_interval_sec: u64

The interval in seconds for the refresh scheduler to check and trigger scheduled refreshes.

join_encoding_type: JoinEncodingType

Determine which encoding will be used to encode join rows in operator cache.

sync_log_store_pause_duration_ms: usize

The timeout for reading from the buffer of the sync log store on barrier. Every epoch we will attempt to read the full buffer of the sync log store. If we hit the timeout, we will stop reading and continue.

sync_log_store_buffer_size: usize

The max buffer size for sync logstore, before we start flushing.

over_window_cache_policy: OverWindowCachePolicy

Cache policy for partition cache in streaming over window. Can be full, recent, recent_first_n or recent_last_n.

unrecognized: Unrecognized<Self>

Implementations

impl StreamingDeveloperConfig

pub fn config_docs( name: String, docs: &mut BTreeMap<String, Vec<(String, String)>>, )

Trait Implementations

impl Clone for StreamingDeveloperConfig

fn clone(&self) -> StreamingDeveloperConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for StreamingDeveloperConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for StreamingDeveloperConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for StreamingDeveloperConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for StreamingDeveloperConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.