Struct IcebergTaskQueue 

pub struct IcebergTaskQueue {
    inner: IcebergTaskQueueInner,
    max_parallelism: u32,
    pending_parallelism_budget: u32,
    schedule_notify: Option<Arc<Notify>>,
}

FIFO task queue with parallelism-based scheduling for Iceberg compaction.

Tasks execute in submission order when sufficient parallelism is available. The queue tracks waiting and running tasks to prevent over-commitment of resources.

Constraints:

• Each task requires 1..=max_parallelism units
• Total waiting parallelism cannot exceed pending_parallelism_budget
• Total running parallelism cannot exceed max_parallelism
• Tasks block until enough parallelism is available

Note: The queue does NOT deduplicate or reorder tasks. Task management (deduplication, merging, cancellation) is Meta’s responsibility.

Fields

inner: IcebergTaskQueueInner

max_parallelism: u32

Maximum concurrent parallelism for running tasks

pending_parallelism_budget: u32

Maximum total parallelism for waiting tasks (backpressure limit)

schedule_notify: Option<Arc<Notify>>

Optional notification for event-driven scheduling

Implementations

impl IcebergTaskQueue

pub fn new(max_parallelism: u32, pending_parallelism_budget: u32) -> Self

pub fn new_with_notify( max_parallelism: u32, pending_parallelism_budget: u32, ) -> (Self, Arc<Notify>)

pub async fn wait_schedulable(&self) -> bool

fn has_schedulable_tasks(&self) -> bool

fn notify_schedulable(&self)

pub fn running_parallelism_sum(&self) -> u32

pub fn waiting_parallelism_sum(&self) -> u32

fn available_parallelism(&self) -> u32

pub fn push( &mut self, meta: IcebergTaskMeta, runner: Option<IcebergCompactionPlanRunner>, ) -> PushResult

Push a task into the queue.

The task is validated and added to the end of the FIFO queue if constraints are met.

pub fn pop(&mut self) -> Option<PoppedIcebergTask>

Pop the next task if sufficient parallelism is available.

Returns None if the queue is empty or the front task cannot fit within the available parallelism budget.

pub fn finish_running(&mut self, task_id: u64) -> bool

Mark a task as finished, freeing its parallelism for other tasks.

Returns true if the task was found and removed, false otherwise.