Struct FetchState 

struct FetchState {
    is_refreshing: bool,
    is_list_finished: bool,
    splits_on_fetch: usize,
    is_batch_finished: Arc<RwLock<bool>>,
    file_queue: VecDeque<(String, JsonbVal)>,
    in_flight_files: Vec<(String, JsonbVal)>,
}

Fields

is_refreshing: bool

Whether we are in a refresh cycle (started by RefreshStart, ended by load finished report)

is_list_finished: bool

Whether the upstream list executor has finished listing all files

splits_on_fetch: usize

Number of files currently being fetched in the active batch reader

is_batch_finished: Arc<RwLock<bool>>

Shared flag indicating whether the current batch reader has finished reading all files

file_queue: VecDeque<(String, JsonbVal)>

Queue of files waiting to be processed

in_flight_files: Vec<(String, JsonbVal)>

Files currently being fetched by the batch reader. Used for at-least-once recovery: on error, these files are re-queued.

Implementations

impl FetchState

fn new() -> Self

fn reset_for_refresh(&mut self)

Reset all state for a new refresh cycle.

fn should_report_load_finished(&self) -> bool

Check if we should report load finished to the barrier manager.

fn mark_refresh_complete(&mut self)

Mark the refresh cycle as complete after reporting load finished.

fn should_start_batch_reader(&self, need_rebuild: bool) -> bool

Check if we should start a new batch reader.

fn mark_file_fetched(&mut self)

Mark one file as successfully fetched.

fn handle_error_recovery(&mut self)

Handle fetch error with at-least-once recovery. Re-queues in-flight files to ensure no file is skipped.

fn enqueue_files(&mut self, files: impl IntoIterator<Item = (String, JsonbVal)>)

Enqueue new file assignments from upstream.