Struct RefreshManager

pub struct RefreshManager {
    metadata_manager: MetadataManager,
    barrier_scheduler: BarrierScheduler,
}

High level design for refresh table

• Three tables:

• Main table: serves queries.

• Staging table: receives refreshed content during Refreshing.

• Progress table: per-VNode progress state for resumable refresh.

• Phased execution:

• Normal → Refreshing → Merging → Cleanup → Normal.

• Refreshing: load and write to staging.

• Merging: chunked sort-merge integrates staging into main; per-VNode progress persists checkpoints.

• Cleanup: purge staging and reset progress.

• Barrier-first responsiveness:

• Executor uses left-priority select_with_strategy, always handling upstream messages/barriers before background merge.

• On barriers, the executor persists progress so restarts resume exactly.

• Meta-managed state:

• refresh_state on each table enforces no concurrent refresh and enables recovery after failures.

• Startup recovery resets lingering Refreshing tables to Idle and lets executors resume Finishing safely.

Progress Table (Conceptual)

Tracks, per VNode:

• last processed position (e.g., last PK),
• completion flag,
• processed row count,
• last checkpoint epoch.

The executor initializes entries on RefreshStart, updates them during merge, and loads them at startup to resume from the last checkpoint.

Barrier Coordination and Completion

• Compute reports:

• refresh_finished_table_ids: indicates a materialized view finished refreshing.

• truncate_tables: staging tables to be cleaned up.

• update refresh_state to Idle,

• schedule/handle LoadFinish,

• drive cleanup work reliably after the storage version commit.

Manager responsible for handling refresh operations on refreshable tables

Fields

metadata_manager: MetadataManager

barrier_scheduler: BarrierScheduler

Implementations

impl RefreshManager

pub fn new( metadata_manager: MetadataManager, barrier_scheduler: BarrierScheduler, ) -> Self

Create a new RefreshManager instance

pub async fn refresh_table( &self, request: RefreshRequest, ) -> MetaResult<RefreshResponse>

Execute a refresh operation for the specified table

This method:

1. Validates that the table exists and is refreshable
2. Checks current refresh state and ensures no concurrent refresh
3. Atomically sets the table state to REFRESHING
4. Sends a refresh command through the barrier system
5. Returns the result of the refresh operation

async fn validate_refreshable_table( &self, table_id: TableId, associated_source_id: TableId, ) -> MetaResult<()>

Validate that the specified table exists and supports refresh operations