Skip to main content

Module update_planner

Module update_planner 

Source
Expand description

Metadata-only planning of Insert and Delete updates. Each plan covers a range of pending snapshots as one net change from a base snapshot to a target snapshot. Plans do not advance ingestion progress: the caller must order Delete before Insert, then acknowledge the target. This is deliberately separate from the legacy append-only planner and List/Fetch protocol.

Structsยง

DeleteEntry ๐Ÿ”’
IcebergUpdateBinding
Persist this binding before bootstrap. Never rediscover the schema or bootstrap snapshot on replay.
IcebergUpdateCursor
Last dispatched file within a deterministic range/phase stream, not a sorted path bound. Recovery replays metadata up to this marker; normal pages retain the live manifest scan. Advancing enumeration does not acknowledge application: List orders or fences phases.
IcebergUpdatePage
IcebergUpdatePlanner
IcebergUpdateTask
IcebergUpdateTaskId
Stable within a source job. snapshot_id is the target of the planned range.
ManifestDiff ๐Ÿ”’
Manifests are immutable, so a manifest path present in both manifest lists contributes the same entries to both snapshots and never needs to be read for the diff itself. The list-level first_row_id of an inherited data manifest may still be assigned after a V2 to V3 upgrade.
UpdateFile ๐Ÿ”’
Unlike the legacy split, preserves partition constants, spec, sequence numbers and name mapping. No predicates, virtual columns, or encryption keys are allowed in this read contract.

Enumsยง

IcebergUpdatePhase

Functionsยง

attached_deletes ๐Ÿ”’
delete_change ๐Ÿ”’
delete_tasks ๐Ÿ”’
Retained files are live in both snapshots, either moved between data manifests or kept in an unchanged one. The latter are looked up only when one of their delete artifacts changed. Removed files retract the rows visible in the parent.
entry_sequence ๐Ÿ”’
max_sequence is the sequence number of a snapshot containing the manifest.
index_deletes ๐Ÿ”’
Rejects multiple live delete artifacts for one data file across manifests, which must all belong to one snapshot.
insert_tasks ๐Ÿ”’
New files are live in added data manifests but not in removed ones. Files moved between data manifests are retained. Same-commit delete artifacts only filter Inserts.
is_compaction ๐Ÿ”’
load_data_files ๐Ÿ”’
Live, validated data files of one data manifest, without delete descriptors.
load_deletes ๐Ÿ”’
Live, validated delete descriptors of one delete manifest.
load_manifest ๐Ÿ”’
may_contain_partition ๐Ÿ”’
Conservative partition-summary pruning: false only when the summaries exclude partition. Iceberg applies a position delete only to data files with its spec and partition, so such a manifest cannot hold a data file that the delete applies to.
of_content ๐Ÿ”’
range_len ๐Ÿ”’
The number of pending snapshots planned as one range. A range ends before the next compaction, and a compaction forms a range of its own: it rewrites rows without changing them, while a net diff across it would retract and re-insert every rewritten row.
read_name_mapping ๐Ÿ”’
stream_files ๐Ÿ”’
Every live data file of a snapshot, for bootstrap. Delete descriptors are indexed before data enumeration, since manifests are not ordered by referenced data path. Only one data manifest and one task page are materialized at a time, in addition to the delete index.
stream_tasks ๐Ÿ”’
Plans one range phase as a deterministic stream; recovery replays it up to after_path. Bootstrap enumerates the whole snapshot. Incremental phases diff the manifest lists of the range base and target instead of indexing every live file of either snapshot.
task ๐Ÿ”’

Type Aliasesยง

DeleteIndex ๐Ÿ”’
Keyed by referenced data-file path.