Module iceberg 

Modules

commit 🔒

commit_retry

Shared iceberg commit-with-retry primitive used by both the V1/V2 sink committer and the V3 sink coordinator worker. Wraps the standard pattern of “reload the table, build an action against the freshly-loaded snapshot, commit, retry on transient errors only”.

config 🔒

create_table 🔒

engine_options 🔒

metadata 🔒

pk_index_file 🔒

Data file naming for pk-index sinks that do not use the legacy file path layout.

position_delete 🔒

Shared Iceberg position-delete (Parquet and Puffin deletion vector) helpers.

prometheus 🔒

writer 🔒

Structs

IcebergCommitResult

IcebergConfig

IcebergOrderKeyField

IcebergPositionDeleteCommitResult

IcebergSink

IcebergSinkCommitter

IcebergSinkWriterArgs

IcebergSinkWriterInner

IcebergWriterMetrics

PkIndexDataFileLocator

Maps between data file UUIDs and data file paths of a pk-index sink table.

PkIndexDataFileNameGenerator

Names every data file {uuid}.parquet with a fresh UUID v7.

PositionDeleteFileNameGenerators

File-name generators shared by all Iceberg position-delete writers.

PositionDeleteReader

Shared decoder with at most one cached Puffin reader/footer. Callers may reuse it across sequential tasks for the same table/FileIO. Decoded bitmaps are never cached by path. Puffin paths must identify immutable files; drop the reader to release its cache.

Enums

CompactionType

Compaction type for Iceberg sink

IcebergMetadataTableType

The supported per-table Iceberg metadata relations.

IcebergSinkWriter

IcebergWriteMode

Constants

COMPACTION_DELETE_FILES_COUNT_THRESHOLD

COMPACTION_INTERVAL_SEC

COMPACTION_MAX_SNAPSHOTS_NUM

COMPACTION_SMALL_FILES_THRESHOLD_MB

COMPACTION_TARGET_FILE_SIZE_MB

COMPACTION_TRIGGER_SNAPSHOT_COUNT

COMPACTION_TYPE

COMPACTION_WRITE_PARQUET_COMPRESSION

COMPACTION_WRITE_PARQUET_MAX_ROW_GROUP_BYTES

COMPACTION_WRITE_PARQUET_MAX_ROW_GROUP_ROWS

DEFAULT_COMPACTION_MAX_SNAPSHOTS_NUM

ENABLE_COMPACTION

ENABLE_MANIFEST_REWRITE

ENABLE_ORPHAN_FILE_CLEANUP

ENABLE_PK_INDEX

ENABLE_SNAPSHOT_EXPIRATION

FORMAT_VERSION

ICEBERG_COMPACTION_TYPE_AUTO

ICEBERG_COMPACTION_TYPE_FILES_WITH_DELETE

ICEBERG_COMPACTION_TYPE_FULL

ICEBERG_COMPACTION_TYPE_SMALL_FILES

ICEBERG_COW_BRANCH

ICEBERG_DEFAULT_WRITE_PARQUET_MAX_ROW_GROUP_BYTES

ICEBERG_ENGINE_OPTION_KEYS

ICEBERG_SINK

ICEBERG_WRITE_MODE_COPY_ON_WRITE

ICEBERG_WRITE_MODE_MERGE_ON_READ

LEGACY_PK_INDEX_FILE_PATH

MANIFEST_REWRITE_MIN_COUNT_TO_MERGE

MANIFEST_REWRITE_TARGET_SIZE_BYTES

ORDER_KEY

ORPHAN_FILE_CLEANUP_MIN_AGE_MILLIS

ORPHAN_FILE_CLEANUP_MIN_AGE_MILLIS_DEFAULT

ORPHAN_FILE_CLEANUP_MIN_AGE_MILLIS_MINIMUM

PARQUET_CREATED_BY

PARTITION_DATA_ID_START

SNAPSHOT_EXPIRATION_CLEAR_EXPIRED_FILES

SNAPSHOT_EXPIRATION_CLEAR_EXPIRED_META_DATA

SNAPSHOT_EXPIRATION_MAX_AGE_MILLIS

SNAPSHOT_EXPIRATION_RETAIN_LAST

WRITE_MODE

Functions

commit_branch

create_and_load_table_impl

Creates the table if configured, loads it, and checks table properties that do not depend on its current schema.

create_and_validate_table_impl

is_iceberg_engine_option

parse_order_key_exprs

parse_parquet_compression

Parse compression codec string to Parquet Compression enum

parse_partition_by_exprs

read_dv_positions_from_data_file

Reads the deletion-vector positions of a single Puffin DV DataFile.

read_parquet_position_deletes_from_file

Reads the positions stored in a V2 Parquet position-delete file into a [DeleteVector].

read_position_deletes_from_file

Reads the deleted positions of a single position-delete DataFile regardless of on-disk format,

resolve_partition_spec

Resolves a partition spec by id from the table metadata, with a consolidated error message. Shared spec-lookup mechanic for the pk-index merger, commit coordinator, and sink commit paths.

resolve_partition_type

Resolves the partition [StructType] for the given spec_id against schema.

scan_iceberg_metadata

Read one Iceberg metadata relation and emit bounded DataChunks.

serialize_data_files_default_spec

Serializes files against the table’s current default partition spec, after truncating oversized column statistics. Shared by the iceberg sink writer’s commit-metadata generation and the pk-index merger’s delete-file serialization.

should_enable_iceberg_cow

truncate_datafile

Truncate large column statistics from DataFile BEFORE serialization.

try_matches_arrow_schema

Try to match our schema with iceberg schema.

validate_compaction_option_compatibility 🔒

validate_explicit_compaction_type 🔒

validate_order_key_columns

validate_row_lineage_column_names

Iceberg V3 stores row lineage in data files as the reserved _row_id and _last_updated_sequence_number columns. A table column with either name is ambiguous with the lineage metadata column, so reject it when creating a V3 table.

write_dv_puffin_file

Writes delete_vector as a single Puffin deletion-vector blob referencing data_file_path, and returns its [DataFile] metadata (content PositionDeletes, format Puffin) with referenced_data_file set.

write_parquet_position_delete_file

Writes delete_vector as a single file-scoped Parquet position-delete file referencing data_file_path, and returns its [DataFile] metadata (content PositionDeletes, format Parquet) with referenced_data_file set.

write_position_delete_file

Write one file-scoped position-delete artifact using the table’s configured on-disk format.

Type Aliases

PositionDeleteFileWriterBuilderType

The concrete V2 position-delete Parquet writer builder type, shared by the iceberg sink writer and the pk-index position-delete merger.