Skip to main content

Module iceberg

Module iceberg 

Source

Modules§

commit 🔒
commit_retry
Shared iceberg commit-with-retry primitive used by both the V1/V2 sink committer and the V3 sink coordinator worker. Wraps the standard pattern of “reload the table, build an action against the freshly-loaded snapshot, commit, retry on transient errors only”.
config 🔒
create_table 🔒
prometheus 🔒
writer 🔒

Structs§

IcebergCommitResult
IcebergConfig
IcebergOrderKeyField
IcebergPositionDeleteMergerCommitResult
IcebergSink
IcebergSinkCommitter
IcebergSinkWriterArgs
IcebergSinkWriterInner
IcebergWriterMetrics

Enums§

CompactionType
Compaction type for Iceberg sink
IcebergSinkWriter
IcebergWriteMode

Constants§

COMPACTION_DELETE_FILES_COUNT_THRESHOLD
COMPACTION_INTERVAL_SEC
COMPACTION_MAX_SNAPSHOTS_NUM
COMPACTION_SMALL_FILES_THRESHOLD_MB
COMPACTION_TARGET_FILE_SIZE_MB
COMPACTION_TRIGGER_SNAPSHOT_COUNT
COMPACTION_TYPE
COMPACTION_WRITE_PARQUET_COMPRESSION
COMPACTION_WRITE_PARQUET_MAX_ROW_GROUP_BYTES
COMPACTION_WRITE_PARQUET_MAX_ROW_GROUP_ROWS
DEFAULT_COMPACTION_MAX_SNAPSHOTS_NUM
ENABLE_COMPACTION
ENABLE_PK_INDEX
ENABLE_SNAPSHOT_EXPIRATION
FORMAT_VERSION
ICEBERG_COMPACTION_TYPE_FILES_WITH_DELETE
ICEBERG_COMPACTION_TYPE_FULL
ICEBERG_COMPACTION_TYPE_SMALL_FILES
ICEBERG_COW_BRANCH
ICEBERG_DEFAULT_WRITE_PARQUET_MAX_ROW_GROUP_BYTES
ICEBERG_SINK
ICEBERG_WRITE_MODE_COPY_ON_WRITE
ICEBERG_WRITE_MODE_MERGE_ON_READ
ORDER_KEY
PARQUET_CREATED_BY
PARTITION_DATA_ID_START
SNAPSHOT_EXPIRATION_CLEAR_EXPIRED_FILES
SNAPSHOT_EXPIRATION_CLEAR_EXPIRED_META_DATA
SNAPSHOT_EXPIRATION_MAX_AGE_MILLIS
SNAPSHOT_EXPIRATION_RETAIN_LAST
WRITE_MODE

Functions§

commit_branch
create_and_validate_table_impl
parse_order_key_exprs
parse_parquet_compression
Parse compression codec string to Parquet Compression enum
parse_partition_by_exprs
resolve_partition_spec
Resolves a partition spec by id from the table metadata, with a consolidated error message. Shared spec-lookup mechanic for the pk-index merger, commit coordinator, and sink commit paths.
resolve_partition_type
Resolves the partition [StructType] for the given spec_id against schema.
serialize_data_files_default_spec
Serializes files against the table’s current default partition spec, after truncating oversized column statistics. Shared by the iceberg sink writer’s commit-metadata generation and the pk-index merger’s delete-file serialization.
should_enable_iceberg_cow
truncate_datafile
Truncate large column statistics from DataFile BEFORE serialization.
try_matches_arrow_schema
Try to match our schema with iceberg schema.
validate_order_key_columns

Type Aliases§

PositionDeleteFileWriterBuilderType
The concrete V2 position-delete Parquet writer builder type, shared by the iceberg sink writer and the pk-index position-delete merger.