pub fn is_parquet_schema_match_source_schema(
arrow_data_type: &DataType,
rw_data_type: &DataType,
) -> boolExpand description
This function checks whether the schema of a Parquet file matches the user-defined schema in RisingWave. It handles the following special cases:
- Arrow’s
timestamp(_, None)types (all four time units) match with RisingWave’sTimestamptype. - Arrow’s
timestamp(_, Some)matches with RisingWave’sTimestamptztype. - Since RisingWave does not have an
UInttype:- Arrow’s
UInt8matches with RisingWave’sInt16. - Arrow’s
UInt16matches with RisingWave’sInt32. - Arrow’s
UInt32matches with RisingWave’sInt64. - Arrow’s
UInt64matches with RisingWave’sDecimal.
- Arrow’s
- Arrow’s
Float16matches with RisingWave’sFloat32.
Nested data type matching:
- Struct: Arrow’s
Structtype matches with RisingWave’sStructtype recursively, requiring that all expected fields exist and match by name and type. Extra Arrow fields are allowed, but a declared name matching multiple Arrow siblings is ambiguous and rejected. - List: Arrow’s
Listtype matches with RisingWave’sListtype recursively, requiring the same element type. - Map: Arrow’s
Maptype matches with RisingWave’sMaptype recursively, requiring the key and value types to match, and the inner struct must have exactly two fields named “key” and “value”.
Nested positions recurse through is_parquet_field_match_source_schema so a variant
extension is recognized at any depth.