Skip to main content

Iceberg

Testing

Important Capabilities​

CapabilityStatusNotes
Data Profiling✅Optionally enabled via configuration.
Descriptions✅Enabled by default.
Detect Deleted Entities✅Enabled via stateful ingestion
Domains❌Currently not supported.
Extract Ownership✅Optionally enabled via configuration by specifying which Iceberg table property holds user or group ownership.
Partition Support❌Currently not supported.
Platform Instance✅Optionally enabled via configuration, an Iceberg instance represents the datalake name where the table is stored.

Integration Details​

The DataHub Iceberg source plugin extracts metadata from Iceberg tables stored in a distributed or local file system. Typically, Iceberg tables are stored in a distributed file system like S3 or Azure Data Lake Storage (ADLS) and registered in a catalog. There are various catalog implementations like Filesystem-based, RDBMS-based or even REST-based catalogs. This Iceberg source plugin relies on the Iceberg python_legacy library and its support for catalogs is limited at the moment. A new version of the Iceberg Python library is currently in development and should fix this. Because of this limitation, this source plugin will only ingest HadoopCatalog-based tables that have a version-hint.text metadata file.

Ingestion of tables happens in 2 steps:

  1. Discover Iceberg tables stored in file system.
  2. Load discovered tables using Iceberg python_legacy library

The current implementation of the Iceberg source plugin will only discover tables stored in a local file system or in ADLS. Support for S3 could be added fairly easily.

CLI based Ingestion​

Install the Plugin​

pip install 'acryl-datahub[iceberg]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: "iceberg"
  config:
    env: PROD
    adls:
      # Will be translated to https://{account_name}.dfs.core.windows.net
      account_name: my_adls_account
      # Can use sas_token or account_key
      sas_token: "${SAS_TOKEN}"
      # account_key: "${ACCOUNT_KEY}"
      container_name: warehouse
      base_path: iceberg
    platform_instance: my_iceberg_catalog
    table_pattern:
      allow:
        - marketing.*
    profiling:
      enabled: true

sink:
  # sink configs


Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
group_ownership_property
string
Iceberg table property to look for a CorpGroup owner. Can only hold a single group value. If property has no value, no owner information will be emitted.
localfs
string
Local path to crawl for Iceberg tables. This is one filesystem type supported by this source and only one can be configured.
max_path_depth
integer
Maximum folder depth to crawl for Iceberg tables. Folders deeper than this value will be silently ignored.
Default: 2
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to
user_ownership_property
string
Iceberg table property to look for a CorpUser owner. Can only hold a single user value. If property has no value, no owner information will be emitted.
Default: owner
env
string
The environment that all assets produced by this connector belong to
Default: PROD
adls
AdlsSourceConfig
Azure Data Lake Storage to crawl for Iceberg tables. This is one filesystem type supported by this source and only one can be configured.
adls.account_name ❓
string
Name of the Azure storage account. See Microsoft official documentation on how to create a storage account.
adls.container_name ❓
string
Azure storage account container name.
adls.account_key
string
Azure storage account access key that can be used as a credential. An account key, a SAS token or a client secret is required for authentication.
adls.base_path
string
Base folder in hierarchical namespaces to start from.
Default: /
adls.client_id
string
Azure client (Application) ID required when a client_secret is used as a credential.
adls.client_secret
string
Azure client secret that can be used as a credential. An account key, a SAS token or a client secret is required for authentication.
adls.sas_token
string
Azure storage account Shared Access Signature (SAS) token that can be used as a credential. An account key, a SAS token or a client secret is required for authentication.
adls.tenant_id
string
Azure tenant (Directory) ID required when a client_secret is used as a credential.
table_pattern
AllowDenyPattern
Regex patterns for tables to filter in ingestion.
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
table_pattern.allow
array(string)
table_pattern.deny
array(string)
table_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
profiling
IcebergProfilingConfig
Default: {'enabled': False, 'include_field_null_count': Tru...
profiling.enabled
boolean
Whether profiling should be done.
Default: False
profiling.include_field_max_value
boolean
Whether to profile for the max value of numeric columns.
Default: True
profiling.include_field_min_value
boolean
Whether to profile for the min value of numeric columns.
Default: True
profiling.include_field_null_count
boolean
Whether to profile for the number of nulls for each column.
Default: True
stateful_ingestion
StatefulStaleMetadataRemovalConfig
Iceberg Stateful Ingestion Config.
stateful_ingestion.enabled
boolean
The type of the ingestion state provider registered with datahub.
Default: False
stateful_ingestion.ignore_new_state
boolean
If set to True, ignores the current checkpoint state.
Default: False
stateful_ingestion.ignore_old_state
boolean
If set to True, ignores the previous checkpoint state.
Default: False
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Concept Mapping​

This ingestion source maps the following Source System Concepts to DataHub Concepts:

Source ConceptDataHub ConceptNotes
icebergData Platform
TableDatasetEach Iceberg table maps to a Dataset named using the parent folders. If a table is stored under my/namespace/table, the dataset name will be my.namespace.table. If a Platform Instance is configured, it will be used as a prefix: <platform_instance>.my.namespace.table.
Table propertyUser (a.k.a CorpUser)The value of a table property can be used as the name of a CorpUser owner. This table property name can be configured with the source option user_ownership_property.
Table propertyCorpGroupThe value of a table property can be used as the name of a CorpGroup owner. This table property name can be configured with the source option group_ownership_property.
Table parent folders (excluding warehouse catalog location)ContainerAvailable in a future release
Table schemaSchemaFieldMaps to the fields defined within the Iceberg table schema definition.

Troubleshooting​

[Common Issue]​

[Provide description of common issues with this integration and steps to resolve]

Code Coordinates​

  • Class Name: datahub.ingestion.source.iceberg.iceberg.IcebergSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Iceberg, feel free to ping us on our Slack.