Bonus: How DataHub Structures Metadata Internally

The Building Blocks of DataHub

The core modules covered how to use DataHub day to day; this bonus module looks under the hood at how it represents the world.

Four concepts you will use in every interaction with DataHub:

  • Entities
  • Aspects
  • URNs (Uniform Resource Names) — each entity's permanent ID
  • Domains
Entity URN Aspects Schema Fields Structured Properties Domain (itself an entity)

What Is an Entity?

  • An entity is a specific "thing" that DataHub tracks -- think of it as one row in a directory
  • DataHub tracks many kinds of entities; the major ones:
    • Dataset: table, view, topic, file, or any structured data asset
    • Dashboard: a Looker dashboard, Tableau workbook, etc.
    • Chart: an individual chart or visualization
Dataset e.g. one specific table Dashboard e.g. a Looker dashboard Chart e.g. a revenue chart

Entities Have...

  • Every entity has a unique resource name (URN) -- its permanent ID, assigned when the entity is created and never changed
  • Almost every entity also has a human-readable name -- the friendly label you see in the UI
  • Example: the table you see as fact_revenue_daily is, under the hood, urn:li:dataset:(urn:li:dataPlatform:snowflake,PROD.FINANCE.FACT_REVENUE_DAILY,PROD)
  • The URN looks scary, but you rarely type one -- the UI handles it; you just need to recognize one when you see it
Name (what you see) fact_revenue_daily URN (what DataHub uses) urn:li:dataset:(urn:li:dataPlatform:snowflake, PROD.FINANCE.FACT_REVENUE_DAILY,PROD) same entity

Entity vs. Entity Type: Don't Mix These Up

  • Entity type: the category (Dataset, Dashboard, User...)
  • Entity: a specific instance of that type
  • Dataset: urn:li:dataset:(urn:li:dataPlatform:snowflake,PROD.FINANCE.FACT_REVENUE_DAILY,PROD)
  • Dashboard: urn:li:dashboard:(looker,dashboards.42)
  • User: urn:li:corpuser:jsmith
  • The confusion shows up in bug reports: "The entity is wrong" -- which entity, specifically?
Entity Type Dataset Entity FACT_REVENUE_DAILY instance of

What Is an Aspect?

  • An aspect is a named bundle of properties attached to an entity
  • One entity has many aspects -- each aspect holds one category of metadata
  • Example: the FACT_REVENUE_DAILY Dataset entity has these aspects, among others:
  • SchemaMetadata: column names, data types, field descriptions
  • DatasetProfile: row counts, null counts, sample values
  • Ownership: who owns this dataset (and in what role)
  • GlobalTags: tags applied to this dataset
FACT_REVENUE_DAILY SchemaMetadata DatasetProfile Ownership GlobalTags

Aspects Are Modular by Design

  • Multiple systems can write different aspects to the same entity simultaneously
  • Example flow for FACT_REVENUE_DAILY:
  • Snowflake ingestion writes the SchemaMetadata aspect (columns, types)
  • dbt ingestion writes the DatasetProperties aspect (description)
  • Airflow integration writes the DataJobInputOutput aspect (pipeline linkage)
  • Data steward (UI) writes the Ownership aspect (assigns an owner)
  • Governance team writes the GlobalTags aspect (adds a PII tag)
FACT_REVENUE_DAILY Snowflake dbt Airflow Data steward Governance team SchemaMetadata DatasetProperties DataJobInputOutput Ownership GlobalTags

What Is a URN? The Permanent Identity of Every Entity

  • URN stands for Uniform Resource Name
  • Every entity in DataHub has exactly one URN -- it never changes
  • Format: urn:li:<entity-type>:(<identifier>)
  • Snowflake table: urn:li:dataset:(urn:li:dataPlatform:snowflake,PROD.FINANCE.FACT_REVENUE_DAILY,PROD)
  • Looker dashboard: urn:li:dashboard:(looker,dashboards.42)
  • Individual user: urn:li:corpuser:jsmith
urn:li namespace dataset entity type snowflake platform FACT_REVENUE_DAILY PROD name + environment urn:li:dataset:(urn:li:dataPlatform:snowflake,PROD.FINANCE.FACT_REVENUE_DAILY,PROD)

Always Reference Assets by URN in Scripts and API Calls

  • Use URNs, not display names, whenever referencing an asset programmatically
  • Display names can change. URNs cannot.
  • In API calls: pass the full URN as the entity identifier
  • In bug reports or support tickets: include the URN so the exact asset is unambiguous
  • In scripts: resolve the display name to a URN once at the start, then use the URN throughout
  • Where to find a URN: the entity's URL in the DataHub UI contains it (URL-encoded)
Display name FACT_REVENUE_DAILY can change URN urn:li:dataset:(...) permanent

Reading URNs Directly from the Browser URL

  • When you navigate to any entity in the DataHub UI, the URN is in the URL
  • Example URL: https://your-datahub.example.com/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Asnowflake%2CPROD.FINANCE.FACT_REVENUE_DAILY%2CPROD%29
  • URL-decoded, the path after /dataset/ is: urn:li:dataset:(urn:li:dataPlatform:snowflake,PROD.FINANCE.FACT_REVENUE_DAILY,PROD)
Brief: Screenshot of a DataHub dataset entity page with the full browser address bar visible, showing the URL-encoded URN in the path after /dataset/.

What Is a Domain? The Organizational Layer

  • A domain is a named business grouping that organizes assets in DataHub
  • Example domains: Finance, Marketing, Engineering, Risk, Compliance, Data Science
  • Every asset can be assigned to one domain -- set via the sidebar, replaces the existing assignment
  • Domains are set up by admins, but visible and useful for everyone
  • What domains enable:
  • Navigation: browse the "Finance domain" to see all Finance assets in one place
  • Governance: policies and Data Products can be scoped per domain
  • Discovery: filter any search by domain to narrow results instantly
Finance Marketing Engineering Risk Compliance Data Science

How Entities, Aspects, URNs, and Domains Fit Together

  • Every asset in DataHub can be described with these four concepts working together
  • Example: the FACT_REVENUE_DAILY Snowflake table
  • Entity: PROD.FINANCE.FACT_REVENUE_DAILY (a Dataset entity)
  • URN: urn:li:dataset:(urn:li:dataPlatform:snowflake,PROD.FINANCE.FACT_REVENUE_DAILY,PROD)
  • Aspects: SchemaMetadata -- 47 columns, types; Ownership -- owned by finance-data-team group
  • Domain: Finance (the business domain this dataset belongs to)
FACT_REVENUE_DAILY Entity: Dataset the specific table URN urn:li:dataset:(...) Aspects Schema + Ownership Domain Finance

Explore a Live DataHub Instance

  • Your task (5 minutes):
  • Navigate to the live DataHub instance shared in the session setup
  • Search for any dataset in the catalog
  • Open the entity page and locate:
  • The entity's URN (click "Copy URN" or read it from the URL)
  • The domain this entity belongs to
  • At least two aspects visible on the page (e.g., the Columns tab = SchemaMetadata, the Owners section in the sidebar = Ownership aspect)
DataHub dataset entity page with the overflow menu open and the Share submenu expanded to show Copy URN; the Domain label is visible in the right sidebar and the entity tabs run across the top

Assign Domains Based on Who Governs the Asset, Not Who Uses It

  • Rule: an asset belongs to the domain of the team responsible for its quality, governance, and lifecycle
  • Not: the domain of every team that queries it
  • Example: FACT_REVENUE_DAILY is owned and governed by the Finance data team
  • It is also consumed by Marketing and Executive Reporting
  • Correct domain: Finance (governance determines domain, not consumption)
  • Consumption relationships are tracked separately via lineage and subscriptions
FACT_REVENUE_DAILY Finance data team Marketing Executive Reporting governs (domain: Finance) consumes consumes

Check for Understanding

  • I have an Airflow DAG called daily_revenue_pipeline. What entity type is it in DataHub, and what would its URN look like?
  • A data engineer says "the entity is missing its owner." What is wrong with this statement, and what information is missing?
  • Your company has a dataset that is used by the Finance team but owned and governed by the Data Engineering team. Which domain should it be assigned to?
1 Entity type & URN 2 Missing owner statement 3 Domain assignment

Summary: The Four Building Blocks

  • Entity: a specific trackable thing in DataHub (Dataset, Dashboard, Pipeline, User, etc.)
  • Aspect: a named bundle of properties on an entity (Schema, Ownership, Tags, Profile...)
  • URN: the permanent, unique identifier for every entity -- use it in scripts and API calls
  • Domain: the business grouping that places an asset in the org chart and scopes governance
  • Key rule to remember: entity type vs. entity (specific instance) -- always be specific
Entity a specific trackable thing Aspect a bundle of properties URN the permanent identifier Domain the business grouping