Unity Catalog / Governance & Security

Unity Catalog hierarchy, grants, service principals, external locations, lineage, row/column security and securing sensitive data on Azure.

Same format as Parts 1 and 2. This part leans on Azure specifics (Entra ID, ADLS Gen2, Azure Key Vault) since that is a common production setup. If it is yours, say so explicitly in the interview; it signals real production experience over textbook Unity Catalog knowledge.

1. Unity Catalog Hierarchy

Q: Walk me through the Unity Catalog object hierarchy.

A:

Metastore
   |
Catalog
   |
Schema
   |
Table / View / Volume

• Metastore: the top-level container, one per region typically, attached to one or more workspaces. It holds the metadata and points at a default storage location; it's created once per account, not per project.
• Catalog: the first real organizational boundary — typically maps to an environment (dev, test, prod) or a business domain (sales, finance). This is usually where I set the broadest access boundary.
• Schema (= database): groups related tables within a catalog — e.g. prod.finance.bronze, prod.finance.silver, prod.finance.gold, mirroring the medallion layers as schemas within a domain catalog.
• Table / View / Volume: the leaf objects. Volumes are Unity Catalog's abstraction for governing non-tabular files (a governed path for arbitrary files — landing zone data, ML artifacts, images) under the same permission model as tables.

Q: Why put environment (dev/test/prod) at the catalog level rather than, say, a naming prefix on tables?

A: Because permissions, ownership, and grants are all scoped at catalog/schema/table level natively — putting environment there means I can grant an entire team USE CATALOG + SELECT on dev broadly while prod stays locked down to a service principal and a small admin group, with one clean permission boundary instead of parsing naming conventions to simulate isolation. It also means catalog-level lineage, audit logs, and quotas are already environment-scoped for free.

2. Managed vs External Tables

Q: What's the difference between a managed and an external table in Unity Catalog?

A: A managed table's data lives in storage that Unity Catalog owns and controls, under the catalog/schema's managed storage location — Databricks manages the full lifecycle, so DROP TABLE deletes the underlying data files too. An external table points at a storage location you specify (LOCATION 'abfss://...') that exists independently of Unity Catalog — DROP TABLE only removes the metadata/registration, the data stays in place. External tables require an external location + storage credential to be registered first (see below).

Q: When do you choose external over managed?

A: External when: the data needs to be accessible/shared outside Unity Catalog's governance boundary (another team's tooling reading directly from ADLS), you need the data to survive a table being dropped and recreated (common for landing zones), or you're migrating an existing lake where data already lives at specific paths and re-landing everything as managed isn't practical. Managed when: you want Unity Catalog to fully own lifecycle and optimize storage layout for you (Predictive Optimization, automatic file management), and there's no reason for anything outside UC to touch the files directly — this is my default for Silver/Gold in a clean-slate build, since it removes a whole class of "orphaned files after drop" cleanup problems.

3. Catalogs

Q: How do you structure catalogs in a real project?

A: Typically one catalog per environment (dev/test/prod) crossed with domain where the org is large enough to warrant it (prod_finance, prod_sales), or domain-first with environment as a schema-level split for smaller orgs. I favor environment-at-catalog because it makes the security boundary unambiguous — a prod service principal literally cannot see a dev catalog unless explicitly granted, which is a much stronger guarantee than "the pipeline code points at different table names by config."

4. Schemas

Q: How do schemas map onto medallion architecture?

A: catalog.bronze, catalog.silver, catalog.gold (or as sub-schemas within a domain catalog) — each layer's access pattern is different (Bronze: pipeline service principals only, tightly restricted; Silver: broader internal analytics access; Gold: business-user/BI-tool facing), so schema-level grants let you express that directly instead of managing per-table grants for every table in every layer.

5. Grants

Q: How does the Unity Catalog grant model work?

A: SQL-standard GRANT <privilege> ON <securable> TO <principal> — privileges include USE CATALOG, USE SCHEMA, SELECT, MODIFY, CREATE TABLE, CREATE SCHEMA, EXECUTE (functions), ALL PRIVILEGES. Grants are hierarchical and inherited — granting SELECT at the catalog level implicitly grants it on everything below, but you can also grant narrowly at the schema or table level. Importantly, to actually read a table you need the chain — USE CATALOG + USE SCHEMA + SELECT on the table — missing any link in the chain blocks access even if the table-level grant is correct. This trips people up constantly: "I granted SELECT on the table but they still can't query it" is almost always a missing USE CATALOG/USE SCHEMA.

Q: How do you audit "who can access this table" given inheritance?

A: SHOW GRANTS ON TABLE ... shows direct grants on that object, but you have to separately check inherited grants at schema/catalog level to get the full picture — I'd script this (or use the account console's permissions view / system tables under system.access.*) rather than trying to reason about it by hand for anything sensitive, since the effective permission is a union across the hierarchy plus group membership.

6. Ownership

Q: What does "ownership" mean in Unity Catalog, distinct from grants?

A: The owner of a securable (catalog/schema/table/etc.) implicitly has full control over it (can grant/revoke, drop, alter) without needing an explicit grant — ownership is usually assigned to a group, not an individual, precisely so it survives someone leaving the team. ALTER <securable> OWNER TO <principal>. Best practice: own production data assets by a group (e.g. data-platform-admins) rather than a person, and audit ownership periodically since an individual owner is both a single point of failure and a silent privilege-escalation risk if their account is compromised.

7. Service Principals

Q: Why use a service principal instead of a personal account for pipeline execution?

A: Pipelines shouldn't run as a human identity — a person leaving the company, rotating their password, or losing MFA access shouldn't break production jobs, and you want an audit trail that clearly says "the ingestion job did this," not "Alexey's personal token did this" (which also fails the moment that person's access is revoked or their permissions change for unrelated reasons). In Azure, a Databricks service principal is backed by an Entra ID app registration; I grant it exactly the catalog/schema/table permissions the job needs (least privilege — see below) and nothing more, and jobs authenticate via OAuth (client credentials) rather than a long-lived PAT where possible.

Q: How do you scope a service principal's permissions for a specific pipeline?

A: One service principal per pipeline/domain (not one shared SP for "all of Databricks") — the Bronze ingestion SP gets MODIFY on Bronze schemas only and read access to the source external locations it needs; it should not have SELECT on Gold financial tables it has no business touching. This containment means a compromised or misconfigured job can't blast-radius into unrelated data.

8. Groups

Q: How do groups fit into the permission model?

A: Grant to groups (synced from Entra ID), not individuals — data-engineers, data-analysts, finance-readonly. This makes onboarding/offboarding a group-membership change in Entra ID rather than a Unity Catalog grant change, and it's what makes an access audit tractable ("who's in finance-readonly" is one query, vs. auditing dozens of individual grants scattered across objects).

9. External Locations

Q: What is an external location and why does Unity Catalog require one instead of just pointing a table at a path?

A: An external location is a governed registration of a storage path (abfss://container@storage_account.dfs.core.windows.net/path) bound to a storage credential, with its own grants (READ FILES, WRITE FILES, CREATE TABLE). It's the layer that lets Unity Catalog govern access to raw cloud storage paths the same way it governs tables — without it, anyone with workspace access could mount/read arbitrary storage paths outside UC's audit and permission model, which defeats the point of centralizing governance. Every external table's location must fall under a registered external location the caller has been granted access to.

10. Storage Credentials

Q: What's a storage credential, and how does it work on Azure specifically?

A: The actual authentication mechanism Unity Catalog uses to reach cloud storage on behalf of a user/job — on Azure, typically a managed identity (system- or user-assigned, backed by Entra ID) with the right RBAC role (Storage Blob Data Contributor, scoped narrowly) on the target storage account, registered in Unity Catalog as a storage credential object. Nobody gets a raw storage account key; access flows through UC's permission model, so revoking a grant in UC actually revokes access — there's no backdoor via a leaked key.

11. Lineage

Q: How does Unity Catalog lineage work and what's it useful for?

A: UC automatically captures table- and column-level lineage from queries/pipeline runs that go through it — you can see, for any table, what upstream tables/columns fed it and what downstream tables/notebooks/dashboards consume it, via the catalog UI or system.access.table_lineage / column_lineage system tables. Practically: impact analysis before a schema change ("what breaks if I drop this column"), debugging a bad number in a dashboard by tracing it back to source, and — relevant to the GDPR discussion earlier — proving/finding everywhere a PII column actually flows to, which is otherwise a very manual audit.

12. Auditing

Q: How do you audit access to sensitive data in Unity Catalog?

A: System tables under system.access (audit, table_lineage, etc.) log every query, grant change, and access event with principal, timestamp, and object — queryable with SQL like any other table, which makes it easy to build a monitoring dashboard or alert on anomalous access patterns (e.g. a service principal suddenly querying a table outside its normal pattern) rather than digging through raw audit log exports. On Azure this can also feed into Azure Monitor / Log Analytics via diagnostic settings for centralized SIEM correlation with other Azure resource audit logs.

13. Row/Column Security

Q: How do you implement row-level and column-level security in Unity Catalog?

A:

• Column-level: dynamic views with CASE WHEN is_account_group_member('finance-readonly') THEN salary ELSE NULL END, or native column masks (ALTER TABLE ... ALTER COLUMN col SET MASK mask_function) — a UDF that redacts/nulls the value based on the calling principal's group membership, applied consistently regardless of which query touches the column.
• Row-level: row filters (ALTER TABLE ... SET ROW FILTER filter_function ON (col)) — a UDF returning a boolean per row based on caller identity, e.g. a sales rep only sees rows where region = current_user_region().

Both are defined once on the table and enforced for every query against it — no per-query WHERE-clause discipline required from analysts, which is the point: security shouldn't depend on everyone remembering to filter correctly.

14. Environment Separation

Q: How do you separate dev/test/prod in Unity Catalog, end to end?

A: Separate catalogs per environment (see Catalogs above), separate service principals per environment (a dev SP can never see prod catalogs even if misconfigured — the grant simply doesn't exist), separate storage accounts/external locations per environment so there's no shared underlying storage a bug could cross-contaminate, and CI/CD (Databricks Asset Bundles, from Part 1) parameterizes the target catalog per deploy environment so the same job code is deployed against dev.finance... in dev and prod.finance... in prod without code changes — the environment binding lives in the bundle's target config, not in notebook logic.

15. "How would you secure sensitive tax data in Databricks?" — the capstone

Identity
   ↓
Azure Entra ID
   ↓
Unity Catalog
   ↓
RBAC (catalog/schema/table grants, to groups & service principals)
   ↓
Catalog/schema/table permissions (environment + domain boundaries)
   ↓
Row/column restrictions (masks, row filters — need-to-know within the table)

Walk through it top-down and then layer in the cross-cutting concerns:

Identity → Entra ID → Unity Catalog: every human and every service principal is an Entra ID identity; Unity Catalog doesn't maintain its own separate user store, it delegates to Entra ID (and Entra groups sync into UC groups), so there's one source of truth for "who is this" across all of Azure, not a Databricks-specific identity silo.

RBAC / catalog-schema-table permissions: tax data lives in its own catalog or schema (prod.finance.tax), granted only to a narrow finance-tax-readers group and the specific pipeline service principal that processes it — not the general data-analysts group that has broad Gold access. USE CATALOG/USE SCHEMA/SELECT chain explicitly granted, nothing inherited from a broader catalog-level grant that might accidentally include tax schemas.

Row/column restrictions: even within the tax schema, column masks on the most sensitive fields (SSN/tax ID equivalents, exact income figures) so that even someone with table access sees a masked value unless they're in a smaller "sees full PII" group — need-to-know within need-to-know.

Encryption: encryption at rest is handled by ADLS (Storage Service Encryption, Microsoft-managed or customer-managed keys via Azure Key Vault for an extra control layer — CMK matters for tax data specifically if there's a regulatory requirement for customer-controlled key rotation/revocation), and in transit via TLS for everything hitting the workspace and storage. For field-level extra sensitivity, application-level encryption of specific columns (tax ID) with keys in Key Vault, decrypted only for principals with explicit access via a UDF — belt-and-suspenders on top of storage-level encryption.

Secrets: no credentials in notebooks/code, ever — Databricks secret scopes backed by Azure Key Vault, referenced as dbutils.secrets.get(scope, key), and the secret itself never appears in logs/notebook output (Databricks redacts known secret values from output). Service principal client secrets, storage account keys (where still used anywhere legacy), API keys for any external tax-calculation service — all in Key Vault, not hardcoded, not in Git.

Private networking: workspace deployed with VNet injection, storage accounts with private endpoints (no public network access), and Private Link so traffic between the Databricks workspace and ADLS/Key Vault never traverses the public internet. For tax data specifically I'd also want NSG rules restricting which subnets can even reach the storage account, and IP access lists on the workspace itself if there's no full VNet injection.

Audit logs: system.access.audit for who-queried-what-when at the Databricks/UC layer, plus Azure Monitor diagnostic settings on the storage account and Key Vault so every read of the underlying files and every secret access is also logged independently — two layers of audit so a gap in one doesn't leave you blind, and both feed a central Log Analytics workspace / SIEM for alerting on anomalies (e.g. bulk export of the tax schema at 2am from an account that's never done that before).

Least privilege: every grant above is scoped to the narrowest group/principal that needs it, time-boxed where the tooling supports it, and periodically reviewed (access recertification) rather than granted once and forgotten — the tax schema access list should be small enough that you can recite who's on it from memory.

Service principals: the pipeline that processes tax data runs as its own dedicated SP, scoped only to the tax schema and its specific source/target locations, with its Entra ID app registration's client secret in Key Vault and rotated on a schedule — never a shared "does everything" SP, precisely so a compromise or misconfiguration of one pipeline can't reach unrelated data.

Closing line: "None of these layers is sufficient alone — network isolation doesn't help if grants are too broad, and tight grants don't help if a secret is sitting in plaintext in a notebook. The tax data is only actually secure where all of these overlap: an authenticated identity, scoped to exactly the schema it needs, over a private network, with sensitive columns masked even from people with table access, every access logged in two independent places."

Quick-fire recap