Databricks Jobs / Workflows

Jobs vs tasks, dependencies, retries, idempotency, backfills and CI/CD with Asset Bundles: model answers for Databricks Workflows interviews.

The examples assume a common Azure stack: ADF orchestration on top of Databricks Workflows, Medallion architecture, Databricks Asset Bundles (DABs) for CI/CD, and metadata-driven GitOps deployment. Use a real project of your own as the concrete example whenever you can — interviewers trust specifics over textbook answers.

1. Jobs vs Tasks

Q: What's the difference between a Databricks Job and a Task?

A: A Job is the top-level orchestration unit — it owns the schedule/trigger, the run history, permissions, and job-level settings (email/webhook notifications, max concurrent runs, job clusters). A Task is a unit of work inside a Job — a notebook, a Python script, a JAR, a dbt task, a SQL task, or even another Job (via "Run Job" task). A single Job is usually a DAG of multiple Tasks, each of which can have its own cluster, its own retry policy, and its own parameters.

Q: Why would you split logic into multiple tasks instead of one big notebook?

A: Isolation of failure (you can retry just the failed task, not re-run the whole pipeline), independent compute sizing (bronze ingestion doesn't need the same cluster as a heavy gold aggregation), parallelism (independent branches run concurrently), and observability (task-level metrics/logs in the run graph instead of one monolithic log).

2. Task Dependencies

Q: How do you define dependencies between tasks?

A: Each task has a depends_on list referencing upstream task keys. Databricks builds a DAG from this and only starts a task once all its upstream dependencies have completed. You can also set run_if conditions (ALL_SUCCESS, AT_LEAST_ONE_SUCCESS, ALL_DONE, AT_LEAST_ONE_FAILED, NONE_FAILED) to control branching — e.g. a cleanup/notification task that should run even if upstream failed.

Q: How would you model Bronze → DQ → Silver → Gold → Validation → Publish as a task DAG?

A: Linear dependency chain by default, but with fan-out where sources are independent: multiple bronze ingestion tasks (per source system) can run in parallel with no dependency between them, all feeding into a DQ gate task that depends on all of them (ALL_SUCCESS). Silver transforms can also fan out per domain/entity, converging into a Gold aggregation task, then a validation task, then a publish task with run_if: ALL_SUCCESS, plus a parallel "on failure" branch (run_if: AT_LEAST_ONE_FAILED) that sends alerts.

3. Parameters

Q: How do you parameterize a Databricks Job?

A: Job-level parameters (job.parameters) are defined once and referenced across tasks with {{job.parameters.param_name}}. Task-level parameters can override or extend these. Typical parameters: run_date/business_date (for idempotent/backfillable logic), environment (dev/test/prod), source_system, full_refresh flag. In a metadata-driven design (like your ADF+Databricks setup), you'd also pass a config_path or pipeline_id pointing at the JSON config in ADLS that drives which tables/entities the task processes — no hardcoded table lists in the notebook.

Q: How do parameters flow from ADF into a Databricks Workflow?

A: ADF's "Databricks Job" / "Databricks Notebook" activity passes baseParameters, which can reference ADF pipeline parameters or trigger metadata (e.g. window start/end from a tumbling window trigger). Those become job/task parameters on the Databricks side, so ADF is the scheduler/orchestrator of when and with what window, and Databricks Workflows own how the DAG executes.

4. Retries

Q: How do you configure retries, and what's your retry strategy?

A: max_retries and min_retry_interval_millis are set per task (or job-wide default). Retries should target transient failures — throttling (429s from ADLS/storage), network blips, cluster start timeouts — not logic errors, which will just fail again identically. I set 2–3 retries with a backoff interval, and I make sure retries are safe by making tasks idempotent (see below) so a retry after partial failure doesn't duplicate data.

Q: What happens if a task fails after all retries are exhausted?

A: The task is marked Failed, downstream tasks with ALL_SUCCESS dependency don't run, the job run is marked Failed, and any configured on-failure notification/webhook fires. I typically wire a dedicated "on-failure" task (run_if: AT_LEAST_ONE_FAILED) that posts to Slack/Teams with the run URL and failed task name, in addition to the built-in email/webhook notification.

5. Timeout

Q: Where do you set timeouts, and why do they matter?

A: timeout_seconds at both job and task level. Without a timeout, a stuck task (e.g. waiting on a lock, a hung Spark job, a cluster that failed to scale) can block the whole schedule and burn compute cost indefinitely. I set task-level timeouts based on historical p95 runtime plus a margin, and a job-level timeout as a hard ceiling to protect the next scheduled run from overlapping.

6. Concurrency

Q: How do you control concurrent runs of the same job?

A: max_concurrent_runs on the job. For most pipelines I set this to 1 — you don't want two runs writing to the same Delta table windows concurrently (write conflicts, double-processing). Where concurrency across different partitions/date-windows is safe and desired (e.g. parallel backfill runs for different historical dates), I raise it, but only once I've confirmed the write pattern is partition/window-scoped and idempotent so overlapping runs can't collide.

7. Scheduling

Q: Cron vs trigger-based scheduling — how do you decide?

A: Databricks Jobs support native cron scheduling (quartz_cron_expression + timezone) and file-arrival triggers, or you let an external orchestrator (ADF tumbling window trigger, Azure DevOps pipeline, or an event) call the job via the Jobs API / "Run Job" REST call. In my setup, ADF owns scheduling because it also owns upstream dependencies (source system extracts, SLA windows) and gives me a single pane of glass across non-Databricks activities too. Databricks-native cron is fine for self-contained pipelines with no external dependency.

Q: How do you avoid schedule overlap with long-running upstream dependencies?

A: Combination of max_concurrent_runs: 1, a job-level timeout shorter than the schedule interval, and — on the ADF side — tumbling window triggers with dependency chains so a window doesn't fire until the prior one (and any upstream trigger) has completed.

8. Job Clusters vs Serverless

Q: Job clusters vs all-purpose clusters vs serverless — trade-offs?

A:

• All-purpose clusters: shared, long-running, good for interactive dev — never use for scheduled production jobs (cost, noisy-neighbor risk, no clean isolation).
• Job clusters: spun up per job run, terminated on completion. Reproducible (versioned in the job/task config), cost-efficient (no idle time), isolated failure domain. This is the default for production.
• Serverless (jobs compute): Databricks manages provisioning entirely — no cluster config, near-instant start, pay-per-use. Good for variable/spiky workloads and when you want to remove cluster-sizing as an operational burden. Trade-off is less control over instance types/spot config and (historically) fewer customization options — I evaluate it per-workload, especially for bursty bronze ingestion where cold-start latency on classic job clusters is a bigger cost than serverless's premium.

Q: How do you size a job cluster for a production task?

A: Start from data volume and shuffle characteristics, benchmark on a representative run, watch Spark UI for spill/skew, then right-size node type and autoscaling min/max. I define these in the DAB resources config so they're versioned and reviewed like code, not clicked together in the UI.

9. Notifications

Q: What notification mechanisms does Databricks Jobs support, and what do you actually wire up?

A: Email notifications on start/success/failure/duration-warning, and webhook notifications (system-defined destinations — Slack, Teams, PagerDuty, generic webhook) at job and task level. In production I wire: failure → Slack channel + on-call webhook with run URL; duration threshold exceeded → warning to the team channel (catches slow degradation before it becomes a hard failure); success → suppressed for routine runs, but tracked via metrics/dashboard instead of notification noise.

10. Failure Handling

Q: Walk me through your failure-handling design for a multi-stage pipeline.

A:

1. Fail fast at the DQ gate — bad data should never reach Silver/Gold. A DQ task runs right after Bronze, checking schema, null-rate, row-count deltas, referential/business rules; on failure it stops the DAG (ALL_SUCCESS dependency downstream) rather than silently propagating bad data forward.
2. Task-level retries for transient errors only.
3. Dead-letter / quarantine pattern — malformed records get written to a quarantine Delta table/path instead of failing the whole batch, so one bad record doesn't block the other 999,999 good ones.
4. Explicit on-failure branch in the job DAG for alerting and, where safe, automated remediation (e.g. clearing a partial write before a retry).
5. Structured logging at each task (run_id, task, row counts in/out, duration) so failures are diagnosable from logs/metrics without re-running.

11. Idempotency

Q: How do you make a Databricks pipeline idempotent?

A: The core rule: re-running the same task with the same parameters (same business date/window) must produce the same end state, not duplicate or corrupt data. Techniques:

• MERGE INTO (upsert) instead of blind INSERT for Silver/Gold writes, keyed on business/natural keys.
• Partition-overwrite pattern: INSERT OVERWRITE / replaceWhere scoped to the exact date partition being processed, so a retry replaces rather than appends.
• Delta Lake's transactional guarantees mean a failed write doesn't leave partial data — either the commit lands or it doesn't.
• Avoid auto-incrementing surrogate keys generated at write time without a stable mapping; use deterministic keys/hashes where possible so re-processing doesn't create duplicate dimension rows.
• For streaming/Auto Loader ingestion, checkpointing + foreachBatch with idempotent writes (or Delta's exactly-once semantics) instead of manual offset tracking.

Q: Give a concrete example.

A: In a medallion pipeline, each Silver task is parameterized by business_date. Instead of appending, it does MERGE INTO silver.table USING bronze_batch ON natural_key WHEN MATCHED THEN UPDATE ... WHEN NOT MATCHED THEN INSERT, and Gold aggregations use replaceWhere("business_date = '{{business_date}}'") on the target partition. Re-running the same date after a failure — or intentionally, for a backfill — produces an identical end state.

12. Backfills

Q: How do you design a pipeline to support backfills cleanly?

A: Because everything is parameterized by business_date/window and writes are idempotent (MERGE / partition-overwrite), a backfill is just "run the job with a different parameter" — either a single historical date, or a loop/Run Job task fan-out across a date range. I keep a full_refresh boolean parameter for the rare cases that need to reprocess from scratch (bypassing incremental watermark logic) vs normal incremental runs. For large backfills I use max_concurrent_runs > 1 with each run scoped to its own non-overlapping partition, so they can run in parallel safely.

Q: How do you track what's already been backfilled / avoid reprocessing?

A: A watermark/control table (or, in your metadata-driven GitOps design, a JSON state file in ADLS rather than a SQL control table) recording last-processed business_date per pipeline/entity. Incremental runs advance the watermark only after Gold + validation succeed — not after Bronze — so a mid-pipeline failure doesn't falsely advance the watermark and skip data.

13. CI/CD

Q: How do you deploy Databricks Jobs/Workflows as part of CI/CD?

A: Databricks Asset Bundles (DABs) — the job DAG, cluster specs, and task definitions live in YAML in the repo, versioned alongside the code (notebooks/Python packages) that runs in them. databricks bundle validate / deploy / run drive it through Azure DevOps pipelines, with separate target environments (dev/test/prod) parameterized in the bundle config — same pattern conceptually as ADF's adf_publish ARM-template branch, but DAB is source-of-truth-in-git for the whole job definition rather than a generated artifact.

Q: What does your actual CI/CD flow look like end-to-end?

A: Feature branch → PR triggers validation (bundle validate, lint, unit tests on transformation logic) → merge to main deploys to dev target automatically → promotion to test/prod is a gated Azure DevOps release with manual approval → DAB deploy updates the job definition (DAG, cluster config, parameters) in that workspace. On the ADF side, ARM templates via the adf_publish branch follow the same promote-through-environments pattern, so both orchestration layers are deployed the same way.

Q: Why DABs over clicking jobs together in the UI or raw Terraform?

A: Git as source of truth (diffable, reviewable, rollback-able), environment parameterization built in, and it deploys the job and the code together as one unit — avoids drift where the job definition in the UI points at a notebook version that's out of sync with what's in the repo. I do use Terraform too, but for the underlying Azure infra (workspace, storage, networking, RBAC) — DABs owns the workflow/job layer on top of that infra.

14. "How would you design a production Databricks pipeline?"

This is the capstone question — expect it, and answer with the diagram plus the reasoning behind each box, not just the shape.

ADF trigger (tumbling window / schedule)
      ↓
Databricks Workflow (job, parameterized by business_date + config_path)
      ↓
Bronze ingestion  ─── parallel per source ───┐
      ↓                                       │
Data Quality gate (schema, nulls, row counts) ┘  — fails fast, quarantines bad records
      ↓  (run_if: ALL_SUCCESS)
Silver transformation (MERGE INTO, per-domain fan-out)
      ↓
Gold aggregation (replaceWhere partition scope)
      ↓
Validation (row-count reconciliation, business-rule checks vs Silver)
      ↓  (run_if: ALL_SUCCESS)
Publish (advance watermark, expose to consumers / semantic layer)

          on-failure branch (run_if: AT_LEAST_ONE_FAILED)
                → Slack/webhook alert with run_id + failed task

Talking points to hit while walking through it:

• Retries: task-level, transient-only, small backoff; retries are safe because every write is idempotent.
• Checkpointing: Auto Loader/streaming sources use Delta checkpoints for exactly-once ingestion; batch sources use the watermark table advanced only after Publish succeeds.
• Logging: structured logs per task (run_id, business_date, rows in/out, duration) — enough to diagnose a failure without re-running.
• Metrics: row counts, null rates, freshness/lag, duration trends pushed to a metrics table or workspace monitoring (Lakehouse Monitoring on key tables) so degradation is visible before it's a hard failure.
• Alerting: failure webhook to on-call, duration-threshold warning separate from hard failure, DQ-gate failures alert with which check failed and sample of offending rows.
• Data-quality checks: schema enforcement, null/uniqueness/referential checks, row-count-delta vs expected volume — gated before Silver, not after.
• Idempotency: MERGE/partition-overwrite everywhere, parameterized by business_date, so retries and backfills are the same mechanism, not special cases.

Closing line that lands well: "The DAG shape is the easy part — the thing that actually makes it production-grade is that every stage is idempotent and parameterized by date, so retry, backfill, and normal run are all the same code path, not three different ones I have to maintain and test separately."

Quick-fire recap (for a last pass before the interview)