Senior Data Engineer Guide
Model answers, theory primers, gotchas and quick-fire recaps for senior data engineering interviews: Databricks, Delta Lake, Spark, SQL, data quality, GenAI, Fabric, Azure and AWS.
1. Databricks Jobs / WorkflowsJobs vs tasks, dependencies, retries, idempotency, backfills and CI/CD with Asset Bundles: model answers for Databricks Workflows interviews.2. Delta Lake Core ConceptsTransaction log, ACID, MERGE internals, OPTIMIZE, VACUUM, Z-ordering, Change Data Feed and a CDC failure-scenario walkthrough.3. Unity Catalog / Governance & SecurityUnity Catalog hierarchy, grants, service principals, external locations, lineage, row/column security and securing sensitive data on Azure.4. Spark Internals & PerformanceDriver and executors, shuffle, AQE, join strategies, skew and salting, plus a ten-step method for debugging a slow Spark job.5. SQL PracticeWindow functions, deduplication, SCD Type 1 and 2, gaps and islands, top-N per group, incremental loads and query tuning, with worked SQL.6. Data Quality & ReconciliationThe seven data-quality dimensions with concrete checks, a stage-by-stage reconciliation framework, and diagnosing source-vs-target mismatches.7. AI / GenAI EngineeringRAG retrieval and evaluation, faithfulness, multi-tenant filtering, agent design, tool-call loops, MCP security and production monitoring.8. Microsoft FabricOneLake, capacity smoothing and throttling, Lakehouse vs Warehouse, Direct Lake, security, CI/CD and Fabric vs Databricks trade-offs.9. Azure Synapse Analytics & Azure Data FactoryADF triggers, integration runtimes, dedicated SQL pool distribution and indexing, serverless SQL, CI/CD with adf_publish, and migration choices.10. Data Engineer: Ingestion, Reliability, Large-Scale Analysis, Team LeadIngestion pipelines that don't break, monitoring, debugging at scale, analysis on billions of rows, stakeholders and leading a team.11. System Design: Near Real-Time EV Charging (OCPP) Pipeline on DatabricksA model system-design answer: near real-time EV charging events on Databricks with dedup, late data, quarantine and a sessions Gold table.12. Case Study: A Real Near-Real-Time Lakehouse Mapped to OCPPThe OCPP design from Part 11 mapped onto a working near-real-time lakehouse on Databricks: what such a pipeline typically has, what OCPP adds, and the lessons it teaches.13. AWS Data Platform Engineer/Architect — Interview GuideAWS data platform interview guide: S3, Glue, EMR, Redshift, Kinesis, Lake Formation, governance and architecture trade-offs.