Databricks Data Engineering The Real World Simulator

Do the job, not a tutorial. Build, defend and ship an end to end Databricks lakehouse, with real engineering tickets!
What you'll learn

---

Build a production Databricks lakehouse end to end, from ten raw source feeds to a live operational app
Make the design decisions a data engineer is paid for, and defend them in review
Handle source data as it really arrives: duplicates, late rows, CDC and slowly changing history
Choose between imperative and declarative pipelines with the confidence of having shipped both
Own a pipeline in production, keeping it trustworthy and fixing it when it fails overnight
Requirements

---

Hands on Databricks experience.
Working knowledge of Python, SQL and PySpark DataFrames.
An intermediate course. Fundamentals are assumed.
Willingness to create a Databricks Free Edition account. No credit card, no cloud subscription.
Description

---

Helios Trading Corporation needs a Data Engineer. That is you.
Helios ships parts across six depots in the solar system. Ten source feeds land every day. Nobody has built the pipeline yet.
This is not a tutorial you follow. It is a job you do.
Simulate, then Execute
Every build starts as a conversation. A short AI Role Play with a colleague, and goals you are measured against.
Then you build what you talked through. Most labs are work tickets: context, task, and acceptance criteria that run as real checks in the notebook.
What you will build1. Design Meet Helios, walk the ten source feeds with the person who owns them, and agree the Medallion lakehouse design that every later section implements.
2. Databricks Jobs The classic build. Auto Loader into Bronze, Structured Streaming and MERGE into Silver, SCD Type 2 for prices and customer tiers, a Gold star schema with point in time pricing, then a five task orchestrated Job with retries, alerts and a schedule.
3. Lakeflow Spark Declarative Pipelines The same platform, rebuilt declaratively. Streaming tables, expectations as quality gates, AUTO CDC. Then an inherited pipeline fails on a night run and you diagnose it from the event log.
4. Genie and AI/BI Dashboards A semantic layer of Metric Views over Gold, so one definition of revenue feeds both a Genie Space and a dashboard. Then you make the case for self serve to the Commercial Director.
5. Lakebase and a Databricks App Sync curated Gold into Lakebase and deploy a live Depot Operations Console.
Source feeds to running app.
Real data, from a purpose built generator
The data comes from a custom synthetic generator written for this course, not a downloaded sample set. Ten interlocking feeds in three formats, around 293,000 order lines across five batches, and it is deterministic, so your numbers match the videos exactly.
The domain is order to cash for a parts distributor. Customers place orders, orders move through a lifecycle, stock leaves the shelf, prices and costs change over time, and some of it comes back as returns. Sales, inventory, pricing and returns, the same shapes you meet in retail, wholesale, manufacturing and logistics.
It behaves like production data too
Duplicate order lines
Late arrivals from the depots with poor uplinks
Dirty quantities and missing keys
An order lifecycle arriving as CDC
Prices that change underneath your joins
No cloud account. No credit card. No Azure subscription.
The course runs end to end on Databricks Free Edition. The skills transfer directly to Azure, AWS and GCP.
Every lab bootstraps to a clean state in under two minutes, so you can start at any section.
Current for 2026
Built on what Databricks ships today
Lakeflow Spark Declarative Pipelines, formerly Delta Live Tables
Lakeflow Jobs
Unity Catalog and Volumes
Serverless compute
Genie Spaces
Metric Views and AI/BI Dashboards
Lakebase and Databricks Apps
These patterns also cover much of the ground tested by the Databricks Certified Data Engineer Associate and Professional exams, though this is a course about doing the job, not passing an exam.
Your instructor
Malvik Vaghadia is a Databricks Partner Champion and a Principal Data Engineering Consultant. He has taught more than 250,000 students across 15 courses on Databricks, PySpark, Delta Lake and Azure.
Helios is waiting. Start your first day.
Who this course is for

---

Anybody who wants to level up as a Databricks Data Engineer
Data engineers who want experience that looks like the job, not another feature tutorial
Databricks users ready to put the features they know into one platform that runs like production
Engineers heading into a data engineering role or interview who want real work to speak from
Homepage

---