Databricks Data Engineering with AI Build a Lakehouse

Master medallion architecture on Databricks: PySpark, SQL, Unity Catalog, Genie Agent and Testing
What you'll learn

---

Build a complete medallion lakehouse on Databricks (Bronze, Silver, Gold) from an empty workspace, using real New York City taxi data
Use Genie Code to generate first-draft PySpark and SQL, then refactor it into typed, tested, production-grade modules
Write unit tests with pytest, and verify each stage against a known row-count anchor using an independent SQL query
Set up Unity Catalog, schemas and Volumes, and version the whole project in a Git-backed repository
Govern your lakehouse with Unity Catalog: catalogs, schemas, permissions and lineage across every layer
Build a semantic layer with Unity Catalog Metric Views: define each business metric once, then query it across any dimension
Query Metric Views with the MEASURE() clause, so every dashboard and question reads from one governed definition.
Set up a Genie Space so business users can ask questions of your Gold data in plain English.
Configure Genie with instructions, example SQL and trusted assets, and learn its limits and best practice.
Build AI/BI Dashboards on Databricks: visualise KPIs, add filters, and publish them for stakeholders.
Commit your work to GitHub at every step, building a clean audit trail.
Understand how Spark works: the DataFrame, lazy evaluation, the driver and executors, and Delta Lake.
Requirements

---

Basic Python. You can read and write simple functions.
Basic SQL. You can write SELECT queries.
A Databricks account. A free option is enough, and the early lessons walk you through the setup.
A GitHub account. It is free.
No prior Databricks, Spark or data engineering experience is required.
Description

---

AI can write data engineering code in seconds. Turning that first draft into a production-grade pipeline is the real skill, and it is what this course teaches.
You will build a complete lakehouse on Databricks, processing millions of real New York City taxi trips through a medallion architecture: raw Bronze, cleaned Silver, aggregated Gold. You will use Genie Code, Databricks' built-in AI assistant, to generate first drafts, then refactor them into typed, tested, version-controlled modules that meet a clear set of coding standards. This is a hands-on project, not a tour of features. You start with an empty workspace and finish with a full solution you can show an employer.
The course goes past the pipeline. You build it, you govern it, and you serve it to real users. Along the way you will
Set up Unity Catalog, schemas, Volumes, and a Git-backed repository from scratch
Ingest raw data into a Bronze Delta table and stamp lineage onto every row
Clean and conform the data in Silver, then aggregate it into Gold tables
Write and refactor PySpark and SQL across all three layers
Write unit tests with pytest, build reusable fixtures, and run them inside Databricks
Verify every stage against a known row-count anchor, on a separate engine, so your numbers are always right
Commit your work to GitHub at each step, building a clean audit trail
Then you make the data useful. You govern the lakehouse with Unity Catalog, so access and lineage stay under control. You build a semantic layer with Unity Catalog Metric Views, where each business metric is defined once and queried across any dimension with the MEASURE() clause. You set up a Genie Space, so business users can ask questions of your Gold data in plain English, backed by trusted metrics. And you build AI/BI Dashboards to track the KPIs and share them with stakeholders.
You will also learn the ideas behind the code, so nothing is a black box. The course explains how a Spark DataFrame works, why Spark is lazy, how the driver and executors split the work, and what Delta Lake gives you. You will review AI-generated code against eight coding standards, and configure the assistant with those standards so its output lands closer to production-ready every time.
The rhythm is simple and repeatable: build it, verify it, commit it. By the end you will have a portfolio-ready, end-to-end lakehouse, from raw files to governed metrics and dashboards, with AI as your assistant rather than your autopilot.
This course suits data engineers, analysts, and BI developers who know some Python and SQL and want to work the modern, AI-assisted way on Databricks. No prior Databricks experience is required.
Who this course is for

---

Data engineers who want a modern, AI-assisted workflow on Databricks
Data analysts who know some SQL and want to move into data engineering.
Python and software developers moving into data work.
Anyone who wants a real, portfolio-ready lakehouse project, not a feature tour.
People who want AI to write code faster, and want to make that code production-grade.
Homepage

---