
Provide a single SparkSession factory with a defined app name, master and shuffle partitions setting that survives a stop, and build a DataFrame from seeded in-memory rows.
Rewrite a driver-side price list loop as DataFrame transformations: select, filter, derive decimal columns, rename, de-duplicate and sort.
Restructure a PySpark report whose debug row counts trigger a job after every step, so building the chain runs no job, producing it runs only the jobs of one collect, and the source DataFrame is left unchanged.
Build a supplier price list normaliser as one module: a single configured SparkSession factory that survives a stop, the input DataFrame from seeded rows, and a lazy chain that derives retail price and margin, filters, renames, de-duplicates and sorts into a catalogue feed that is collected once.
Learn Spark hands-on, from DataFrames and schemas to joins, performance tuning, and streaming pipelines.
This path starts Spark with DataFrames: creating a SparkSession and transforming data, then schemas, types and nulls. You should be comfortable with basic programming and with filtering and grouping rows of data, as you would in SQL. No earlier Spark experience is needed. The schemas module is about getting types right and ends with a typing layer for customer orders.
2 modules: SparkSession & DataFrame Basics and Schemas, Types & Nulls.
8 exercises, all easy.
Challenges run on Airflow.
No. Every challenge runs in a browser-based IDE with everything already set up, and your code is graded automatically against a test suite.