StaffCoder
Spark
SparkAirflow
2 modules · 8 challengesSign up
Module 2 of 2 · 4 challenges

Spark Schemas, Types & Nulls

Declare explicit schemas, cast safely under ANSI rules, and handle nulls with a clear, deliberate policy.
Finish every challenge, or complete the star challenge, to complete the module.

Challenges

4 challenges

Say What You Mean

Sign up to unlock
—No data yeteasy
About this challenge

Declare an explicit PySpark schema for orders with exact decimal money, dates, a nested address struct and nullability, and apply it so bad rows are rejected at creation.

Solved in a browser IDE and graded by tests. Sign up to solve

Casting Under ANSI Rules

Sign up to unlock
—No data yeteasy
About this challenge

Fix a PySpark typing job that switched ANSI mode off: keep ANSI on, convert string columns with tolerant try_* casts and a per-row reject count, and let the strict cast, integer overflow and division by zero raise their ANSI errors.

Solved in a browser IDE and graded by tests. Sign up to solve

Nulls Are Not Values

Sign up to unlock
—No data yeteasy
About this challenge

Fix a customer null policy in PySpark: a null-safe not-equal filter that keeps customers with no region, defaults filled only in the defined columns, drops only when the defined columns are null, a first-non-null contact in priority order, and a sort that places nulls last.

Solved in a browser IDE and graded by tests. Sign up to solve

Customer Order Typing Layer

STAR
Solving it completes the module
—No data yeteasy
About this challenge

Build a PySpark typing layer for raw customer orders: an explicit nested landing schema that rejects bad rows at creation, ANSI-safe tolerant and strict conversions with a per-order reject count and rejects summary, and a null policy of null-safe filtering, per-column defaults, a drop rule, a first-non-null contact and nulls-last sorting, with an exact decimal invoice total.

Solved in a browser IDE and graded by tests. Sign up to solve

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.