
Already know some of this? A 2-minute test skips the lessons you know.
Take the placement testA free 26-lesson course built from Ansh Lamba's 5h 54m of YouTube video. 4 days at 90 minutes a day, with study notes and a quiz in every lesson. A 2-minute placement test skips what you already know.
Taught by Ansh Lamba on YouTube. Videos play in YouTube's own player, so the creator keeps every view.
Understand the course structure, instructor background, and what Apache Spark is as a distributed computing engine.
Explain the master-slave architecture of Spark including the driver node, cluster manager, and worker/executor nodes.
Read a CSV file into a DataFrame using spark.read with format, inferSchema, and header options, and inspect the file path via dbutils.fs.ls.
Read a single-line JSON file using spark.read.format('json') with multiLine=false option and verify the data.
Define custom schemas using DDL strings and StructType/StructField to override inferred types, and apply them during data reading.
Apply column selection, row filtering (including complex boolean logic with isNull/in), column renaming, adding constant columns with lit, and deriving new columns from existing ones.
Cast column types with cast(), manipulate dates by adding/subtracting days, compute day differences with datediff, and reformat date strings with date_format.
Drop rows with nulls using how='any' or 'all', fill nulls with constant values, and target specific columns using subset parameter.
Transform string case with initcap, lower, upper, and rename the resulting column using alias for cleaner output.
Replace specific string values in a column (e.g., 'Regular'->'R', 'Low Fat'->'LF') using regexp_replace chained via withColumn.
Perform groupBy aggregations including sum, avg, grouping by multiple columns, applying multiple aggregations at once, and collecting values into lists with collect_list.
Reshape data with pivot (rows to columns) and unpivot using stack, applying aggregations like sum/avg in the pivot table.
Execute inner, left, right, and anti (left-anti) joins on DataFrames, understanding which records are kept or dropped in each type.
Generate row numbers, ranks, and dense ranks over windows defined by partitionBy and orderBy clauses for deduplication and surrogate keys.
Access neighboring rows with lead/lag, retrieve boundary values with first/last_value, and bucket rows into percentiles with ntile.
Create a Python function, register it as a Spark UDF via udf(), apply it in withColumn, and understand the serialization performance penalty.
Write DataFrames to CSV and Parquet using DataFrameWriter API, control behavior with save modes, and partition output by column values.
Distinguish managed (Spark controls data+metadata) from external tables (user controls location), create temporary views, run SQL via spark.sql, and convert results back to DataFrames.