
Already know some of this? A 2-minute test skips the lessons you know.
Take the placement testA free 22-lesson course built from freeCodeCamp.org's 3h 49m of YouTube video. 3 days at 90 minutes a day, with study notes and a quiz in every lesson. A 2-minute placement test skips what you already know.
Taught by freeCodeCamp.org on YouTube. Videos play in YouTube's own player, so the creator keeps every view.
Understand the course goals, the difference between basic ETL and senior platform engineering, and the consequences of ignoring pipeline resilience.
Implement API key authentication via headers, respect rate limits by checking headers and sleeping, and paginate through results using Link headers to retrieve all pages without memory overflow.
Write a normalization function that flattens nested dictionaries (actor, payload), extracts list items (topics) into separate records, parses ISO timestamps to Unix time, and safely handles missing nested fields.
Create a DuckDB connection, define a table schema manually, insert normalized records using parameterized SQL, and verify the loaded data.
Wrap a paginated API getter into a dlt resource using the @dlt.resource decorator, run a dlt pipeline to DuckDB, and observe automatic schema inference, nested field unpacking, and child table creation.
Store sensitive API tokens in environment variables or dlt's secrets vault, reference them in pipeline code, and avoid committing credentials to version control.
Explore the schema dlt infers from GitHub events, including root tables, unpacked nested structures (commits, payload), generated join keys (_dlt_id, _dlt_parent_id), and metadata tables (_dlt_loads, _dlt_version).
Define data contracts as agreements between producer and consumer, configure schema validation modes (evolve, freeze, discard_row, discard_value), and apply business rule checks to ensure data quality.
Configure write dispositions (append, merge, replace), implement incremental loading using a cursor field (created_at) to load only new or updated records, and avoid full reprocessing.
Use dlt with SQLAlchemy or ConnectorX to replicate databases, backfill historical data in chunks, and implement Slowly Changing Dimension (SCD2) to track history of mutable records like customer addresses.
Tune memory buffers for extraction/normalization/load, enable async HTTP requests to overlap network wait time, and set parallel workers per pipeline stage while respecting source/destination limits.
Switch destination from DuckDB to local filesystem (Parquet) and S3 (Iceberg), configure Hive-style or hidden partitioning, and register tables in AWS Glue or Athena catalogs for queryability.
Understand columnar MPP databases (BigQuery, Snowflake, Redshift), use file-based staging (Parquet on GCS/S3) for fast bulk loads, and configure dlt pipeline with staging and destination.
Create a GitHub Actions workflow YAML, store secrets in repository settings, trigger dlt pipeline runs on schedule or push, and verify deployment logs.
Set up a cron job on a VM, manage Python virtual environment paths, configure environment variables for credentials, and verify successful scheduled runs via log files.
Scaffold a Dagster project, wrap dlt resources in a Dagster asset, install the project locally, run `dagster dev`, and observe asset-based orchestration with lineage tracking.
Install Airflow locally via `airflow standalone`, initialize its database, create a DAG file importing dlt resources, run the scheduler/webserver, and trigger pipeline runs through the UI.
Use LLMs to generate dlt pipeline code from API docs, iterate on hallucinations (wrong destinations, missing partitioning), refine prompts with examples, and modernize legacy pipelines to Iceberg/DuckDB.
Recap key principles: continuous learning (20% time), social learning through implementation and sharing, and contributing personal dlt projects back to the community.