
Already know some of this? A 2-minute test skips the lessons you know.
Take the placement testA free 24-lesson course built from DataBeli's 4h 15m of YouTube video. 3 days at 90 minutes a day, with study notes and a quiz in every lesson. A 2-minute placement test skips what you already know.
Taught by DataBeli on YouTube. Videos play in YouTube's own player, so the creator keeps every view.
Understand what Azure Data Factory is, its key features (data ingestion, orchestration, transformation, scheduling, monitoring, security), and why it is a popular serverless data integration service.
Create a free Azure account, set up a resource group, and deploy an Azure Data Factory instance using the Azure portal.
Identify the core building blocks of ADF—linked services, datasets, integration runtimes (Azure vs self-hosted), and pipelines with activities—and how they connect sources to targets.
Provision an Azure SQL Database, configure firewall rules, create a linked service in ADF using SQL authentication, and test the connection.
Build a pipeline with a Lookup activity to read rows from a SQL table, inspect JSON output, and filter results with a custom query.
Create an ADLS Gen2 storage account with hierarchical namespace, build a binary-copy pipeline, and land data from Azure SQL into ADLS using the Copy Data activity.
Add dataset parameters for table and schema names, use dynamic expressions (e.g., toLower) to make datasets reusable across tables.
Create a Logic App with an HTTP trigger and Gmail action, parameterize subject and body, and test email delivery from the Logic Apps designer.
Use the Web activity in ADF to POST to the Logic App URL, pass pipeline parameters dynamically via JSON payload, and verify end-to-end email notification.
Build a consolidated email pipeline using Set Variable and Append Variable activities to collect per-table status, handle JSON escaping issues with replace(), and send a single summary email after a ForEach loop.
Create a watermark table in SQL, populate initial timestamps, and design a lookup activity to read the last watermark value per table.
Chain watermark lookup, parameterized copy data (with dynamic file names using concat and utcNow), and watermark update via a script activity to achieve end-to-end incremental ingestion.
Add a Lookup to count incremental rows, then use an If Condition activity to run Copy Data only when new data exists, otherwise skip to failure path.
Read a table list from metadata, iterate with ForEach, pass each table name to a child pipeline via Execute Pipeline, and control sequential vs parallel execution with batch count.
Append per-table success/failure status to an array variable inside ForEach, fix JSON quoting issues with replace(), and send a consolidated email after loop completion.
Create source/target containers in ADLS, upload sample files, and build a binary Copy Data pipeline to move files without format conversion.
Retrieve file metadata (child items, size, last modified) with Get Metadata, then use it to drive conditional logic in subsequent activities.
Insert a Wait activity for delay, then use Delete activity with wildcard and recursive options to remove filtered files from the source container.
Configure Databricks and Synapse linked services, trigger notebooks/jobs with parameters, and understand Data Flow as a low-code alternative.
Create schedule-based triggers (daily, weekly, monthly), tumbling window triggers, and understand custom Event Grid triggers.
Configure a Storage Event trigger on blob creation/deletion, pass the triggering file name to the pipeline via trigger parameters, and verify event-driven ingestion.
Navigate the Monitor hub: filter pipeline runs, inspect activity inputs/outputs, view Gantt charts, export CSV, and use pipeline run IDs for debugging.
Connect ADF to a GitHub repo, work on feature branches, create pull requests, publish from main branch, and use JSON view for bulk search/replace across pipelines.
Create an Azure Key Vault, grant ADF managed identity access, store SQL passwords as secrets, and reference them in linked services for secure credential management.