
Already know some of this? A 2-minute test skips the lessons you know.
Take the placement testA free 21-lesson course built from Darshil Parmar's 4h 31m of YouTube video. 3 days at 90 minutes a day, with study notes and a quiz in every lesson. A 2-minute placement test skips what you already know.
Taught by Darshil Parmar on YouTube. Videos play in YouTube's own player, so the creator keeps every view.
Understand the end-to-end AI data engineering project scope, the Zomato food delivery business context, and the high-level architecture covering S3, Snowflake, dbt, Airflow, and three AI layers.
Explore the seven CSV files (users, restaurants, orders, order_items, food, menu, reviews), their relationships via a star-schema data model, and the simulated data volumes (10M orders, 23M line items, 300k reviews).
Create an S3 bucket with a raw/ folder structure, manually upload each CSV into its dedicated folder, and learn why manual upload is used for clarity versus scripting for large-scale data.
Sign up for a Snowflake trial, create a storage integration with AWS IAM roles and external IDs for secure, credential-less access to the S3 bucket, and verify the trust relationship.
Create a CSV file format with delimiter, header skip, and null handling, then define an external stage pointing to the S3 raw folder using the storage integration to list and access files.
Create raw schema tables with all columns as strings to preserve fidelity, then use COPY INTO with the stage and file format to bulk-load each CSV from S3 into Snowflake.
Run `dbt init` to scaffold the project, configure profiles.yml with Snowflake credentials, define sources.yml pointing to the raw schema, and understand dbt's role as a transformation orchestrator.
Write staging models (stg_restaurants, stg_users, stg_food, stg_menu, stg_orders, stg_order_items, stg_reviews) that clean data types, handle nulls, and join via `ref()` to create a dependency graph.
Contrast ELT vs ETL, explain the bronze/silver/gold layers, and review dimensional modeling fundamentals (fact tables, dimension tables, star schema) before building the gold layer.
Create dimension tables (dim_restaurants, dim_customer with age segments, dim_date) and fact tables (fct_orders, fct_order_items), then build analytical data marts (daily city revenue, restaurant performance, delivery SLA) for BI-ready queries.
Add schema tests (unique, not_null, accepted_values) in staging.yml and marts.yml, run `dbt test`, generate docs with `dbt docs generate`, and explore the lineage graph to visualize model dependencies.
Containerize Airflow with Docker Compose mounting the dbt project, write a DAG with SQLExecuteQueryOperator (COPY raw) and BashOperator (dbt build), fix profile mounting, and trigger a successful scheduled run.
Learn why AI belongs in data pipelines, how LLMs predict tokens, what embeddings and cosine similarity enable, and the applied RAG pattern (embed → retrieve → augment → generate) without deep research math.
Build enrich_reviews.py: connect to Snowflake, create an ai.enrichment table, fetch unprocessed reviews, call OpenAI chat.completions with a structured system prompt to output JSON (sentiment_label, score, topic, key_issue), and write results back to Snowflake.
Build rag_chat.py with Streamlit: load reviews, embed with OpenAI, cache embeddings, implement cosine similarity search for top-k relevant reviews, feed context + question to LLM, and render answers with source review expanders.
Build text_to_sql.py: define schema context for marts tables, write a system prompt enforcing SELECT-only and forbidden DDL/DML words, embed the question, retrieve relevant schema context, generate SQL via LLM, execute safely on Snowflake, and display results in Streamlit.
Extend the Airflow DAG to include AI tasks: run enrichment script, build dbt models tagged 'ai', and schedule the complete end-to-end batch + AI pipeline for daily automated execution.
Review the completed architecture (S3 → Snowflake → dbt → Airflow → AI layers), suggest incremental loading improvements, revoke API keys, clean up cloud resources, and outline further learning paths on the DataVidhya platform.