
Already know some of this? A 2-minute test skips the lessons you know.
Take the placement testA free 22-lesson course built from Alpha Brains Courses's 3h 49m of YouTube video. 3 days at 90 minutes a day, with study notes and a quiz in every lesson. A 2-minute placement test skips what you already know.
Taught by Alpha Brains Courses on YouTube. Videos play in YouTube's own player, so the creator keeps every view.
Explains the four big data challenges and how Kafka specifically addresses the data transfer challenge with focus on security, reliability, speed, availability, and scalability.
Details LinkedIn's evolution from a monolithic app to SOA, the problems with custom pipelines, and the birth of Kafka as a universal pipeline based on commit logs.
Covers Kafka's open-source journey, Scala/Java implementation, definition as a distributed streaming platform, and core concepts: clusters, records, topics, partitions, and immutability.
Compares Kafka with AMQP, RabbitMQ, JMS, Amazon Kinesis, and ActiveMQ, then highlights Kafka's strengths in scalability, reliability, performance, and durability.
Examines how Twitter, Netflix, and Yahoo integrate Kafka into their architectures for real-time analytics, monitoring, and data pipelines, demonstrating Kafka's role in existing systems.
Explains the shift from monolithic to distributed systems, the role of Kafka brokers and Zookeeper, publish-subscribe model, and the producer/consumer relationship with topics.
Defines topics as message categories, partitions for scalability and redundancy, key-based partitioning via hashing, and consumer offsets for tracking read position.
Details Kafka's three roles: real-time messaging with fault-tolerant storage (default 7-day retention), and stream processing via Kafka Streams API for real-time transformations.
Covers production hardware requirements (memory, CPU, disk, network), Linux file system tuning (noatime, XFS options, journaling trade-offs), and kernel parameter optimization for Kafka performance.
Guides through JDK installation verification and setup, then Zookeeper installation via package managers (yum/dnf/apt) or manual tarball, and essential configuration (tickTime, dataDir, clientPort).
Demonstrates Kafka broker download, extraction, startup, and topic creation/alteration using kafka-topics.sh with replication factor, partitions, and Zookeeper connection.
Shows end-to-end verification using kafka-console-producer.sh and kafka-console-consumer.sh, then assigns a hands-on lab to install OS, JDK, Zookeeper, Kafka, and test a topic.
Explains cluster architecture (brokers, Zookeeper metadata, partitions), replication factor, leader/follower model, ISR (in-sync replicas), quorum, and commit acknowledgment flow.
Details required producer configs (bootstrap.servers, key/value serializers), optional tuning (acks, retries, batch.size, linger.ms, buffer.memory), and serialization choices (String, Avro, JSON, Protobuf, Thrift).
Compares fire-and-forget, synchronous, and asynchronous send modes with trade-offs; explains Avro/JSON schema registry usage to avoid message bloat; demonstrates partition key hashing.
Covers consumer offset tracking, consumer groups for parallel partition consumption, leader-based partition assignment, rebalance triggers (add/remove/crash), and rebalance listener hooks (onPartitionsRevoked/Assigned).
Walks through consumer properties (bootstrap, deserializers, group.id), subscribe/poll loop, and three commit strategies: auto (5s default), sync (blocking), async (non-blocking with callback), including enable.auto.commit=false nuance.
Demonstrates kafka-topics.sh --create/--list/--describe/--alter/--delete; explains partition addition (no removal), replication factor, and topic detail inspection.
Explains automatic leader re-election via auto.leader.rebalance.enable, manual re-election with kafka-preferred-replica-election.sh, and mirroring topics across clusters with kafka-mirror-maker.sh using whitelist/blacklist.
Shows how to set default producer/consumer byte-rate quotas in server.properties and custom per-client/group quotas via kafka-configs.sh --alter to prevent broker denial-of-service.
Introduces Kafka Connect for ingesting external data; covers standalone/distributed modes, connector configs (offset.storage, key/value converters), transformations (SMTs), and REST endpoints for connector lifecycle management.