Use Code TRYNOW15 for a One-Time, Extra 15% OFF at KodeKloud
Data

Event Streaming with Kafka

Learn to build real-time systems that react instantly to streaming data with Apache Kafka
Raghunandana Sanur
Staff Data Engineer & MLOps Engineer at Talabat
DevOps Pre-Requisite Course
Play Button
Fill this form to get a notification when course is released.
book
8
Lessons
book
Challenges
Article icon
62
Topics

What you’ll learn

Our students work at..

Description

Imagine building systems that react instantly to streams of data—fraud alerts triggered within milliseconds, dashboards pulsing with live updates, and digital platforms scaling to millions of users in real time. In this hands-on course on Apache Kafka, you'll learn the basics of these capabilities to life. Through a dynamic mix of real-world examples, interactive labs, and practical demonstrations, you’ll progress from the basics of event streaming to deploying advanced, production-ready architectures.

Course Outline:

Foundations of Event Streaming

Begin with a solid understanding of event streaming concepts and event-driven architecture, exploring how Apache Kafka powers real-time data flows in industries like finance and beyond. You’ll examine critical use cases, deploy your own Kafka cluster and user interface using Docker, and create your first Kafka topic.

Building Blocks of Kafka

Dive into Kafka’s core architecture, including brokers, topics, partitions, and replication, learning how these elements work together to store, organize, and reliably deliver streaming data.

Kafka Producers & Consumers: The Message Flow

Understand how producers and consumers form the backbone of Kafka's event pipeline. Learn message serialization, the significance of message keys, and reliability strategies like acknowledgments and consumer groups. Reinforce concepts by building and configuring producers and consumers, exploring advanced operations like rebalancing.

Deep Dive into Kafka: Beyond the Basics

Advance your skills by looking at Kafka’s offset management, error handling (including poison pill scenarios), and the roles of ZooKeeper and KRaft for cluster management. 

Confluent Kafka and its Offerings

Explore the operational complexities of running Kafka at scale, and discover how Confluent Cloud simplifies management and deployment for cloud-native event streaming.

Kafka Connect: Effortless Data Pipelines

Learn how Kafka Connect enables seamless streaming of data to and from external systems. Build your own pipeline to stream Kafka data to Amazon S3, gaining practical insights into connector configuration and deployment.

Building an Event-Driven System

Design and implement a complete event-driven architecture utilizing Kafka, from infrastructure setup on AWS EC2 to frontend and backend integration. Consolidate your skills with a capstone lab, bringing together all course concepts for real-world application in an interactive environment.

Target Audience:

This course is ideal for software developers, DevOps engineers, data engineers, and technical leads eager to design, deploy, and manage real-time, event-driven systems using industry-leading technologies.

By the end of this course, you’ll be fully equipped to leverage Kafka and related tools for real-time data streaming, unlocking new levels of innovation and efficiency for your projects and organization.

Read More

What our students say

About the instructor

Raghunandana Krishnamurthy is a seasoned Staff Data Engineer and MLOps expert, skilled in navigating both GCP and AWS cloud platforms to accelerate model development and deployment. His experience spans modernizing legacy data systems, architecting hybrid infrastructures, and ensuring data quality for diverse applications. He used to hold  Associate AWS Solution Architect certification, Cloudera Hadoop Admin certification, Airflow certification, and Databricks Lakehouse certification 

A technical leader and passionate trainer, Raghunandana excels at building and maintaining big data platforms, championing DevOps best practices, and fostering team alignment. With hands-on expertise in tools like SageMaker, VertexAI, Prometheus, Grafana, and extensive DevOps tools focusing on Data Engineering and MLOps.

No items found.

Introduction

lock
lock
2
Topics
Lesson Content

Module Content

Course Introduction 03:03
How to Reach Out to KodeKloud and Engage with the Community

Foundations of Event Streaming

lock
lock
9
Topics
Lesson Content

Module Content

What is Event Streaming? 04:44
Introduction to Apache Kafka 02:36
Event-Driven Architecture Basics 09:36
Kafka in Finance: Real-time Transaction Processing 03:21
Use cases and Real-World Applications -- Use Case 2 04:37
Demo: Setting up Kafka Cluster and Kafka UI using Docker 04:33
Demo: Creating First Topic in Kafka 01:31
Quiz: Foundations of Event Streaming
Lab: Setting up Kafka Cluster and Kafka UI using Docker and creating your first topic

Building Blocks of Kafka

lock
lock
7
Topics
Lesson Content

Module Content

Kafka Brokers: The Foundation of Message Storage 07:46
Understanding Kafka Topics: Organizing Your Data Streams 07:27
Kafka Partitions: Scaling and Parallelizing Your Data Processing 07:16
Kafka Replication: Ensuring Data Reliability and Fault Tolerance 05:22
Demo: Topics, Partitions, and Brokers 06:05
Quiz: Building Blocks of Kafka
Lab: Topics, Partitions, and Brokers

Kafka Producers & Consumers: The Message Flow

lock
lock
15
Topics
Lesson Content

Module Content

What is a Kafka Producer? 02:58
Demo: Kafka Producers 07:33
Lab: Kafka Producers
Serialization in Kafka 05:03
Understanding the Role of Message Keys 05:54
Producer Acknowledgments (Acks) and Reliability Guarantees 04:06
What is a Kafka Consumer? 05:34
Demo: Kafka Consumers 05:19
Lab: Kafka Consumers
Consumer Groups and How They Work 07:09
Consumer Rebalancing 03:46
Demo: Consumer Rebalancing 08:49
Lab: Consumer Rebalancing
Quiz: Kafka Producers & Consumers: The Message Flow
Section Recap 00:53

Deep Dive into Kafka: Beyond the Basics

lock
lock
14
Topics
Lesson Content

Module Content

Offset Management in Kafka 07:13
Poison Pill in Kafka 02:54
Mitigation Strategies - Handling Poison Pill 03:25
ZooKeeper's Role in Kafka 07:33
Demo: Kafka Setup with Zookeeper 04:09
Lab: Kafka Setup with Zookeeper
Kafka KRaft 06:46
KRaft in Action : New Broker joining Kafka Cluster 02:55
Demo: Kafka Setup with KRaft 04:27
Lab: Kafka Setup with KRaft
Kafka Security 03:28
Securing Kafka 06:09
Quiz: Deep Dive into Kafka: Beyond the Basics
Section Recap 03:18

Confluent Kafka and Its Offerings

lock
lock
4
Topics
Lesson Content

Module Content

Beyond Installation: The True Complexity of Managing Kafka 04:01
Confluent Cloud: Kafka Reimagined for the Cloud Era 07:40
Quiz: Confluent Kafka and its offerings
Demo: Getting Started with Confluent for Free 08:31

Kafka Connect: Effortless Data Pipelines

lock
lock
5
Topics
Lesson Content

Module Content

Streaming data from Kafka to other systems 07:44
Demo: Kafka Connect – Setting up Kafka using KRaft 08:08
Demo: Setting up S3 Kafka Connect 10:18
Demo: Kafka Connect in Action 03:37
Lab: S3 Kafka connect

Project: Building an Event-Driven System

lock
lock
6
Topics
Lesson Content

Module Content

Architecture Overview of Event-Driven System 03:04
Demo: Setting up Kafka on EC2 10:04
Demo: Starting Our Frontend Interface 07:07
Demo: Starting Our Internal Interface 04:35
Lab: Building an Event-Driven System
Section Recap 01:11
Play Button
Fill this form to get a notification when course is released.
This course comes with hands-on cloud labs
book
8
Modules
book
Lessons
Article icon
62
Lessons
check mark
Course Certificate
Videos icon
04.22
Hours of Video
laptop
Hours of Labs
Story Format
Videos icon
Videos
Case Studies
ondemand_video icon
Demo
laptop
Labs
laptop
Cloud Labs
checklist
Mock exams
Quizzes
Discord Community Support
people icon
Community support
language icon
Closed Captions