Powering Real-time Analytics with Apache Kafka and Spark

1. us dsi © Copyright 2025. United States Data Science Institute. All Rights Reserved .org

2. rom sensors to social media and online transactions to browsing activities, F primary driver behind this data explosion. (Source: Statista) the amount of data generated today is enormous. An estimated 463 zettabytes of data are generated every day, with the internet being the The data generated at an unprecedented rate presents great opportunities for organizations. However, it is real-time analytics that makes a business distinguished and stay ahead of the competition. Data Science Pipeline for Real-Time Analytics Real-time analytics help organizations analyze data right at the moment it is generated so that organizations can take immediate action. Today, technologies like Apache Kafka and Apache Spark play a very important role in powering this real time analytics. Apache Kafka It is a distributed event streaming platform designed for high-throughout, real-time data processing. It was developed by LinkedIn and now it is part of the Apache Software Foundation. Through this, applications can publish, subscribe to, store, and process streams of records easily. It is widely used to build real-time data science pipelines and streaming applications. Apache Kafka can handle millions of events per second across all industries. Since it can support multiple producers and consumers, it has become an ideal choice for several applications like log aggregation, fraud detection, and data integration between systems. Apache Spark Apache Spark is an open-source, distributed computing system that helps with faster data processing and analytics. It is known for its speed and ease of use in the industry. It can handle huge data workloads even from clusters of computers. Apache Spark supports various tasks such as batch processing, real-time streaming, machine learning, graph computation, etc. It has in-memory processing capabilities that can significantly boost performance as compared to other traditional frameworks like Hadoop MapReduce. Because of its APIs in Java, Python, R, and Scala, it is used in applications like building data applications and advanced big data analytics. us dsi © Copyright 2025. United States Data Science Institute. All Rights Reserved .org

3. Building a Data Science Pipeline for Real-time Analytics Below are the steps to build an effective data science pipeline using Apache Kafka and Spark for real-time analysis. 1. Setting Up Kafka The first step is to download and install Kafka. Visit Apache Kafka to download the latest version and then extract it to your preferred directory. However, note that it requires Zookeeper to run. So, you need to start Zookeeper before launching Kafka. Kafka will start itself once Zookeeper is up and running. bin/zookeeper-server-start.sh config/zookeeper.properties bin/kafka-server-start.sh config/server.properties The next step is creating a topic to send and receive data. For this example, we will use the topic sensor_data. bin/kafka-topics.sh --create --topic sensor_data --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 Kafka is now ready to receive data. 2. Setting Up Kafka Producer A Kafka Producer is required to send data to Kafka topics. In this case, we’ll create a Python script to simulate a sensor producer. It will generate random sensor readings like temperature, humidity, and sensor IDS, which will be published to a Kafka topic named sensor_data. from kafka import KafkaProducer import json import random import time # Send data to Kafka topic every second while True: data = { 'sensor_id': random.randint(1, 100), 'temperature': random.uniform(20.0, 30.0), 'humidity': random.uniform(30.0, 70.0), 'timestamp': time.time() } producer.send('sensor_data', value=data) time.sleep(1) # Send data every second This script will generate random sensor data and send it every second to sensor_topic. us dsi © Copyright 2025. United States Data Science Institute. All Rights Reserved .org

4. 3. Machine Learning for Real-Time Predictions At last, we will leverage machine learning for real-time predictions. We will be using Spark’s MLlib library to create a simple logistic regression model. This model will then predict if the temperature is high or normal, depending on sensor data. from pyspark.ml.classification import LogisticRegression from pyspark.ml.feature import VectorAssembler from pyspark.ml import Pipeline # Prepare features and labels for logistic regression assembler = VectorAssembler(inputCols=["temperature", "humidity"], outputCol="features") lr = LogisticRegression(labelCol="label", featuresCol="features") # Create a pipeline with feature assembler and logistic regression pipeline = Pipeline(stages=[assembler, lr]) # Assuming sensor_data_df has a 'label' column for training model = pipeline.fit(sensor_data_df) # Apply the model to make predictions on real-time data (without displaying) predictions = model.transform(sensor_data_df) This code will create a logistic regression model. It will train the model with available data and use it to predict whether the temperature is high or normal. Real-Time Data Analytics Pipeline: Best Practices 1. Ensure that Kafka and Spark are scalable and handle more data with the growth of systems. 2. Optimize Spark resource usage to avoid system overloads and ensure efficient processing. 3. Implement a schema registry to manage changes in the Kafka data structure smoothly 4. Define suitable data retention policies in Kafka to control storage duration effectively 5. Tune Spark’s batch size to balance processing speed with data accuracy. The Final Thoughts! Real-time data analysis can be made smooth and effortless using the powerful tools Kafka and Spark. While Kafka is used to collect and store incoming data, Spark is an efficient tool to process and analyze data rapidly. Together, both these tools can help businesses make decisions faster. Moreover, using machine learning with Spark also offers real-time predictions and makes the system more useful. Learn how to make your data ready and prepare your organization for data readiness to unleash the true potential of AI, with Top Data Science Certifications from USDSI . ® GET USDSI CERTIFIED. REGISTER TODAY! ® usdsi.org © Copyright 2025. United States Data Science Institute. All Rights Reserved

5. GET CERTIFIED Texas 539 W. Commerce St #4201 Dallas TX 75208 info.tx@usdsi.org © Copyright 2025. United States Data Science Institute. All Rights Reserved