A new breed of ‘Fast Data’ architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. - Dean Wampler (Renowned author of many big data technology-related books)
Dean Wampler makes an important point in one of his webinars. The demand for stream processing is increasing every day in today’s era. The main reason behind it is, processing only volumes of data is not sufficient but processing data at faster rates and making insights out of it in real-time is very essential so that organizations can react to changing business conditions in real-time.
And hence, there is a need to understand the concept of “stream processing “and the technology behind it.
Spark Streaming Vs Kafka Stream
Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. The following table briefly explains, the key differences between the two.
| Sr.No | Spark Streaming | Kafka Streams |
|---|
| 1 | Data received from live input data streams is Divided into Micro-batched for processing. | processes per data stream(real real-time) |
| 2 | A separate processing Cluster is required | No separate processing cluster is required. |
| 3 | Needs re-configuration for Scaling | Scales easily by just adding java processes, No reconfiguration required. |
| 4 | At least one semantics | Exactly one semantics |
| 5 | Spark streaming is better at processing groups of rows(groups,by,ml,window functions, etc.) | Kafka streams provide true a-record-at-a-time processing capabilities. it's better for functions like row parsing, data cleansing, etc. |
| 6 | Spark streaming is a standalone framework. | Kafka stream can be used as part of microservice, as it's just a library. |
| 7 | Kafka stores data in Topic i.e., in a buffer memory. | Spark uses RDD to store data in a distributed manner (i.e., cache, local space) |
| 8 | It supports multiple languages such as Java, Scala, R, and Python. | Java is the primary language that Apache Kafka supports. |
Spark Streaming Vs Kafka Stream: Detailed Comparision
1. ETL Transformation
In this case, Kafka doesn't offer only ETL services. Instead, it streams data from source to destination using the Kafka Connect API and the Kafka Streams API. You may make data streams with Kafka using the Kafka Connect API (E and L in ETL).
Given that it is built on Kafka's failover strategy, the Connect API benefits from Kafka's scalability. As a result, it offers a single method for managing all connections. For stream processing and transformations, one can utilize the Kafka Streams API, which offers T in ETL.
Due to the fact that Spark enables users to retrieve, store, and alter data. And the ETL procedure is made possible by shifting it from source to destination.
2. Latency
Spark is a superior choice if latency is not a concern (in comparison to Kafka) and you want source freedom with compatibility. Kafka is the best option, nevertheless, latency is a severe issue and real-time processing with time frames shorter than milliseconds is needed.
Kafka offers better fault tolerance because of its event-driven processing. Its interoperability with other kinds of systems, however, might appear to be extremely difficult.
3. Processing Type
Kafka analyses events as they often take place. A continuous processing model is an outcome. Spark divides the input streams into tiny batches for processing using the micro-batch processing technique.
4. Language supported
While Spark is renowned for supporting a wide range of programming languages and frameworks, Kafka does not support any programming language for data transformation. In other words, because Apache Spark uses current machine learning frameworks and processes graphs, it has the ability to do more than merely understand data.
5. Memory Management
RDD is used by Spark to store data in a distributed fashion (i.e., cache, local space). Spark's primary data structure is Resilient Distributed Datasets (RDD). It is a distributed collection of immutable things.
Each dataset in an RDD is split into logical divisions that may be calculated on several cluster nodes. RDDs can include any kind of Python, Java, or Scala object, including classes that the user has specified.
Kafka keeps data in Topics, or in a memory buffer. Partitions are the fundamental Kafka storage unit. By configuring logs, we can specify where Kafka will store these partitions. Kafka saves partitions in segments to make it simple to locate and delete certain messages. The size of a segment is 1 GB by default. New messages created by producers will be written in a new segment whenever the previous one is filled. Kafka will never remove an active segment as it removes a whole segment (in which data is being written currently).
What is Stream Processing?
Think of streaming as an unbounded, continuous real-time flow of records, and processing these records in a similar timeframe is stream processing.
AWS (Amazon Web Services) defines “Streaming Data” as data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of Kilobytes). This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows and used for a wide variety of analytics including correlations, aggregations, filtering, and sampling.
In the stream processing method, continuous computation happens as the data flows through the system.
Stream processing is highly beneficial if the events you wish to track are happening frequently and close together in time. It is also best to utilize if the event needs to be detected right away and responded to quickly.
There is a subtle difference between stream processing, real-time processing (Rear real-time), and complex event processing (CEP). Let’s quickly look at the examples to understand the difference.
- Stream Processing: Stream processing is useful for tasks like fraud detection and cybersecurity. If transaction data is stream-processed, fraudulent transactions can be identified and stopped before they are even complete.
- Real-time Processing: If event time is very relevant and latencies in the second's range are completely unacceptable then it’s called Real-time (Rear real-time) processing. For example, flight control systems for space programs
- Complex Event Processing (CEP): CEP utilizes event-by-event processing and aggregation (for example, on potentially out-of-order events from a variety of sources, often with large numbers of rules or business logic).
We have multiple tools available to accomplish the above-mentioned Stream, Realtime or Complex Event Processing. Spark Streaming, Kafka Stream, Flink, Storm, Akka, and Structured streaming are to name a few.
We will try to understand Spark streaming and Kafka streaming in depth further in this article. Historically, these are occupying significant market share.
Apache Kafka Stream
Kafka is actually a message broker with a really good performance so that all your data can flow through it before being redistributed to applications. Kafka works as a data pipeline.
Typically, Kafka Stream supports per-second stream processing with millisecond latency.
Kafka Streams is a client library for processing and analyzing data stored in Kafka. Kafka streams can process data in 2 ways.
- Kafka -> Kafka: When Kafka Streams performs aggregations, filtering, etc., and writes back the data to Kafka, it achieves amazing scalability, high availability, high throughput, etc. if configured correctly.
It also does not do mini batching, which is “real streaming”.
- Kafka -> External Systems (‘Kafka -> Database’ or ‘Kafka -> Data science model’): Typically, any streaming library (Spark, Flink, NiFi, etc) uses Kafka as a message broker. It would read the messages from Kafka and then break them into mini-time windows to process them further.
Representative view of Kafka streaming:
Note:
- Sources here could be event logs, webpage events, etc.
- DB/Models would be accessed via any other streaming application, which in turn is using Kafka streams here.
Kafka Streams is built upon important stream processing concepts such as properly distinguishing between event time and processing time, windowing support, and simple (yet efficient) management of application state. It is based on many concepts already contained in Kafka, such as scaling by partitioning.
Also, for this reason, it comes as a lightweight library that can be integrated into an application.
The application can then be operated as desired, as mentioned below:
- Standalone, in an application server
- As a Docker container, or
- Directly, via a resource manager such as Mesos.
Why one will love using dedicated Apache Kafka Streams?
- Elastic, highly scalable, fault-tolerant
- Deploy to containers, VMs, bare metal, cloud
- Equally viable for small, medium, & large use cases
- Fully integrated with Kafka security
- Write standard Java and Scala applications
- Exactly-once processing semantics
- No separate processing cluster required
- Develop on Mac, Linux, Windows
Apache Spark Streaming
Spark Streaming receives live input data streams, it collects data for some time, builds RDD, and divides the data into micro-batches, which are then processed by the Spark engine to generate the final stream of results in micro-batches. The following data flow diagram explains the working of Spark streaming.
Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data.
DStreams can be created either from input data streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs. Think about RDD as the underlying concept for distributing data over a cluster of computers.
Why one will love using Apache Spark Streaming?
It makes it very easy for developers to use a single framework to satisfy all the processing needs. They can use MLib (Spark's machine learning library) to train models offline and directly use them online for scoring live data in Spark Streaming. In fact, some models perform continuous, online learning, and scoring.
Not all real-life use-cases need data to be processed in real real-time, a few seconds delay is tolerated over having a unified framework like Spark Streaming and volumes of data processing. It provides a range of capabilities by integrating with other spark tools to do a variety of data processing.
Looking to dive into the world of data science? Our Data Science Basics Course is the perfect starting point! Uncover the power of data analysis and gain valuable insights. Join us today and unlock your data-driven potential.
Kafka Streams Use-cases
Following are a couple of many industry Use cases where Kafka stream is being used:
- The New York Times: The New York Times uses Apache Kafka and Kafka Streams to store and distribute, in real-time, published content to the various applications and systems that make it available to the readers.
- Pinterest: Pinterest uses Apache Kafka and the Kafka Streams at a large scale to power the real-time, predictive budgeting system of their advertising infrastructure. With Kafka Streams, spending predictions are more accurate than ever.
- Zalando: As the leading online fashion retailer in Europe, Zalando uses Kafka as an ESB (Enterprise Service Bus), which helps us in transitioning from a monolithic to a microservices architecture. Using Kafka for processing event streams enables our technical team to do near-real-time business intelligence.
- Trivago: Trivago is a global hotel search platform. We are focused on reshaping the way travelers search for and compare hotels while enabling hotel advertisers to grow their businesses by providing access to a broad audience of travelers via our websites and apps. As of 2017, we offer access to approximately 1.8 million hotels and other accommodations in over 190 countries. We use Kafka, Kafka Connect, and Kafka Streams to enable our developers to access data freely in the company. Kafka Streams power parts of our analytics pipeline and deliver endless options to explore and operate on the data sources we have at hand.
Broadly, Kafka is suitable for microservices integration use cases and has wider flexibility.
Spark Streaming Use-cases
Following are a couple of the many industries' use cases where spark streaming is being used:
- Booking.com: We are using Spark Streaming for building online Machine Learning (ML) features that are used in Booking.com for real-time prediction of behavior and preferences of our users, demand for hotels, and improve processes in customer support.
- Yelp: Yelp’s ad platform handles millions of ad requests every day. To generate ad metrics and analytics in real-time, they built the ad event tracking and analyzing pipeline on top of Spark Streaming. It allows Yelp to manage a large number of active ad campaigns and greatly reduces over-delivery. It also enables them to share ad metrics with advertisers in a timelier fashion.
- Spark Streaming’s ever-growing user base consists of household names like Uber, Netflix, and Pinterest.
Broadly, spark streaming is suitable for requirements with batch processing for massive datasets, for bulk processing and has use-cases more than just data streaming.
Dean Wampler explains factors to evaluation for tool basis Use-cases beautifully, as mentioned below:
| Sr.No | Evaluation Characteristic | Response Time window | Typical Use Case Requirement |
|---|
| 1. | Latency tolerance | Pico to Microseconds (Real Real-time) | Flight control system for space programs etc. |
| Latency tolerance | < 100 Microseconds | Regular stock trading market transactions, Medical diagnostic equipment output |
| Latency tolerance | < 10 milliseconds | Credit cards verification window when consumers buy stuff online |
| Latency tolerance | < 100 milliseconds | human attention required Dashboards, Machine learning models |
| Latency tolerance | < 1 second to minutes | Machine learning model training |
| Latency tolerance | 1 minute and above | Periodic short jobs
(typical ETL applications) |
| 2. | Evaluation Characteristic
| Transaction/events frequency
| Typical Use Case Requirement
|
| Velocity | <10K-100K per second | Websites |
| Velocity | >1M per second | Nest Thermostat, Big spikes during the specific time periods. |
| 3 | Evaluation Characteristic
| Types of data processing
| NA |
Data Processing Requirement
| 1. SQL |
NA |
| 2. ETL |
| 3. Dataflow |
| 4. Training and/or Serving Machine learning models |
| Data Processing Requirement | 1. Bulk data processing | NA |
| 2. Individual Events/Transaction processing |
| 4. | Evaluation Characteristic
| Use of tool
| NA
|
| Flexibility of implementation | 1. Kafka: flexible as provides a library. | NA |
| 2. Spark: Not flexible as it’s part of a distributed framework |
Conclusion
Kafka Streams is still best used in a ‘Kafka -> Kafka’ context, while Spark Streaming could be used for a ‘Kafka -> Database’ or ‘Kafka -> Data science model’ type of context.
Although, when these 2 technologies are connected, they bring complete data collection and processing capabilities together and are widely used in commercialized use cases and occupy significant market share.
