Search

How Big is ‘Big Data’, Anyway?

When I got introduced to the data-world with my first corporate induction training, about 10 years ago. I was then still processing the difference between Data and Information. The following helped me understand the same:Data: It is raw information (unprocessed facts and figures) without any context for e.g. Number 20Information: structured Data grouped together which can have interpretation. E.g $20 for a toy.Knowledge: combination of information, experience and insight that may benefit the individual for the organisation. E.g. $20 for a toy in Black Friday Sale in a mall.Wisdom: Knowledge becomes wisdom when one can assimilate and apply this knowledge to make the right decisions. E.g. One who wants to buy a toy will wait for the Black Friday Sale to get it at a cheaper price.By the time I started understanding above differences, ‘Big data’ term was already making it big and then the obvious question in mind was,” When to call ‘data’ à ‘ Big data’? “I then made an attempt to understand ‘how big is a data to be called big data?’ and here, I have a big revelation to make, for all of you reading this article, that ‘Big Data’ is actually misleading term and it is irrelevant with “Bigness of data” but it is to be used in relevance. In fact, it is a term which needs to be understood, only in perspective.The simplest one I could find relevant is, Big data is the data that cannot be stored with traditional storages, cannot be processed with traditional methods/ways and within a short period of time (and these references would still be valid as time advances.) but this is not textbook or only definition of big data. Interestingly, One who finds one set of data as big data can be traditional data for others so truly it cannot be bounded in words but loosely can be described through numerous examples. I am sure by the end of the article you will be able to answer the question for yourself. Let’s start.Do you know? - NASA researchers Michael Cox and David Ellsworth use the term “big data” for the first time to describe a familiar challenge in the 1990s supercomputers generating massive amounts of information - in Cox and Ellsworth’s case, simulations of airflow around aircraft - that cannot be processed and visualized.If you go through a brief history of big data, you would know data which is not fitting into memory or disk was called ‘Big data problem’ back in 1997.As the years passed by innovations were on rising and disruptions were made so the data universe is growing all the time. Let’s understand a few widely available and stated statistics for ‘big data’ (Collected around 2017 or before) >>On average, people send about 500 million tweets per day.Snapchat users share 527,760 photos in a minute Instagram users post 46,740 photos in a minute More than 120 professionals join LinkedIn in a minute Users watch 4,146,600 YouTube videos in a minuteThe average U.S. customer uses 1.8 gigabytes of data per month on his or her cell phone plan.Amazon sells 600 items per second.On average, each person who uses email receives 88 emails per day and send 34. That adds up to more than 200 billion emails each day.MasterCard processes 74 billion transactions per year.Commercial airlines make about 5,800 flights per day.You might be interested to read through Domo’s Data Never Sleeps 5.0 report, for the numbers generated every minute of the day.Understanding that the above stats are probably about 1.5-2 years older and data is ever-growing, it helps to establish the fact that ‘big data‘ is a moving target and…. In short,“Today’s big data is tomorrow’s small data.”Now that we have some knowledge about transactions/tweets/snaps in a day, Let’s also understand how much data, all these “One-minute Quickies” are generating. Let’s talk about some volumes too. Afterall volumes are one of the characteristics of big data but mind you, not only characteristic of big data. It is believed that, In a single day, the world produces 2.5 quintillion bytes (2.3 trillion gigabytes) of data, in layman's terms, this is the equivalent of everyone in the world downloading 60 episodes of Breaking Bad, in HD, 20 times! [Source: VCloud 2012] and According to estimates, the volume of data worldwide doubles every 1.2 years.IDC predicts that the collective sum of the world's data will grow from 33 zettabytes this year to a 175ZB by 2025, for a compounded annual growth rate of 61 per cent. The 175ZB figure represents a 9 per cent increase over last year's prediction of data growth by 2025 – As per the report published in Dec’2018.But, do you know: how much would be 1 zettabyte of data? Let’s understand. One zettabyte is equal to one sextillion bytes or 1021 (1,000,000,000,000,000,000,000) bytes or, one zettabyte is roughly equal to a trillion gigabytes.Fun Fact: There is a legit term coined as The Zettabyte Era (Today’s Era).The Zettabyte Era can also be understood as an age of growth of all forms of digital data that exist in the world which includes the public Internet, but also all other forms of digital data such as stored data from security cameras or voice data from cell-phone calls.You must check out this infographic by economywatch (taken from SearchEngineJournal) to understand how much data zettabyte consists of, putting it into context with current data storage capabilities and usage.Today’s ‘big data’ is generated from majority 3 sources i.e.People Generated: Social media uploads, Mails etc. Machine Generated: M2M (machine to machine) interactions, IOT devices etc. Business Generated: Data generated and stored into today’s OLTPs, OLAPs, Data warehouses, data marts, reports, operational data throughout the enterprise/organization.Various analytics tools available in the market today, help in solving big data challenges by providing ways for storing this data, process this data and make valuable insights from this data.As we discussed, big data is moving target as time advances, it is also interesting to know even today, data which is not of huge size but is difficult to process and of relatively smaller volume would still be categorized as Big Data. For example, unstructured data in emails, from social media platforms, data which is required to process with real-time/near real-time etc. all the examples we have seen so far, all of it is big data.   But, It would be a mistake to assume that, Big Data only as data that is analyzed using Hadoop, Spark or another complex analytics platform. As big data is moving the target and it’s ever-growing, also with various disruptive sources of data are being introduced every day, to process this data newer tools would be invented, and hence big data cannot just remain a function of tools being used to analyze it. To conclude, as discussed at the starting of the article, it would still be appropriate and reasonable to say, this moving target of big data which would always be challenged for storage, processing methods and process it within a short period as well. So big data is a function of volume and/or time and/or storage and/or variety. It was fun and exciting to know what different aspects are hidden in ‘BIG DATA’ word and I thoroughly enjoyed solving this mystery.Did you enjoy solving it too?Do let us know how was experience through comments below.Happy Learning!!!
Rated 4.5/5 based on 23 customer reviews

How Big is ‘Big Data’, Anyway?

14K
How Big is ‘Big Data’, Anyway?

When I got introduced to the data-world with my first corporate induction training, about 10 years ago. I was then still processing the difference between Data and Information. The following helped me understand the same:

  • Data: It is raw information (unprocessed facts and figures) without any context for e.g. Number 20
  • Information: structured Data grouped together which can have interpretation. E.g $20 for a toy.
  • Knowledge: combination of information, experience and insight that may benefit the individual for the organisation. E.g. $20 for a toy in Black Friday Sale in a mall.
  • Wisdom: Knowledge becomes wisdom when one can assimilate and apply this knowledge to make the right decisions. E.g. One who wants to buy a toy will wait for the Black Friday Sale to get it at a cheaper price.

By the time I started understanding above differences, ‘Big data’ term was already making it big and then the obvious question in mind was,” When to call ‘dataà ‘ Big data’? “

I then made an attempt to understand ‘how big is a data to be called big data?’ and here, I have a big revelation to make, for all of you reading this article, that ‘Big Data’ is actually misleading term and it is irrelevant with “Bigness of data” but it is to be used in relevance. In fact, it is a term which needs to be understood, only in perspective.

The simplest one I could find relevant is, Big data is the data that cannot be stored with traditional storages, cannot be processed with traditional methods/ways and within a short period of time (and these references would still be valid as time advances.) but this is not textbook or only definition of big data. Interestingly, One who finds one set of data as big data can be traditional data for others so truly it cannot be bounded in words but loosely can be described through numerous examples. I am sure by the end of the article you will be able to answer the question for yourself. Let’s start.

Do you know? - NASA researchers Michael Cox and David Ellsworth use the term “big data” for the first time to describe a familiar challenge in the 1990s supercomputers generating massive amounts of information - in Cox and Ellsworth’s case, simulations of airflow around aircraft - that cannot be processed and visualized.

If you go through a brief history of big data, you would know data which is not fitting into memory or disk was called ‘Big data problem’ back in 1997.

As the years passed by innovations were on rising and disruptions were made so the data universe is growing all the time. Let’s understand a few widely available and stated statistics for ‘big data’ (Collected around 2017 or before) >>

You might be interested to read through Domo’s Data Never Sleeps 5.0 report, for the numbers generated every minute of the day.

Understanding that the above stats are probably about 1.5-2 years older and data is ever-growing, it helps to establish the fact that ‘big data‘ is a moving target and…. 

In short,

“Today’s big data is tomorrow’s small data.”

Now that we have some knowledge about transactions/tweets/snaps in a day, Let’s also understand how much data, all these “One-minute Quickies” are generating. Let’s talk about some volumes too. Afterall volumes are one of the characteristics of big data but mind you, not only characteristic of big data. 

It is believed that, In a single day, the world produces 2.5 quintillion bytes (2.3 trillion gigabytes) of data, in layman's terms, this is the equivalent of everyone in the world downloading 60 episodes of Breaking Bad, in HD, 20 times! [Source: VCloud 2012] and According to estimates, the volume of data worldwide doubles every 1.2 years.

IDC predicts that the collective sum of the world's data will grow from 33 zettabytes this year to a 175ZB by 2025, for a compounded annual growth rate of 61 per cent. The 175ZB figure represents a 9 per cent increase over last year's prediction of data growth by 2025 – As per the report published in Dec’2018.

But, do you know: how much would be 1 zettabyte of data? Let’s understand. 

One zettabyte is equal to one sextillion bytes or 1021 (1,000,000,000,000,000,000,000) bytes or, one zettabyte is roughly equal to a trillion gigabytes.

Fun Fact: There is a legit term coined as The Zettabyte Era (Today’s Era).
The Zettabyte Era can also be understood as an age of growth of all forms of digital data that exist in the world which includes the public Internet, but also all other forms of digital data such as stored data from security cameras or voice data from cell-phone calls.

You must check out this infographic by economywatch (taken from SearchEngineJournal) to understand how much data zettabyte consists of, putting it into context with current data storage capabilities and usage.

Today’s ‘big data’ is generated from majority 3 sources i.e.

  1. People Generated: Social media uploads, Mails etc. 
  2. Machine Generated: M2M (machine to machine) interactions, IOT devices etc. 
  3. Business Generated: Data generated and stored into today’s OLTPs, OLAPs, Data warehouses, data marts, reports, operational data throughout the enterprise/organization.

Various analytics tools available in the market today, help in solving big data challenges by providing ways for storing this data, process this data and make valuable insights from this data.

As we discussed, big data is moving target as time advances, it is also interesting to know even today, data which is not of huge size but is difficult to process and of relatively smaller volume would still be categorized as Big Data. For example, unstructured data in emails, from social media platforms, data which is required to process with real-time/near real-time etc. all the examples we have seen so far, all of it is big data.   

But, It would be a mistake to assume that, Big Data only as data that is analyzed using Hadoop, Spark or another complex analytics platform. As big data is moving the target and it’s ever-growing, also with various disruptive sources of data are being introduced every day, to process this data newer tools would be invented, and hence big data cannot just remain a function of tools being used to analyze it. 

To conclude, as discussed at the starting of the article, it would still be appropriate and reasonable to say, this moving target of big data which would always be challenged for storage, processing methods and process it within a short period as well. So big data is a function of volume and/or time and/or storage and/or variety. 

It was fun and exciting to know what different aspects are hidden in ‘BIG DATA’ word and I thoroughly enjoyed solving this mystery.
Did you enjoy solving it too?
Do let us know how was experience through comments below.

Happy Learning!!!

Shruti

Shruti Deshpande

Blog Author

10+ years of data-rich experience in the IT industry. It started with data warehousing technologies into data modelling to BI application Architect and solution architect.


Big Data enthusiast and data analytics is my personal interest. I do believe it has endless opportunities and potential to make the world a sustainable place. Happy to ride on this tide.


*Disclaimer* - Expressed views are the personal views of the author and are not to be mistaken for the employer or any other organization’s views.

Join the Discussion

Your email address will not be published. Required fields are marked *

Suggested Blogs

5 Big Data Challenges in 2020

The year 2019 saw some enthralling changes in volume and variety of data across businesses, worldwide. The surge in data generation is only going to continue. Foresighted enterprises are the ones who will be able to leverage this data for maximum profitability through data processing and handling techniques. With the rise in opportunities related to Big Data, challenges are also bound to increase.Below are the 5 major Big Data challenges that enterprises face in 2020:1. The Need for More Trained ProfessionalsResearch shows that since 2018, 2.5 quintillion bytes (or 2.5 exabytes) of information is being generated every day. The previous two years have seen significantly more noteworthy increments in the quantity of streams, posts, searches and writings, which have cumulatively produced an enormous amount of data. Additionally, this number is only growing by the day. A study has predicted that by 2025, each person will be making a bewildering 463 exabytes of information every day.A report by Indeed, showed a 29 percent surge in the demand for data scientists yearly and a 344 percent increase since 2013 till date. However, the searches by job seekers skilled in data science continue to grow at a snail’s pace at 14 percent. In August 2018, LinkedIn reported claimed that US alone needs 151,717 professionals with data science skills. This along with a 15 percent discrepancy between job postings and job searches on Indeed, makes it quite evident that the demand for data scientists outstrips supply. The greatest data processing challenge of 2020 is the lack of qualified data scientists with the skill set and expertise to handle this gigantic volume of data.2. Inability to process large volumes of dataOut of the 2.5 quintillion data produced, only 60 percent workers spend days on it to make sense of it. A major portion of raw data is usually irrelevant. And about 43 percent companies still struggle or aren’t fully satisfied with the filtered data. 3. Syncing Across Data SourcesOnce you import data into Big Data platforms you may also realize that data copies migrated from a wide range of sources on different rates and schedules can rapidly get out of the synchronization with the originating system. This implies two things, one, the data coming from one source is out of date when compared to another source. Two, it creates a commonality of data definitions, concepts, metadata and the like. The traditional data management and data warehouses, and the sequence of data transformation, extraction and migration- all arise a situation in which there are risks for data to become unsynchronized.4. Lack of adequate data governanceData collected from multiple sources should have some correlation to each other so that it can be considered usable by enterprises. In a recent Big Data Maturity Survey, the lack of stringent data governance was recognized the fastest-growing area of concern. Organizations often have to setup the right personnel, policies and technology to ensure that data governance is achieved. This itself could be a challenge for a lot of enterprises.5. Threat of compromised data securityWhile Big Data opens plenty of opportunities for organizations to grow their businesses, there’s an inherent risk of data security. Some of the biggest cyber threats to big players like Panera Bread, Facebook, Equifax and Marriot have brought to light the fact that literally no one is immune to cyberattacks. As far as Big Data is concerned, data security should be high on their priorities as most modern businesses are vulnerable to fake data generation, especially if cybercriminals have access to the database of a business. However, regulating access is one of the primary challenges for companies who frequently work with large sets of data. Even the way Big Data is designed makes it harder for enterprises to ensure data security. Working with data distributed across multiple systems makes it both cumbersome and risky.Overcoming Big Data challenges in 2020Whether it’s ensuring data governance and security or hiring skilled professionals, enterprises should leave no stone unturned when it comes to overcoming the above Big Data challenges. Several courses and online certifications are available to specialize in tackling each of these challenges in Big Data. Training existing personnel with the analytical tools of Big Data will help businesses unearth insightful data about customer. Frameworks related to Big Data can help in qualitative analysis of the raw information.
Rated 4.5/5 based on 0 customer reviews
1292
5 Big Data Challenges in 2020

The year 2019 saw some enthralling changes in volu... Read More

Top In-demand Jobs During Coronavirus Pandemic

With the global positive cases for the COVID-19 reaching over two crores globally, and over 281,000 jobs lost in the US alone, the impact of the coronavirus pandemic already has been catastrophic for workers worldwide. While tourism and the supply chain industries are the hardest hit, the healthcare and transportation sectors have faced less severe heat. According to a Goldman Sachs report, the number of unemployed individuals in the US can climb up to 2.25 million. However, despite these alarming figures, the NBC News states that this is merely 20% of the total unemployment rate of the US. Job portals like LinkedIn, Shine, and Monster are also witnessing continued hiring for specific roles. So, what are these roles defining the pandemic job sector? Top In-demand Jobs During Coronavirus Pandemic Healthcare specialist For obvious reasons, the demand for healthcare specialists has spiked up globally. This includes doctors, nurses, surgical technologists, virologists, diagnostic technicians, pharmacists, and medical equipment providers. Logistics personnel This largely involves shipping and delivery companies that include a broad profile of employees, right from warehouse managers, transportation-oriented job roles, and packaging and fulfillment jobs. Presently, Amazon is hiring over 1,00,000 workers for its operations while making amends in the salaries and timings to accommodate the situation.  Online learning companies Teaching and learning are at the forefront of the current global scenario. With most of the individuals either working from home or anticipating a loss of a job, several of them are resorting to upskilling or attaining new skills to embrace broader job roles. The demand for teachers or trainers for these courses and academic counselors has also shot up. Remote learning facilities and online upskilling have made these courses much more accessible to individuals as well.  Remote meeting and communication companies The entirety of remote working is heavily dependant on communication and meeting tools such as Zoom, Slack, and Microsoft teams. The efficiency of these tools and the effectivity of managing projects with remote communication has enabled several industries to sustain global pandemic. Even project management is taking an all-new shape thanks to these modern tools. Moreover, several schools are also relying on these tools to continue education through online classes.  Psychologists/Mental health-related businesses Many companies and individuals are seeking help to cope up with the undercurrent. This has created a surge in the demand for psychologists. Businesses like PwC and Starbucks have introduced/enhanced their mental health coaching. Mental health and wellness apps like Headspace have seen a 400% increase in the demand from top companies like Adobe and GE.  Data analysts Hiring companies like Shine have seen a surge in the hiring of data analysts. The simple reason being that there is a constant demand for information about the coronavirus, its status, its impact on the global economy, different markets, and many other industries. Companies are also hiring data analysts rapidly to study current customer behavior and reach out to public sentiments.  How to find a job during the coronavirus pandemicWhether you are looking for a job change, have already faced the heat of the coronavirus, or are at the risk of losing your job, here are some ways to stay afloat despite the trying times.  Be proactive on job portals, especially professional networking sites like LinkedIn to expand your network Practise phone and video job interviews Expand your work portfolio by on-boarding more freelance projects Pick up new skills by leveraging on the online courses available  Stay focused on your current job even in uncertain times Job security is of paramount importance during a global crisis like this. Andrew Seaman, an editor at LinkedIn notes that recruiters are going by the ‘business as usual approach’, despite concerns about COVID-19. The only change, he remarks, is that the interviews may be conducted over a video call, rather than in person. If the outbreak is not contained soon enough though, hiring may eventually take a hit. 
Rated 4.5/5 based on 0 customer reviews
8507
Top In-demand Jobs During Coronavirus Pandemic

With the global positive cases for the COVID-19 re... Read More

Apache Kafka Vs Apache Spark: Know the Differences

A new breed of ‘Fast Data’ architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. - Dean Wampler (Renowned author of many big data technology-related books)Dean Wampler makes an important point in one of his webinars. The demand for stream processing is increasing every day in today’s era. The main reason behind it is, processing only volumes of data is not sufficient but processing data at faster rates and making insights out of it in real time is very essential so that organization can react to changing business conditions in real time.And hence, there is a need to understand the concept “stream processing “and technology behind it. So, what is Stream Processing?Think of streaming as an unbounded, continuous real-time flow of records and processing these records in similar timeframe is stream processing.AWS (Amazon Web Services) defines “Streaming Data” is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of Kilobytes). This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows and used for a wide variety of analytics including correlations, aggregations, filtering, and sampling.In stream processing method, continuous computation happens as the data flows through the system.Stream processing is highly beneficial if the events you wish to track are happening frequently and close together in time. It is also best to utilize if the event needs to be detected right away and responded to quickly.There is a subtle difference between stream processing, real-time processing (Rear real-time) and complex event processing (CEP). Let’s quickly look at the examples to understand the difference. Stream Processing: Stream processing is useful for tasks like fraud detection and cybersecurity. If transaction data is stream-processed, fraudulent transactions can be identified and stopped before they are even complete.Real-time Processing: If event time is very relevant and latencies in the second's range are completely unacceptable then it’s called Real-time (Rear real-time) processing. For ex. flight control system for space programsComplex Event Processing (CEP): CEP utilizes event-by-event processing and aggregation (for example, on potentially out-of-order events from a variety of sources, often with large numbers of rules or business logic).We have multiple tools available to accomplish above-mentioned Stream, Realtime or Complex event Processing. Spark Streaming, Kafka Stream, Flink, Storm, Akka, Structured streaming are to name a few. We will try to understand Spark streaming and Kafka stream in depth further in this article. As historically, these are occupying significant market share. Apache Kafka Stream: Kafka is actually a message broker with a really good performance so that all your data can flow through it before being redistributed to applications. Kafka works as a data pipeline.Typically, Kafka Stream supports per-second stream processing with millisecond latency.  Kafka Streams is a client library for processing and analyzing data stored in Kafka. Kafka streams can process data in 2 ways. Kafka -> Kafka: When Kafka Streams performs aggregations, filtering etc. and writes back the data to Kafka, it achieves amazing scalability, high availability, high throughput etc.  if configured correctly. It also does not do mini batching, which is “real streaming”.Kafka -> External Systems (‘Kafka -> Database’ or ‘Kafka -> Data science model’): Typically, any streaming library (Spark, Flink, NiFi etc) uses Kafka for a message broker. It would read the messages from Kafka and then break it into mini time windows to process it further. Representative view of Kafka streaming: Note:Sources here could be event logs, webpage events etc. etc. DB/Models would be accessed via any other streaming application, which in turn is using Kafka streams here. Kafka Streams is built upon important stream processing concepts such as properly distinguishing between event time and processing time, windowing support, and simple (yet efficient) management of application state. It is based on many concepts already contained in Kafka, such as scaling by partitioning.Also, for this reason, it comes as a lightweight library that can be integrated into an application.The application can then be operated as desired, as mentioned below: Standalone, in an application serverAs a Docker container, or Directly, via a resource manager such as Mesos.Why one will love using dedicated Apache Kafka Streams?Elastic, highly scalable, fault-tolerantDeploy to containers, VMs, bare metal, cloudEqually viable for small, medium, & large use casesFully integrated with Kafka securityWrite standard Java and Scala applicationsExactly-once processing semanticsNo separate processing cluster requiredDevelop on Mac, Linux, WindowsApache Spark Streaming:Spark Streaming receives live input data streams, it collects data for some time, builds RDD, divides the data into micro-batches, which are then processed by the Spark engine to generate the final stream of results in micro-batches. Following data flow diagram explains the working of Spark streaming. Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data. DStreams can be created either from input data streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs. Think about RDD as the underlying concept for distributing data over a cluster of computers. Why one will love using Apache Spark Streaming?It makes it very easy for developers to use a single framework to satisfy all the processing needs. They can use MLib (Spark's machine learning library) to train models offline and directly use them online for scoring live data in Spark Streaming. In fact, some models perform continuous, online learning, and scoring.Not all real-life use-cases need data to be processed at real real-time, few seconds delay is tolerated over having a unified framework like Spark Streaming and volumes of data processing. It provides a range of capabilities by integrating with other spark tools to do a variety of data processing.  Spark Streaming Vs Kafka StreamNow that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Following table briefly explain you, key differences between the two. Sr.NoSpark streamingKafka Streams1Data received form live input data streams is Divided into Micro-batched for processing.processes per data stream(real real-time)2Separated processing Cluster is requriedNo separated processing cluster is requried.3Needs re-configuration for Scaling Scales easily by just adding java processes, No reconfiguration requried.4At least one semanticsExactly one semantics5Spark streaming is better at processing group of rows(groups,by,ml,window functions etc.)Kafka streams provides true a-record-at-a-time processing capabilities. it's better for functions like rows parsing, data cleansing etc.6Spark streaming is standalone framework.Kafka stream can be used as part of microservice,as it's just a library.Kafka streams Use-cases:Following are a couple of many industry Use cases where Kafka stream is being used: The New York Times: The New York Times uses Apache Kafka and Kafka Streams to store and distribute, in real-time, published content to the various applications and systems that make it available to the readers.Pinterest: Pinterest uses Apache Kafka and the Kafka Streams at large scale to power the real-time, predictive budgeting system of their advertising infrastructure. With Kafka Streams, spend predictions are more accurate than ever.Zalando: As the leading online fashion retailer in Europe, Zalando uses Kafka as an ESB (Enterprise Service Bus), which helps us in transitioning from a monolithic to a micro services architecture. Using Kafka for processing event streams enables our technical team to do near-real time business intelligence.Trivago: Trivago is a global hotel search platform. We are focused on reshaping the way travellers search for and compare hotels while enabling hotel advertisers to grow their businesses by providing access to a broad audience of travellers via our websites and apps. As of 2017, we offer access to approximately 1.8 million hotels and other accommodations in over 190 countries. We use Kafka, Kafka Connect, and Kafka Streams to enable our developers to access data freely in the company. Kafka Streams powers parts of our analytics pipeline and delivers endless options to explore and operate on the data sources we have at hand.Broadly, Kafka is suitable for microservices integration use cases and have wider flexibility.Spark Streaming Use-cases:Following are a couple of the many industries use-cases where spark streaming is being used: Booking.com: We are using Spark Streaming for building online Machine Learning (ML) features that are used in Booking.com for real-time prediction of behaviour and preferences of our users, demand for hotels and improve processes in customer support. Yelp: Yelp’s ad platform handles millions of ad requests every day. To generate ad metrics and analytics in real-time, they built the ad event tracking and analyzing pipeline on top of Spark Streaming. It allows Yelp to manage a large number of active ad campaigns and greatly reduce over-delivery. It also enables them to share ad metrics with advertisers in a timelier fashion.Spark Streaming’s ever-growing user base consists of household names like Uber, Netflix, and Pinterest.Broadly, spark streaming is suitable for requirements with batch processing for massive datasets, for bulk processing and have use-cases more than just data streaming. Dean Wampler explains factors to evaluation for tool basis Use-cases beautifully, as mentioned below: Sr.NoEvaluation CharacteristicResponse Time windowTypical Use Case Requirement1.Latency tolerancePico to Microseconds (Real Real time)Flight control system for space programs etc.Latency tolerance< 100 MicrosecondsRegular stock trading market transactions, Medical diagnostic equipment outputLatency tolerance< 10 millisecondsCredit cards verification window when consumer buy stuff onlineLatency tolerance< 100 millisecondshuman attention required Dashboards, Machine learning modelsLatency tolerance< 1 second to minutesMachine learning model trainingLatency tolerance1 minute and abovePeriodic short jobs(typical ETL applications)2.Evaluation CharacteristicTransaction/events frequencyTypical Use Case RequirementVelocity1M per secondNest Thermostat, Big spikes during specific time period.3Evaluation CharacteristicTypes of data processingNAData Processing Requirement1. SQLNA2. ETL3. Dataflow4. Training and/or Serving Machine learning modelsData Processing Requirement1. Bulk data processingNA2. Individual Events/Transaction processing4.Evaluation CharacteristicUse of toolNAFlexibility of implementation1. Kafka : flexible as provides library.NA2. Spark: Not flexible as it’s part of a distributed frameworkConclusionKafka Streams is still best used in a ‘Kafka -> Kafka’ context, while Spark Streaming could be used for a ‘Kafka -> Database’ or ‘Kafka -> Data science model’ type of context.Although, when these 2 technologies are connected, they bring complete data collection and processing capabilities together and are widely used in commercialized use cases and occupy significant market share. 
Rated 4.5/5 based on 1 customer reviews
10378
Apache Kafka Vs Apache Spark: Know the Differences

A new breed of ‘Fast Data’ architectures has e... Read More