Search

What is Big Data — An Introductory Guide

The massive world of Big DataIf one strolls around any IT office premises, over every decade (nowadays time span is even lesser, almost every 3-4 years) one would overhear professionals discussing new jargons from the hottest trends in technology. Around 5 -6 years ago, one such word has started ruling IT services is ‘BIG data’ and still has been interpreted by a layman to tech geeks in various ways.Although services industries started talking about big data solutions widely from 5-6 years, it is believed that the term was in use since the 1990s by John Mashey from Silicon Graphics, whereas credit for coining the term ‘big data’ aligning to its modern definition goes to Roger Mougalas from O’Reilly Media in 2005.Let’s first understand why everyone going gaga about ‘BIG data’ and what are the real-world problems it is supposed to solve and then we will try to answer what and how aspects of it.Why is Big Data essential for today’s digital world?Pre smart-phones era, internet and web world were around for many years, but smart-phones made it mobile with on-the-go usage. Social Media, mobile apps started generating tons of data. At the same time, smart-bands, wearable devices ( IoT, M2M ), have given newer dimensions for data generation. This newly generated data became a new oil to the world. If this data is stored and analyzed, it has the potential to give tremendous insights which could be put to use in numerous ways.You will be amazed to see the real-world use cases of BIG data. Every industry has a unique use case and is even unique to every client who is implementing the solutions. Ranging from data-driven personalized campaigning (you do see that item you have browsed on some ‘xyz’ site onto Facebook scrolling, ever wondered how?) to predictive maintenance of huge pipes across countries carrying oils, where manual monitoring is practically impossible. To relate this to our day to day life, every click, every swipe, every share and every like we casually do on social media is helping today’s industries to take future calculated business decisions. How do you think Netflix predicted the success of ‘House of Cards’ and spent $100 million on the same? Big data analytics is the simple answer.Talking about all this, the biggest challenge in the past was traditional methods used to store, curate and analyze data, which had limitations to process this data generated from newer sources and which were huge in volumes generated from heterogeneous sources and was being generated  really fast(To give you an idea, roughly 2.5 quintillion data is generated per day as on today – Refer infographic released by Domo called “Data Never Sleeps 5.0.” ), Which given rise to term BIG data and related solutions.Understanding Big Data: Experts’ viewpoint BIG data literally means Massive data (loosely > 1TB) but that’s not the only aspect of it. Distributed data or even complex datasets which could not be analyzed through traditional methods can be categorized into ‘Big data’ and hence Big data theoretical definition makes a lot of sense with this background:“Gartner (2012) defines, Big data is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation.”Generic data possessing characteristics of big data are 3Vs namely Variety, Velocity, and VolumeBut due to the changing nature of data in today’s world and to gain most insights of it, 3 more Vs are added to the definition of BIG DATA, namely Variability, Veracity and Value.The diagram below illustrates each V in detail:Diagram: 6 V’s of Big DataThis 6Vs help understanding the characteristics of “BIG Data” but let’s also understand types of data in BIG Data processing.  “Variety” of above characteristics caters to different types of data can be processed through big data tools and technologies. Let’s drill down a bit for understanding what those are:Structured ex. Mainframes, traditional databases like Teradata, Netezza, Oracle, etc.Unstructured ex. Tweets, Facebook posts, emails, etc.Semi/Multi structured or Hybrid ex. E-commerce, demographic, weather data, etc.As the technology is advancing, the variety of data is available and its storage, processing, and analysis are made possible by big data. Traditional data processing techniques were able to process only structured data.Now, that we understand what big data and limitations of old traditional techniques are of handling such data, we could safely say, we need new technology to handle this data and gain insights out of it. Before going further, do you know, what were the traditional data management techniques?Traditional Techniques of Data Processing are:RDBMS (Relational Database Management System)Data warehousing and DataMartOn a high level, RDBMS catered to OLTP needs and data warehousing/DataMart facilitated OLAP needs. But both the systems work with structured data.I hope. now one can answer, ‘what is big data?’ conceptually and theoretically both.So, it’s time that we understand how it is being done in actual implementations.only storing of “big data” will not help the organizations, what’s important is to turn data into insights and business value and to do so, following are the key infrastructure elements:Data collectionData storageData analysis andData visualization/outputAll major big data processing framework offerings are based on these building blocks.And in an alignment of the above building blocks, following are the top 5 big data processing frameworks that are currently being used in the market:1. Apache Hadoop : Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.First up is the all-time classic, and one of the top frameworks in use today. So prevalent is it, that it has almost become synonymous with Big Data.2 Apache Spark : unified analytics engine for large-scale data processing.Apache Spark and Hadoop are often contrasted as an "either/or" choice,  but that isn't really the case.Above two frameworks are popular but apart from that following 3 are available and are comparable frameworks:3. Apache Storm : free and open source distributed real-time computation system. You can also take up Apache Storm training to learn more about Apache Storm.4. Apache Flink : streaming dataflow engine, aiming to provide facilities for distributed computation over streams of data. Treating batch processes as a special case of streaming data, Flink is effectively both batch and real-time processing framework, but one which clearly puts streaming first.5. Apache Samza : distributed Stream processing framework.Frameworks help processing data through building blocks and generate required insights. The framework is supported by the whopping number of tools providing the required functionality.Big Data processing frameworks and technology landscapeBig data tools and technology landscape can be better understood with layered big data architecture. Give a good read to a great article by Navdeep singh Gill on XENONSTACK for understanding the layered architecture of big data.By taking inspiration from layered architecture, different available tools in the market are mapped to layers to understand big data technology landscape in depth. Note that, layered architecture fits very well with infrastructure elements/building blocks discussed in the above section.Few of the tools are briefed below for further understanding:  1. Data Collection / Ingestion Layer Cassandra: is a free and open-source, distributed, wide column store, NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failureKafka: is used for building real-time data pipelines and streaming apps. Event streaming platformFlume: log collector in HadoopHBase: columnar database in Hadoop2. Processing Layer Pig: scripting language in the Hadoop frameworkMapReduce: processing language in Hadoop3. Data Query Layer Impala: Cloudera Impala:  modern, open source, distributed SQL query engine for Apache Hadoop. (often compared with hive)Hive: Data Warehouse software for data Query and analysisPresto: Presto is a high performance, distributed SQL query engine for big data. Its architecture allows users to query a variety of data sources such as Hadoop, AWS S3, Alluxio, MySQL, Cassandra, Apache Kafka, and MongoDB4. Analytical EngineTensorFlow: n source machine learning library for research and production.5. Data storage LayerIgnite: open-source distributed database, caching and processing platform designed to store and compute on large volumes of data across a cluster of nodesPhoenix: hortonworks: Apache Phoenix is an open source, massively parallel, relational database engine supporting OLTP for Hadoop using Apache HBase as its backing storePolyBase: s a new feature in SQL Server 2016. It is used to query relational and non-relational databases (NoSQL). You can use PolyBase to query tables and files in Hadoop or in Azure Blob Storage. You can also import or export data to/from Hadoop.Sqoop: ETL toolBig data in EXCEL: Few people like to process big datasets with current excel capabilities and it's known as Big Data in Excel6. Data Visualization LayerMicrosoft HDInsight: Azure HDInsight is a Hadoop service offering hosted in Azure that enables clusters of managed Hadoop instances. Azure HDInsight deploys and provisions Apache Hadoop clusters in the cloud, providing a software framework designed to manage, analyze, and report on big data with high reliability and availability. Hadoop administration training will give you all the technical understanding required to manage a Hadoop cluster, either in a development or a production environment.Best Practices in Big Data  Every organization, industry, business, may it be small or big wants to get benefit out of “big data” but it's essential to understand that it can prove of maximum potential only if organization adhere to best practices before adapting big data:Answering 5 basic questions help clients know the need for adapting Big Data for organizationTry to answer why Big Data is required for the organization. What problem would it help solve?Ask the right questions.Foster collaboration between business and technology teams.Analyze only what is required to use.Start small and grow incrementally.Big Data industry use-cases We talked about all the things in the Big Data world except real use cases of big data. In the starting, we did discuss few but let me give you insights into the real world and interesting big data use cases and for a few, it’s no longer a secret ☺. In fact, it’s penetrating to the extent you name the industry and plenty of use cases can be told. Let’s begin.1.Streaming PlatformsAs I had given an example of ‘House of Cards’ at the start of the article, it’s not a secret that Netflix uses Big Data analytics. Netflix spent $100mn on 26 episodes of ‘House of Cards’ as they knew the show would appeal to viewers of original British House of Cards and built in director David Fincher and actor Kevin Spacey. Netflix typically collects behavioral data and it then uses this data to create a better experience for the user.But Netflix uses Big Data for more than that, they monitor and analyze traffic details for various devices, spot problem areas and adjust network infrastructure to prepare for future demand. (later is action out of Big Data analytics, how big data analysis is put to use). They also try to get insights into types of content viewers to prefer and help them make informed decisions.   Apart from Netflix, Spotify is also a known great use case.2. Advertising and Media / Campaigning /EntertainmentFor decades marketers were forced to launch campaigns while blindly relying on gut instinct and hoping for the best. That all changed with digitization and big data world. Nowadays, data-driven campaigns and marketing is on the rise and to be successful in this landscape, a modern marketing campaign must integrate a range of intelligent approaches to identify customers, segment, measure results, analyze data and build upon feedback in real time. All needs to be done in real time, along with the customer’s profile and history, based on his purchasing patterns and other relevant information and Big Data solutions are the perfect fit.Event-driven marketing is also could be achieved through big data, which is another way of successful marketing in today’s world. That basically indicates, keeping track of events customer are directly and indirectly involved with and campaign exactly when a customer would need it rather than random campaigns. For. Ex if you have searched for a product on Amazon/Flipkart, you would see related advertisements on other social media apps you casually browse through. Bang on, you would end up purchasing it as you anyway needed options best to choose from.3. Healthcare IndustryHealthcare is one of the classic use case industries for Big Data applications. The industry generates a huge amount of data.Patients medical history, past records, treatments given, available and latest medicines, Medicinal latest available research the list of raw data is endless.All this data can help give insights and Big Data can contribute to the industry in the following ways:Diagnosis time could be reduced, and exact requirement treatment could be started immediately. Most of the illnesses could be treated if a diagnosis is perfect and treatment can be started in time. This can be achieved through evidence-based past medical data available for similar treatments to doctor treating the illness, patients’ available history and feeding symptoms real-time into the system.  Government Health department can monitor if a bunch of people from geography reporting of similar symptoms, predictive measures could be taken in nearby locations to avoid outbreak as a cause for such illness could be the same.   The list is long, above were few representative examples.4. SecurityDue to social media outbreak, today, personal information is at stake. Almost everything is digital, and majority personal information is available in the public domain and hence privacy and security are major concerns with the rise in social media. Following are few such applications for big data.Cyber Crimes are common nowadays and big data can help to detect, predicting crimes.Threat analysis and detection could be done with big data.  5. Travel and TourismFlight booking sites, IRCTC track the clicks and hits along with IP address, login information, and other details and as per demand can do dynamic pricing for the flights/ trains. Big Data helps in dynamic pricing and mind you it’s real time. Am sure each one of us has experienced this. Now you know who is doing it :DTelecommunications, Public sector, Education, Social media and gaming, Energy and utility every industry have implemented are implementing several of these Big Data use cases day in and day out. If you look around am sure you would find them on the rise.Big Data is helping everyone industries, consumers, clients to make informed decisions, whatever it may be and hence wherever there is such a need, Big Data can come handy.Challenges faced by Big Data in the real world for adaptationAlthough the world is going gaga about big data, there are still a few challenges to implement and adopt Big Data and hence service industries are still striving towards resolving those challenges to implement best Big Data solution without flaws.An October 2016 report from Gartner found that organizations were getting stuck at the pilot stage of their big data initiatives. "Only 15 percent of businesses reported deploying their big data project to production, effectively unchanged from last year (14 per cent)," the firm said.Let’s discuss a few of them to understand what are they?1. Understanding Big Data and answering Why for the organization one is working with.As I started the article saying there are many versions of Big Data and understanding real use cases for organization decision makers are working with is still a challenge. Everyone wants to ride on a wave but not knowing the right path is still a struggle. As every organization is unique thus its utmost important to answer ‘why big data’ for each organization. This remains a major challenge for decision makers to adapt to big data.2. Understanding Data sources for the organizationIn today’s world, there are hundreds and thousands of ways information is being generated and being aware of all these sources and ingest all of them into big data platforms to get accurate insight is essential. Identifying sources is a challenge to address.It's no surprise, then, that the IDG report found, "Managing unstructured data is growing as a challenge – rising from 31 per cent in 2015 to 45 per cent in 2016."Different tools and technologies are on the rise to address this challenge.3. Shortage if Big Data Talent and retaining themBig Data is changing technology and there are a whopping number of tools in the Big Data technology landscape. It is demanded out of Big Data professionals to excel in those current tools and keep up self to ever-changing needs. This gets difficult for employees and employers to create and retain talent within the organization.The solution to this would be constant upskilling, re-skilling and cross-skilling and increasing budget of organization for retaining talent and help them train.4. The Veracity VThis V is a challenge as this V means inconsistent, incomplete data processing. To gain insights through big data model, the biggest step is to predict and fill missing information.This is a tricky part as filling missing information can lead to decreasing accuracy of insights/ analytics etc.To address this concern, there is a bunch of tools. Data curation is an important step in big data and should have a proper model. But also, to keep in mind that Big Data is never 100% accurate and one must deal with it.5. SecurityThis aspect is given low priority during the design and build phases of Big Data implementations and security loopholes can cost an organization and hence it’s essential to put security first while designing and developing Big Data solutions. Also, equally important to act responsibly for implementations for regulatory requirements like GDPR.  6. Gaining Valuable InsightsMachine learning data models go through multiple iterations to conclude on insights as they also face issues like missing data and hence the accuracy. To increase accuracy, lots of re-processing is required, which has its own lifecycle. Increasing accuracy of insights is a challenge and which relates to missing data piece. Which most likely can be addressed by addressing missing data challenge.This can also be caused due to unavailability of information from all data sources. Incomplete information would lead to incomplete insights which may not benefit to required potential.Addressing these discussed challenges would help to gain valuable insights through available solutions.With Big Data, the opportunities are endless. Once understood, the world is yours!!!!Also, now that you understand BIG DATA, it's worth understanding the next steps:Gary King, who is a professor at Harvard says “Big data is not about the data. It is about the analytics”You can also take up Big Data and Hadoop training to enhance your skills furthermore.Did the article helps you to understand today’s massive world of big data and getting a sneak peek into it Do let us know through the comment section below?
Rated 4.5/5 based on 2 customer reviews

What is Big Data — An Introductory Guide

7K
What is Big Data — An Introductory Guide

The massive world of Big Data

If one strolls around any IT office premises, over every decade (nowadays time span is even lesser, almost every 3-4 years) one would overhear professionals discussing new jargons from the hottest trends in technology. Around 5 -6 years ago, one such word has started ruling IT services is ‘BIG data’ and still has been interpreted by a layman to tech geeks in various ways.

Although services industries started talking about big data solutions widely from 5-6 years, it is believed that the term was in use since the 1990s by John Mashey from Silicon Graphics, whereas credit for coining the term ‘big data’ aligning to its modern definition goes to Roger Mougalas from O’Reilly Media in 2005.

Let’s first understand why everyone going gaga about ‘BIG data’ and what are the real-world problems it is supposed to solve and then we will try to answer what and how aspects of it.

Why is Big Data essential for today’s digital world?

Pre smart-phones era, internet and web world were around for many years, but smart-phones made it mobile with on-the-go usage. Social Media, mobile apps started generating tons of data. At the same time, smart-bands, wearable devices ( IoT, M2M ), have given newer dimensions for data generation. This newly generated data became a new oil to the world. If this data is stored and analyzed, it has the potential to give tremendous insights which could be put to use in numerous ways.

You will be amazed to see the real-world use cases of BIG data. Every industry has a unique use case and is even unique to every client who is implementing the solutions. Ranging from data-driven personalized campaigning (you do see that item you have browsed on some ‘xyz’ site onto Facebook scrolling, ever wondered how?) to predictive maintenance of huge pipes across countries carrying oils, where manual monitoring is practically impossible. To relate this to our day to day life, every click, every swipe, every share and every like we casually do on social media is helping today’s industries to take future calculated business decisions. How do you think Netflix predicted the success of ‘House of Cards’ and spent $100 million on the same? Big data analytics is the simple answer.

Talking about all this, the biggest challenge in the past was traditional methods used to store, curate and analyze data, which had limitations to process this data generated from newer sources and which were huge in volumes generated from heterogeneous sources and was being generated  really fast(To give you an idea, roughly 2.5 quintillion data is generated per day as on today – Refer infographic released by Domo called “Data Never Sleeps 5.0.” ), Which given rise to term BIG data and related solutions.

Understanding Big Data: Experts’ viewpoint 

BIG data literally means Massive data (loosely > 1TB) but that’s not the only aspect of it. Distributed data or even complex datasets which could not be analyzed through traditional methods can be categorized into ‘Big data’ and hence Big data theoretical definition makes a lot of sense with this background:

“Gartner (2012) defines, Big data is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation.”

Generic data possessing characteristics of big data are 3Vs namely Variety, Velocity, and Volume

But due to the changing nature of data in today’s world and to gain most insights of it, 3 more Vs are added to the definition of BIG DATA, namely Variability, Veracity and Value.

The diagram below illustrates each V in detail:

 6 V’s of Big Data

Diagram: 6 V’s of Big Data

This 6Vs help understanding the characteristics of “BIG Data” but let’s also understand types of data in BIG Data processing.  
“Variety” of above characteristics caters to different types of data can be processed through big data tools and technologies. Let’s drill down a bit for understanding what those are:

  1. Structured ex. Mainframes, traditional databases like Teradata, Netezza, Oracle, etc.
  2. Unstructured ex. Tweets, Facebook posts, emails, etc.
  3. Semi/Multi structured or Hybrid ex. E-commerce, demographic, weather data, etc.

As the technology is advancing, the variety of data is available and its storage, processing, and analysis are made possible by big data. Traditional data processing techniques were able to process only structured data.

Now, that we understand what big data and limitations of old traditional techniques are of handling such data, we could safely say, we need new technology to handle this data and gain insights out of it. Before going further, do you know, what were the traditional data management techniques?

Traditional Techniques of Data Processing are:

  1. RDBMS (Relational Database Management System)
  2. Data warehousing and DataMart

On a high level, RDBMS catered to OLTP needs and data warehousing/DataMart facilitated OLAP needs. But both the systems work with structured data.

I hope. now one can answer, ‘what is big data?’ conceptually and theoretically both.

So, it’s time that we understand how it is being done in actual implementations.

only storing of “big data” will not help the organizations, what’s important is to turn data into insights and business value and to do so, following are the key infrastructure elements:

  • Data collection
  • Data storage
  • Data analysis and
  • Data visualization/output

All major big data processing framework offerings are based on these building blocks.

Traditional Techniques of Data Processing

And in an alignment of the above building blocks, following are the top 5 big data processing frameworks that are currently being used in the market:

1. Apache Hadoop : Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.First up is the all-time classic, and one of the top frameworks in use today. So prevalent is it, that it has almost become synonymous with Big Data.

2 Apache Spark : unified analytics engine for large-scale data processing.

Apache Spark and Hadoop are often contrasted as an "either/or" choice,  but that isn't really the case.

Above two frameworks are popular but apart from that following 3 are available and are comparable frameworks:

3. Apache Storm : free and open source distributed real-time computation system. You can also take up Apache Storm training to learn more about Apache Storm.

4. Apache Flink : streaming dataflow engine, aiming to provide facilities for distributed computation over streams of data. Treating batch processes as a special case of streaming data, Flink is effectively both batch and real-time processing framework, but one which clearly puts streaming first.

5. Apache Samza : distributed Stream processing framework.

Frameworks help processing data through building blocks and generate required insights. The framework is supported by the whopping number of tools providing the required functionality.

Big Data processing frameworks and technology landscape

Big data tools and technology landscape can be better understood with layered big data architecture. Give a good read to a great article by Navdeep singh Gill on XENONSTACK for understanding the layered architecture of big data.

By taking inspiration from layered architecture, different available tools in the market are mapped to layers to understand big data technology landscape in depth. Note that, layered architecture fits very well with infrastructure elements/building blocks discussed in the above section.

 Framework and technology landscape

Few of the tools are briefed below for further understanding:  

1. Data Collection / Ingestion Layer 

  • Cassandra: is a free and open-source, distributed, wide column store, NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure
  • Kafka: is used for building real-time data pipelines and streaming apps. Event streaming platform
  • Flume: log collector in Hadoop
  • HBase: columnar database in Hadoop

2. Processing Layer 

  • Pig: scripting language in the Hadoop framework
  • MapReduce: processing language in Hadoop

3. Data Query Layer 

  • Impala: Cloudera Impala:  modern, open source, distributed SQL query engine for Apache Hadoop. (often compared with hive)
  • Hive: Data Warehouse software for data Query and analysis
  • Presto: Presto is a high performance, distributed SQL query engine for big data. Its architecture allows users to query a variety of data sources such as Hadoop, AWS S3, Alluxio, MySQL, Cassandra, Apache Kafka, and MongoDB

4. Analytical Engine

  • TensorFlow: n source machine learning library for research and production.

5. Data storage Layer

  • Ignite: open-source distributed database, caching and processing platform designed to store and compute on large volumes of data across a cluster of nodes
  • Phoenix: hortonworks: Apache Phoenix is an open source, massively parallel, relational database engine supporting OLTP for Hadoop using Apache HBase as its backing store
  • PolyBase: s a new feature in SQL Server 2016. It is used to query relational and non-relational databases (NoSQL). You can use PolyBase to query tables and files in Hadoop or in Azure Blob Storage. You can also import or export data to/from Hadoop.
  • Sqoop: ETL tool
  • Big data in EXCEL: Few people like to process big datasets with current excel capabilities and it's known as Big Data in Excel

6. Data Visualization Layer

  • Microsoft HDInsight: Azure HDInsight is a Hadoop service offering hosted in Azure that enables clusters of managed Hadoop instances. Azure HDInsight deploys and provisions Apache Hadoop clusters in the cloud, providing a software framework designed to manage, analyze, and report on big data with high reliability and availability. Hadoop administration training will give you all the technical understanding required to manage a Hadoop cluster, either in a development or a production environment.

Best Practices in Big Data  

Every organization, industry, business, may it be small or big wants to get benefit out of “big data” but it's essential to understand that it can prove of maximum potential only if organization adhere to best practices before adapting big data:

Answering 5 basic questions help clients know the need for adapting Big Data for organization

  1. Try to answer why Big Data is required for the organization. What problem would it help solve?
  2. Ask the right questions.
  3. Foster collaboration between business and technology teams.
  4. Analyze only what is required to use.
  5. Start small and grow incrementally.

Big Data industry use-cases 

We talked about all the things in the Big Data world except real use cases of big data. In the starting, we did discuss few but let me give you insights into the real world and interesting big data use cases and for a few, it’s no longer a secret ☺. In fact, it’s penetrating to the extent you name the industry and plenty of use cases can be told. Let’s begin.

1.Streaming Platforms

As I had given an example of ‘House of Cards’ at the start of the article, it’s not a secret that Netflix uses Big Data analytics. Netflix spent $100mn on 26 episodes of ‘House of Cards’ as they knew the show would appeal to viewers of original British House of Cards and built in director David Fincher and actor Kevin Spacey. Netflix typically collects behavioral data and it then uses this data to create a better experience for the user.

But Netflix uses Big Data for more than that, they monitor and analyze traffic details for various devices, spot problem areas and adjust network infrastructure to prepare for future demand. (later is action out of Big Data analytics, how big data analysis is put to use). They also try to get insights into types of content viewers to prefer and help them make informed decisions.   

Streaming Platforms

Apart from Netflix, Spotify is also a known great use case.

2. Advertising and Media / Campaigning /Entertainment

For decades marketers were forced to launch campaigns while blindly relying on gut instinct and hoping for the best. That all changed with digitization and big data world. Nowadays, data-driven campaigns and marketing is on the rise and to be successful in this landscape, a modern marketing campaign must integrate a range of intelligent approaches to identify customers, segment, measure results, analyze data and build upon feedback in real time. All needs to be done in real time, along with the customer’s profile and history, based on his purchasing patterns and other relevant information and Big Data solutions are the perfect fit.

Event-driven marketing is also could be achieved through big data, which is another way of successful marketing in today’s world. That basically indicates, keeping track of events customer are directly and indirectly involved with and campaign exactly when a customer would need it rather than random campaigns. For. Ex if you have searched for a product on Amazon/Flipkart, you would see related advertisements on other social media apps you casually browse through. Bang on, you would end up purchasing it as you anyway needed options best to choose from.

Advertising and Media

3. Healthcare Industry

Healthcare is one of the classic use case industries for Big Data applications. The industry generates a huge amount of data.

Patients medical history, past records, treatments given, available and latest medicines, Medicinal latest available research the list of raw data is endless.

All this data can help give insights and Big Data can contribute to the industry in the following ways:

  1. Diagnosis time could be reduced, and exact requirement treatment could be started immediately. Most of the illnesses could be treated if a diagnosis is perfect and treatment can be started in time. This can be achieved through evidence-based past medical data available for similar treatments to doctor treating the illness, patients’ available history and feeding symptoms real-time into the system.  
  2. Government Health department can monitor if a bunch of people from geography reporting of similar symptoms, predictive measures could be taken in nearby locations to avoid outbreak as a cause for such illness could be the same.   

The list is long, above were few representative examples.

4. Security

Due to social media outbreak, today, personal information is at stake. Almost everything is digital, and majority personal information is available in the public domain and hence privacy and security are major concerns with the rise in social media. Following are few such applications for big data.

  1. Cyber Crimes are common nowadays and big data can help to detect, predicting crimes.
  2. Threat analysis and detection could be done with big data.  

5. Travel and Tourism

Flight booking sites, IRCTC track the clicks and hits along with IP address, login information, and other details and as per demand can do dynamic pricing for the flights/ trains. Big Data helps in dynamic pricing and mind you it’s real time. Am sure each one of us has experienced this. Now you know who is doing it :D

Telecommunications, Public sector, Education, Social media and gaming, Energy and utility every industry have implemented are implementing several of these Big Data use cases day in and day out. If you look around am sure you would find them on the rise.

Big Data is helping everyone industries, consumers, clients to make informed decisions, whatever it may be and hence wherever there is such a need, Big Data can come handy.

Challenges faced by Big Data in the real world for adaptation

Challenges faced by Big Data in the real world for adaptation

Although the world is going gaga about big data, there are still a few challenges to implement and adopt Big Data and hence service industries are still striving towards resolving those challenges to implement best Big Data solution without flaws.

An October 2016 report from Gartner found that organizations were getting stuck at the pilot stage of their big data initiatives. "Only 15 percent of businesses reported deploying their big data project to production, effectively unchanged from last year (14 per cent)," the firm said.

Let’s discuss a few of them to understand what are they?

1. Understanding Big Data and answering Why for the organization one is working with.

As I started the article saying there are many versions of Big Data and understanding real use cases for organization decision makers are working with is still a challenge. Everyone wants to ride on a wave but not knowing the right path is still a struggle. As every organization is unique thus its utmost important to answer ‘why big data’ for each organization. This remains a major challenge for decision makers to adapt to big data.

2. Understanding Data sources for the organization

In today’s world, there are hundreds and thousands of ways information is being generated and being aware of all these sources and ingest all of them into big data platforms to get accurate insight is essential. Identifying sources is a challenge to address.

It's no surprise, then, that the IDG report found, "Managing unstructured data is growing as a challenge – rising from 31 per cent in 2015 to 45 per cent in 2016."

Different tools and technologies are on the rise to address this challenge.

3. Shortage if Big Data Talent and retaining them

Big Data is changing technology and there are a whopping number of tools in the Big Data technology landscape. It is demanded out of Big Data professionals to excel in those current tools and keep up self to ever-changing needs. This gets difficult for employees and employers to create and retain talent within the organization.

The solution to this would be constant upskilling, re-skilling and cross-skilling and increasing budget of organization for retaining talent and help them train.

4. The Veracity V

This V is a challenge as this V means inconsistent, incomplete data processing. To gain insights through big data model, the biggest step is to predict and fill missing information.

This is a tricky part as filling missing information can lead to decreasing accuracy of insights/ analytics etc.

To address this concern, there is a bunch of tools. Data curation is an important step in big data and should have a proper model. But also, to keep in mind that Big Data is never 100% accurate and one must deal with it.

5. Security

This aspect is given low priority during the design and build phases of Big Data implementations and security loopholes can cost an organization and hence it’s essential to put security first while designing and developing Big Data solutions. Also, equally important to act responsibly for implementations for regulatory requirements like GDPR.  

6. Gaining Valuable Insights

Machine learning data models go through multiple iterations to conclude on insights as they also face issues like missing data and hence the accuracy. To increase accuracy, lots of re-processing is required, which has its own lifecycle. Increasing accuracy of insights is a challenge and which relates to missing data piece. Which most likely can be addressed by addressing missing data challenge.

This can also be caused due to unavailability of information from all data sources. Incomplete information would lead to incomplete insights which may not benefit to required potential.

Addressing these discussed challenges would help to gain valuable insights through available solutions.

With Big Data, the opportunities are endless. Once understood, the world is yours!!!!

Also, now that you understand BIG DATA, it's worth understanding the next steps:

Gary King, who is a professor at Harvard says “Big data is not about the data. It is about the analytics”

You can also take up Big Data and Hadoop training to enhance your skills furthermore.

Did the article helps you to understand today’s massive world of big data and getting a sneak peek into it Do let us know through the comment section below?

Shruti

Shruti Deshpande

Blog Author

10+ years of data-rich experience in the IT industry. It started with data warehousing technologies into data modelling to BI application Architect and solution architect.


Big Data enthusiast and data analytics is my personal interest. I do believe it has endless opportunities and potential to make the world a sustainable place. Happy to ride on this tide.


*Disclaimer* - Expressed views are the personal views of the author and are not to be mistaken for the employer or any other organization’s views.

Join the Discussion

Your email address will not be published. Required fields are marked *

2 comments

shivkumar 01 Jun 2019 1 likes

Thanks for sharing this amazing blog.It is really an informative post.

Nisha 18 Jun 2019 1 likes

The article looks good and the way of presentation is nice.

Suggested Blogs

Apache Kafka Vs Apache Spark: Know the Differences

A new breed of ‘Fast Data’ architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. - Dean Wampler (Renowned author of many big data technology-related books)Dean Wampler makes an important point in one of his webinars. The demand for stream processing is increasing every day in today’s era. The main reason behind it is, processing only volumes of data is not sufficient but processing data at faster rates and making insights out of it in real time is very essential so that organization can react to changing business conditions in real time.And hence, there is a need to understand the concept “stream processing “and technology behind it. So, what is Stream Processing?Think of streaming as an unbounded, continuous real-time flow of records and processing these records in similar timeframe is stream processing. AWS (Amazon Web Services) defines “Streaming Data” is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of Kilobytes). This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows and used for a wide variety of analytics including correlations, aggregations, filtering, and sampling.In stream processing method, continuous computation happens as the data flows through the system.Stream processing is highly beneficial if the events you wish to track are happening frequently and close together in time. It is also best to utilize if the event needs to be detected right away and responded to quickly.There is a subtle difference between stream processing, real-time processing (Rear real-time) and complex event processing (CEP). Let’s quickly look at the examples to understand the difference. Stream Processing: Stream processing is useful for tasks like fraud detection and cybersecurity. If transaction data is stream-processed, fraudulent transactions can be identified and stopped before they are even complete.Real-time Processing: If event time is very relevant and latencies in the second's range are completely unacceptable then it’s called Real-time (Rear real-time) processing. For ex. flight control system for space programsComplex Event Processing (CEP): CEP utilizes event-by-event processing and aggregation (for example, on potentially out-of-order events from a variety of sources, often with large numbers of rules or business logic).We have multiple tools available to accomplish above-mentioned Stream, Realtime or Complex event Processing. Spark Streaming, Kafka Stream, Flink, Storm, Akka, Structured streaming are to name a few. We will try to understand Spark streaming and Kafka stream in depth further in this article. As historically, these are occupying significant market share. Apache Kafka Stream: Kafka is actually a message broker with a really good performance so that all your data can flow through it before being redistributed to applications. Kafka works as a data pipeline.Typically, Kafka Stream supports per-second stream processing with millisecond latency.  Kafka Streams is a client library for processing and analyzing data stored in Kafka. Kafka streams can process data in 2 ways. Kafka -> Kafka: When Kafka Streams performs aggregations, filtering etc. and writes back the data to Kafka, it achieves amazing scalability, high availability, high throughput etc.  if configured correctly. It also does not do mini batching, which is “real streaming”.Kafka -> External Systems (‘Kafka -> Database’ or ‘Kafka -> Data science model’): Typically, any streaming library (Spark, Flink, NiFi etc) uses Kafka for a message broker. It would read the messages from Kafka and then break it into mini time windows to process it further. Representative view of Kafka streaming: Note:Sources here could be event logs, webpage events etc. etc. DB/Models would be accessed via any other streaming application, which in turn is using Kafka streams here. Kafka Streams is built upon important stream processing concepts such as properly distinguishing between event time and processing time, windowing support, and simple (yet efficient) management of application state. It is based on many concepts already contained in Kafka, such as scaling by partitioning.Also, for this reason, it comes as a lightweight library that can be integrated into an application.The application can then be operated as desired, as mentioned below: Standalone, in an application serverAs a Docker container, or Directly, via a resource manager such as Mesos.Why one will love using dedicated Apache Kafka Streams?Elastic, highly scalable, fault-tolerantDeploy to containers, VMs, bare metal, cloudEqually viable for small, medium, & large use casesFully integrated with Kafka securityWrite standard Java and Scala applicationsExactly-once processing semanticsNo separate processing cluster requiredDevelop on Mac, Linux, WindowsApache Spark Streaming:Spark Streaming receives live input data streams, it collects data for some time, builds RDD, divides the data into micro-batches, which are then processed by the Spark engine to generate the final stream of results in micro-batches. Following data flow diagram explains the working of Spark streaming. Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data. DStreams can be created either from input data streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs. Think about RDD as the underlying concept for distributing data over a cluster of computers. Why one will love using Apache Spark Streaming?It makes it very easy for developers to use a single framework to satisfy all the processing needs. They can use MLib (Spark's machine learning library) to train models offline and directly use them online for scoring live data in Spark Streaming. In fact, some models perform continuous, online learning, and scoring.Not all real-life use-cases need data to be processed at real real-time, few seconds delay is tolerated over having a unified framework like Spark Streaming and volumes of data processing. It provides a range of capabilities by integrating with other spark tools to do a variety of data processing.  Spark Streaming Vs Kafka StreamNow that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Following table briefly explain you, key differences between the two. Sr.NoSpark streamingKafka Streams1Data received form live input data streams is Divided into Micro-batched for processing.processes per data stream(real real-time)2Separated processing Cluster is requriedNo separated processing cluster is requried.3Needs re-configuration for Scaling Scales easily by just adding java processes, No reconfiguration requried.4At least one semanticsExactly one semantics5Spark streaming is better at processing group of rows(groups,by,ml,window functions etc.)Kafka streams provides true a-record-at-a-time processing capabilities. it's better for functions like rows parsing, data cleansing etc.6Spark streaming is standalone framework.Kafka stream can be used as part of microservice,as it's just a library.Kafka streams Use-cases:Following are a couple of many industry Use cases where Kafka stream is being used: The New York Times: The New York Times uses Apache Kafka and Kafka Streams to store and distribute, in real-time, published content to the various applications and systems that make it available to the readers.Pinterest: Pinterest uses Apache Kafka and the Kafka Streams at large scale to power the real-time, predictive budgeting system of their advertising infrastructure. With Kafka Streams, spend predictions are more accurate than ever.Zalando: As the leading online fashion retailer in Europe, Zalando uses Kafka as an ESB (Enterprise Service Bus), which helps us in transitioning from a monolithic to a micro services architecture. Using Kafka for processing event streams enables our technical team to do near-real time business intelligence.Trivago: Trivago is a global hotel search platform. We are focused on reshaping the way travellers search for and compare hotels while enabling hotel advertisers to grow their businesses by providing access to a broad audience of travellers via our websites and apps. As of 2017, we offer access to approximately 1.8 million hotels and other accommodations in over 190 countries. We use Kafka, Kafka Connect, and Kafka Streams to enable our developers to access data freely in the company. Kafka Streams powers parts of our analytics pipeline and delivers endless options to explore and operate on the data sources we have at hand.Broadly, Kafka is suitable for microservices integration use cases and have wider flexibility.Spark Streaming Use-cases:Following are a couple of the many industries use-cases where spark streaming is being used: Booking.com: We are using Spark Streaming for building online Machine Learning (ML) features that are used in Booking.com for real-time prediction of behaviour and preferences of our users, demand for hotels and improve processes in customer support. Yelp: Yelp’s ad platform handles millions of ad requests every day. To generate ad metrics and analytics in real-time, they built the ad event tracking and analyzing pipeline on top of Spark Streaming. It allows Yelp to manage a large number of active ad campaigns and greatly reduce over-delivery. It also enables them to share ad metrics with advertisers in a timelier fashion.Spark Streaming’s ever-growing user base consists of household names like Uber, Netflix, and Pinterest.Broadly, spark streaming is suitable for requirements with batch processing for massive datasets, for bulk processing and have use-cases more than just data streaming. Dean Wampler explains factors to evaluation for tool basis Use-cases beautifully, as mentioned below: Sr.NoEvaluation CharacteristicResponse Time windowTypical Use Case Requirement1.Latency tolerancePico to Microseconds (Real Real time)Flight control system for space programs etc.Latency tolerance< 100 MicrosecondsRegular stock trading market transactions, Medical diagnostic equipment outputLatency tolerance< 10 millisecondsCredit cards verification window when consumer buy stuff onlineLatency tolerance< 100 millisecondshuman attention required Dashboards, Machine learning modelsLatency tolerance< 1 second to minutesMachine learning model trainingLatency tolerance1 minute and abovePeriodic short jobs(typical ETL applications)2.Evaluation CharacteristicTransaction/events frequencyTypical Use Case RequirementVelocity1M per secondNest Thermostat, Big spikes during specific time period.3Evaluation CharacteristicTypes of data processingNAData Processing Requirement1. SQLNA2. ETL3. Dataflow4. Training and/or Serving Machine learning modelsData Processing Requirement1. Bulk data processingNA2. Individual Events/Transaction processing4.Evaluation CharacteristicUse of toolNAFlexibility of implementation1. Kafka : flexible as provides library.NA2. Spark: Not flexible as it’s part of a distributed frameworkConclusionKafka Streams is still best used in a ‘Kafka -> Kafka’ context, while Spark Streaming could be used for a ‘Kafka -> Database’ or ‘Kafka -> Data science model’ type of context.Although, when these 2 technologies are connected, they bring complete data collection and processing capabilities together and are widely used in commercialized use cases and occupy significant market share. 
Rated 4.5/5 based on 19 customer reviews
9874
Apache Kafka Vs Apache Spark: Know the Differences

A new breed of ‘Fast Data’ architectures has e... Read More

Apache Spark Vs Hadoop - Head to Head Comparison

Over the past few years, data science has been one of the most sought-after multidisciplinary fields in the world today. It has established itself as an essential component of numerous industries such as marketing optimisation, risk management, marketing analytics. fraud detection, agriculture, etc. Understandably, this has lead to increasing demand for resorting to different approaches to data.When we talk about Apache Spark and Hadoop, it is really difficult to compare them with each other. We should be aware that both possess important features in the world of data science and big data. Hadoop excels over Apache Spark in some business applications, but when processing speed and ease of use is taken into account, Apache Spark has its own advantages that make it unique. The most important thing to note is, neither of these two can replace each other. However, since they are compatible with each other, they can be used together to produce very effective results for many big data applications.To analyse how important these two platforms are, there is a set of parameters with which we can discuss their efficiencies such as performance, ease of use, cost, data processing, compatibility, fault tolerance, scalability, and security. In this article, we will talk about Apache Spark and Hadoop individually for a bit, followed by stressing these parameters to better understand their significance in data science and big data.What is Hadoop?Hadoop, also known as Apache Hadoop, is a project formed by Apache.org that includes a software library and a framework that enables the usage of simple programming models to distributed processing of large data sets (big data) across computer clusters. Hadoop is quite efficient in scaling up from single computer systems to a lot of commodity system, offering substantial local storage. Due to this, Hadoop is considered as an omnipresent heavyweight in the big data analytics space. There are modules that work together to form the Hadoop framework. Here are the main Hadoop framework modules:Hadoop CommonHadoop Distributed File System (HDFS)Hadoop YARNHadoop MapReduceHadoop’s core is based on the above four modules followed by many others like Ambari, Avro, Cassandra, Hive, Pig, Oozie, Flume, and Sqoop. These are responsible for improving and extending Hadoop’s power to big data applications and large data set processing.Hadoop is utilised by numerous companies using big data sets and analytics and is the de facto model for big data applications. Initially, it was designed to take care of crawling and searching billions of web pages and collecting their information into a database, This resulted in Hadoop Distributed File System (HDFS), a distributed file system designed to run on commodity hardware and Hadoop MapReduce, a processing technique and a program model for distributed computing based on java.Hadoop comes handy when companies find data sets too large and complex to not being able to process the information in reasonably sufficient time. Since crawling and searching the web are text-based tasks, Hadoop MapReduce comes in handy as it is an exceptional text processing engine.An Overview of Apache SparkAn open-source distributed general-purpose cluster-computing framework, Apache Spark is considered as a fast and general engine for large-scale data processing. Compared to heavyweight Hadoop’s Big Data framework, Spark is very lightweight and faster by nearly 100 times. Although the facts say so, in fact, Spark runs up to 10 times faster on disk. Apart from that, it can perform batch processing but it really is good at streaming workloads, interactive queries, and machine-based learning.✓Streaming workloads✓Interactive queries✓Machine-based learning.Spark engine’s real-time data processing capability has a clear edge over Hadoop MapReduce’s disk-bound, batch processing one. Not only is Spark compatible with Hadoop and its modules, but it is also listed as a module on Hadoop’s project page. And because Spark can run in Hadoop clusters through YARN (Yet Another Resource Negotiator), it has its own page and a standalone mode. It can run as a Hadoop module and as a standalone solution which makes it difficult to make direct comparisons.Despite these facts, Spark is expected to diverge and might even replace Hadoop, especially in terms of faster access to processed data. Spark’s cluster computing feature enables it to compete with only Hadoop MapReduce and not the entire Hadoop ecosystem. That is why it can use HDFS despite not having its own distributed file system. To be concise, Hadoop MapReduce uses persistent storage whereas Spark uses Resilient Distributed Datasets (RDDs). What is RDD? This will be stressed in the Fault Tolerance section.The differences between Apache Spark and HadoopLet us have a look at the parameters using which we can compare the features of Apache Spark with Hadoop.Apache Spark vs Hadoop in a nutshellApache SparkParametersHadoopProcesses everything in memoryPerformance-wiseHadoop MapReduce uses batch processingHas user-friendly APIs for multiple programming languagesEase of UseHas add-ons such as Hive and PigSpark systems cost moreCostsHadoop MapReduce systems cost lesserShares every Hadoop MapReduce compatibilityCompatibilityCompliments Apache Spark seamlesslyHas GraphX, its own graph computation libraryData ProcessingHadoop MapReduce operates in sequential stepsSpark uses Resilient Distributed Datasets (RDDs)Fault ToleranceUtilises TaskTrackers to keep the JobTracker tickingComparatively lesser scalabilityScalabilityLarge ScalabilityProvides authentication via shared secret (password authentication)SecuritySupports Kerberos authenticationPerformance-wiseSpark is definitely faster when compared to Hadoop MapReduce. However, they cannot be compared because they perform processing in different styles. Spark is way faster because it processes everything in memory, even using disk for data that does not all fit into memory. The in-memory processing of Spark performs near real-time analytics for data from machine learning, log monitoring, marketing campaigns, Internet of Things sensors, security analytics, and social media sites. Hadoop MapReduce, on the other hand, utilises the batch-processing method so it understandably was never created for mesmerising speed. As a matter of fact, it was initially created to continuously gather information from websites during the times when data in or near real-time were not required.Ease of UseSpark does not only have a good reputation for its excellent performance, but it is also relatively easy to use along with providing additional support for languages like user-friendly APIs for Scala, Java, Python, and Spark SQL. Since Spark SQL is quite comparable to SQL 92, the user requires no additional knowledge to use it.Supported Languages:APIs for ScalaJavaPythonSpark SQL.Additionally, Spark is armed with an interactive mode to allow developers and users get instant feedback for questions and other actions. Hadoop MapReduce makes up for the lack of any interactive mode with add-ons like Hive and Pig, thus easing the workflow of Hadoop MapReduce.CostsApache Spark and Apache Hadoop MapReduce are both free open-source software.However, because Hadoop MapReduce’s processing is disk-based, it utilises standard volumes of memory. This results in companies buying faster disks with a lot of disk space to run Hadoop MapReduce. In stark contrast to this, Spark requires a lot of memory but compensates by settling with a standard amount of disk space running at standard speeds.Apache Spark and Apache Hadoop CompatibilityBoth Spark and Hadoop MapReduce are compatible with each other. Moreover, Spark shares every Hadoop MapReduce compatibility for data sources, file formats, and business intelligence tools via JDBC and ODBC.Data ProcessingHadoop MapReduce is a batch-processing engine. So how does it work? Well, it works in sequential steps.Step 1: Reads data from the clusterStep 2: Performs its operation on the dataStep 3: Writes the results back to the clusterStep 4: Reads updated data from the clusterStep 5: Performs the next data operationStep 6: Writes those results back to the clusterStep 7: Repeat.Spark performs in a similar manner, but the process doesn’t go on. It includes a single step and then to memory.Step 1: Reads data from the clusterStep 2: Performs its operation on the dataStep 3: Writes it back to the cluster.Moreover, Spark has GraphX, its own graph computation library. GraphX presents the same data as graphs and collections. Users have the option to use Resilient Distributed Datasets (RDDs) to transform and join graphs. This will be further addressed below in the Fault Tolerance section.Fault ToleranceThere are two different ways in which Hadoop MapReduce and Spark resolve the fault tolerance issue. Hadoop MapReduce utilises nodes like TaskTrackers to keep the JobTracker ticking. On the process being interrupted, the JobTracker reassigns every pending and in-progress operation to another TaskTracker. Although this process effectively provides fault tolerance, the completion times might get majorly affected even for operations having just a single failure.Spark, in this case, applies Resilient Distributed Datasets (RDDs), fault-tolerant collections of elements that can be operated side by side. References can be provided by RDDs in the form of datasets in an external storage system like shared filesystems, HDFS, HBase, or whatever available data source. This results in allowing a Hadoop InputFormat and Spark can create RDDs from every storage source that is backed by Hadoop. That covers local filesystems or one of those listed earlier.Below-mentioned is five main properties that an RDD possesses:A list of partitionsA function for computing each splitA list of dependencies on other RDDsA Partitioner for key-value RDDs by choice (provided that the RDD is hash-partitioned)Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)The persistence of RDDs to cache a dataset in memory across operations enables the speeding up of future actions by possibly ten folds. The cache of Spark is fault-tolerant, it will recomputed automatically by making use of the original transformations provided any partition of an RDD is lost.ScalabilityIn terms of scaling up, both Hadoop MapReduce and Spark are on equal terms in using the HDFS. Reports say that Yahoo holds a 42,000 node Hadoop cluster with no bounds while the most comprehensive Spark cluster holds 8,000 nodes. However, in order to support output expectations, the cluster sizes are expected to grow along with that of big data.SecurityKerberos authentication, considered to be quite hectic to manage is supported by Hadoop. Nevertheless, companies have been assisted by third-party vendors to leverage Active Directory Kerberos and LDAP for authentication and also allow data encrypt for in-flight and data at rest. Access control lists (ACLs) a traditional file permissions model are supported by Hadoop while it provides Service Level Authorization for user control in job submission, resulting in clients having the right permissions without any fail.For Spark though, it presently offers somewhat inadequate security as it provides authentication via shared secret (password authentication). However, if the user runs Spark on HDFS, then it can utilise HDFS ACLs and file-level permissions. Moreover, running Spark on YARN will enable the latter to have the capacity of using Kerberos authentication. That is the security takeaway from using Spark.  ConclusionApache Spark and Apache Hadoop form the perfect combination for business applications. Where Hadoop MapReduce has been a revelation in the big data market for businesses requiring huge datasets to be brought under control by commodity systems, Apache Spark’s speed and comparative ease of use compliments the low-cost operation involving Hadoop MapReduce.Like we discussed at the beginning of this article that neither of these two can replace one another, Spark and Hadoop form a lethal and effective symbiotic partnership. While Hadoop has features like a distributed file system that Spark does not have, the latter presents real-time, in-memory processing for the required data sets. Both Hadoop and Spark form the perfect combination for the ideal big data scenario. Rest assured, in this situation, both working in the same team is what goes in favour of big data professionals.You would be interested to know that Knowledgehut offers world-class training for Apache Spark and Hadoop. Feel free to check these courses to enhance your knowledge about both Apache Spark and Hadoop.
Rated 4.5/5 based on 2 customer reviews
6639
Apache Spark Vs Hadoop - Head to Head Comparison

Over the past few years, data science has been one... Read More

Apache Spark Vs Apache Storm - Head To Head Comparison

In today’s world, the need for real-time data streaming is growing exponentially due to the increase in real-time data. With streaming technologies leading the world of Big Data, it might be tough for the users to choose the appropriate real-time streaming platform. Two of the most popular real-time technologies that might consider for opting are Apache Spark and Apache Storm. One major key difference between the frameworks Spark and Storm is that Spark performs Data-Parallel computations, whereas Storm occupies Task-Parallel computations. Read along to know more differences between Apache Spark and Apache Storm, and understand which one is better to adopt on the basis of different features. Comparison Table: Apache Spark Vs. Apache StormSr. NoParameterApache SparkApache Storm1.Processing  ModelBatch ProcessingMicro-batch processing2.Programming LanguageSupports lesser languages like Java, Scala.Support smultiple languages, such as Scala, Java, Clojure.3.Stream SourcesHDFSSpout4.MessagingAkka, NettyZeroMQ, Netty5.Resource ManagementYarn and Meson are responsible.Yarn and Mesos are responsible.6.Low LatencyHigher latency as compared to SparkBetter latency with lesser constraints7.Stream PrimitivesDStreamTuple, Partition8.Development CostSame code can be used for batch and stream processing.Same code cannot be used for batch and stream processing.9.State ManagementSupports State ManagementSupports State Management as well10.Message Delivery GuaranteesSupports one message processing mode: ‘at least once’.Supports three message processing mode: ‘at least once’, ‘at most once’, ‘exactly once’.11.Fault ToleranceIf a process fails, Spark restarts workers via resource managers. (YARN, Mesos)If a process fails, the supervisor process starts automatically.12.Throughput100k records per node per second10k records per node per second13.PersistenceMapStatePer RDD14.ProvisioningBasic monitoring using GangliaApache Ambaripache Spark: Apache Spark is a general-purpose, lighting fast, cluster-computing technology framework, used for fast computation on large-scale data processing. It can manage both batch and real-time analytics and data processing workloads.  Spark was developed at UC Berkeley in the year 2009. Apache Storm:Apache Storm is an open-source, scalable fault-tolerant, and real-time stream processing computation system. It is a framework for real-time distributed data processing, which focuses on stream processing or event processing. It can be used with any programming language and can be integrated using any queuing or database technology.  Apache Storm was developed by a team led by Nathan Marz at BackType Labs. Apache Spark Vs. Apache Storm1. Processing Model: Apache Storm supports micro-batch processing, while Apache Spark supports batch processing. 2. Programming Language:Storm applications can be created using multiple languages like Java, Scala and Clojure, while Spark applications can be created using Java and Scala.3. Stream Sources:For Storm, the source of stream processing is Spout, while that for Spark is HDFS. 4. Messaging:Storm uses ZeroMQ and Netty as its messaging layer while Spark is using a combination of Nettu and Akka for distributing the messages throughout the executors. 5. Resource Management:Yarn and Meson are responsible for resource management in Spark, while Yarn and Mesos are responsible for resource management in Storm. 6. Low Latency: Spark provides higher latency as compared to Apache Storm, whereas Storm can provide better latency with fewer restrictions.7. Stream Primitives:Spark provides with stream transforming operators which transform DStream into another, while Storm provides with various primitives which perform tuple level of processing at the stream level (functions, filters). 8. Development Cost:It is possible for Spark to use the same code base for both stream processing and batch processing. Whereas for Storm, the same code base cannot be used for both stream processing and batch processing.  9. State Management: The changing and maintaining state in Apache Spark can be updated via UpdateStateByKey, but no pluggable strategy can be applied in the external system for the implementation of state. Whereas Storm does not provide any framework for the storage of any intervening bolt output as a state. Hence, each application has to create a state for itself whenever required. 10. Message Delivery Guarantees (Handling the message level failures):Apache Spark supports only one message processing mode, viz, ‘at least once’, whereas Storm supports three message processing modes, viz, ‘at least once’ (Tuples are processed at least one time, but can be processed more than once), ‘at most once’  and ‘exactly once’ (T^uples are processed at least once). Storm’s reliability mechanisms are scalable, distributed and fault-tolerant. 11. Fault-Tolerant:Apache Spark and Apache Storm, both are fault tolerant to nearly the same extent. If a process fails in Apache Storm, then the supervisor process will restart it automatically, as the state management is managed by Zookeeper, while Spark restarts its workers with the help of resource managers, who may be Mesos, YARN or its separate manager.12. Ease of Development: In the case of Storm, there are effective and easy to use APIs which show that the nature of topology is DAG. The Storm tuples are written dynamically. In the case of Spark, it consists of Java and Scala APIs with practical programming, making topology code a bit difficult to understand. But since the API documentation and samples are easily available for the developers, it is now easier. Summing Up: Apache Spark Vs Apache StormApache Storm and Apache Spark both offer great solutions to solve the transformation problems and streaming ingestions. Moreover, both can be a part of a Hadoop cluster to process data. While Storm acts as a solution for real-time stream processing, developers might find it to be quite complex to develop applications due to its limited resources. The industry is always on a lookout for a generalized solution, which has the ability to solve all types of problems, such as Batch processing, interactive processing, iterative processing and stream processing. Keeping all these points in mind, this is where Apache Spark steals the limelight as it is mostly considered as a general-purpose computation engine, making it a highly demanding tool by IT professionals. It can handle various types of problems and provides a flexible environment to in. Moreover, developers find it to be easy and are able to integrate it well with Hadoop. 
Rated 4.5/5 based on 2 customer reviews
6612
Apache Spark Vs Apache Storm - Head To Head Compar...

In today’s world, the need for real-time data st... Read More