Search

Apache Spark Vs Hadoop - Head to Head Comparison

Over the past few years, data science has been one of the most sought-after multidisciplinary fields in the world today. It has established itself as an essential component of numerous industries such as marketing optimisation, risk management, marketing analytics. fraud detection, agriculture, etc. Understandably, this has lead to increasing demand for resorting to different approaches to data.When we talk about Apache Spark and Hadoop, it is really difficult to compare them with each other. We should be aware that both possess important features in the world of data science and big data. Hadoop excels over Apache Spark in some business applications, but when processing speed and ease of use is taken into account, Apache Spark has its own advantages that make it unique. The most important thing to note is, neither of these two can replace each other. However, since they are compatible with each other, they can be used together to produce very effective results for many big data applications.To analyse how important these two platforms are, there is a set of parameters with which we can discuss their efficiencies such as performance, ease of use, cost, data processing, compatibility, fault tolerance, scalability, and security. In this article, we will talk about Apache Spark and Hadoop individually for a bit, followed by stressing these parameters to better understand their significance in data science and big data.What is Hadoop?Hadoop, also known as Apache Hadoop, is a project formed by Apache.org that includes a software library and a framework that enables the usage of simple programming models to distributed processing of large data sets (big data) across computer clusters. Hadoop is quite efficient in scaling up from single computer systems to a lot of commodity system, offering substantial local storage. Due to this, Hadoop is considered as an omnipresent heavyweight in the big data analytics space. There are modules that work together to form the Hadoop framework. Here are the main Hadoop framework modules:Hadoop CommonHadoop Distributed File System (HDFS)Hadoop YARNHadoop MapReduceHadoop’s core is based on the above four modules followed by many others like Ambari, Avro, Cassandra, Hive, Pig, Oozie, Flume, and Sqoop. These are responsible for improving and extending Hadoop’s power to big data applications and large data set processing.Hadoop is utilised by numerous companies using big data sets and analytics and is the de facto model for big data applications. Initially, it was designed to take care of crawling and searching billions of web pages and collecting their information into a database, This resulted in Hadoop Distributed File System (HDFS), a distributed file system designed to run on commodity hardware and Hadoop MapReduce, a processing technique and a program model for distributed computing based on java.Hadoop comes handy when companies find data sets too large and complex to not being able to process the information in reasonably sufficient time. Since crawling and searching the web are text-based tasks, Hadoop MapReduce comes in handy as it is an exceptional text processing engine.An Overview of Apache SparkAn open-source distributed general-purpose cluster-computing framework, Apache Spark is considered as a fast and general engine for large-scale data processing. Compared to heavyweight Hadoop’s Big Data framework, Spark is very lightweight and faster by nearly 100 times. Although the facts say so, in fact, Spark runs up to 10 times faster on disk. Apart from that, it can perform batch processing but it really is good at streaming workloads, interactive queries, and machine-based learning.✓Streaming workloads✓Interactive queries✓Machine-based learning.Spark engine’s real-time data processing capability has a clear edge over Hadoop MapReduce’s disk-bound, batch processing one. Not only is Spark compatible with Hadoop and its modules, but it is also listed as a module on Hadoop’s project page. And because Spark can run in Hadoop clusters through YARN (Yet Another Resource Negotiator), it has its own page and a standalone mode. It can run as a Hadoop module and as a standalone solution which makes it difficult to make direct comparisons.Despite these facts, Spark is expected to diverge and might even replace Hadoop, especially in terms of faster access to processed data. Spark’s cluster computing feature enables it to compete with only Hadoop MapReduce and not the entire Hadoop ecosystem. That is why it can use HDFS despite not having its own distributed file system. To be concise, Hadoop MapReduce uses persistent storage whereas Spark uses Resilient Distributed Datasets (RDDs). What is RDD? This will be stressed in the Fault Tolerance section.The differences between Apache Spark and HadoopLet us have a look at the parameters using which we can compare the features of Apache Spark with Hadoop.Apache Spark vs Hadoop in a nutshellApache SparkParametersHadoopProcesses everything in memoryPerformance-wiseHadoop MapReduce uses batch processingHas user-friendly APIs for multiple programming languagesEase of UseHas add-ons such as Hive and PigSpark systems cost moreCostsHadoop MapReduce systems cost lesserShares every Hadoop MapReduce compatibilityCompatibilityCompliments Apache Spark seamlesslyHas GraphX, its own graph computation libraryData ProcessingHadoop MapReduce operates in sequential stepsSpark uses Resilient Distributed Datasets (RDDs)Fault ToleranceUtilises TaskTrackers to keep the JobTracker tickingComparatively lesser scalabilityScalabilityLarge ScalabilityProvides authentication via shared secret (password authentication)SecuritySupports Kerberos authenticationPerformance-wiseSpark is definitely faster when compared to Hadoop MapReduce. However, they cannot be compared because they perform processing in different styles. Spark is way faster because it processes everything in memory, even using disk for data that does not all fit into memory. The in-memory processing of Spark performs near real-time analytics for data from machine learning, log monitoring, marketing campaigns, Internet of Things sensors, security analytics, and social media sites. Hadoop MapReduce, on the other hand, utilises the batch-processing method so it understandably was never created for mesmerising speed. As a matter of fact, it was initially created to continuously gather information from websites during the times when data in or near real-time were not required.Ease of UseSpark does not only have a good reputation for its excellent performance, but it is also relatively easy to use along with providing additional support for languages like user-friendly APIs for Scala, Java, Python, and Spark SQL. Since Spark SQL is quite comparable to SQL 92, the user requires no additional knowledge to use it.Supported Languages:APIs for ScalaJavaPythonSpark SQL.Additionally, Spark is armed with an interactive mode to allow developers and users get instant feedback for questions and other actions. Hadoop MapReduce makes up for the lack of any interactive mode with add-ons like Hive and Pig, thus easing the workflow of Hadoop MapReduce.CostsApache Spark and Apache Hadoop MapReduce are both free open-source software.However, because Hadoop MapReduce’s processing is disk-based, it utilises standard volumes of memory. This results in companies buying faster disks with a lot of disk space to run Hadoop MapReduce. In stark contrast to this, Spark requires a lot of memory but compensates by settling with a standard amount of disk space running at standard speeds.Apache Spark and Apache Hadoop CompatibilityBoth Spark and Hadoop MapReduce are compatible with each other. Moreover, Spark shares every Hadoop MapReduce compatibility for data sources, file formats, and business intelligence tools via JDBC and ODBC.Data ProcessingHadoop MapReduce is a batch-processing engine. So how does it work? Well, it works in sequential steps.Step 1: Reads data from the clusterStep 2: Performs its operation on the dataStep 3: Writes the results back to the clusterStep 4: Reads updated data from the clusterStep 5: Performs the next data operationStep 6: Writes those results back to the clusterStep 7: Repeat.Spark performs in a similar manner, but the process doesn’t go on. It includes a single step and then to memory.Step 1: Reads data from the clusterStep 2: Performs its operation on the dataStep 3: Writes it back to the cluster.Moreover, Spark has GraphX, its own graph computation library. GraphX presents the same data as graphs and collections. Users have the option to use Resilient Distributed Datasets (RDDs) to transform and join graphs. This will be further addressed below in the Fault Tolerance section.Fault ToleranceThere are two different ways in which Hadoop MapReduce and Spark resolve the fault tolerance issue. Hadoop MapReduce utilises nodes like TaskTrackers to keep the JobTracker ticking. On the process being interrupted, the JobTracker reassigns every pending and in-progress operation to another TaskTracker. Although this process effectively provides fault tolerance, the completion times might get majorly affected even for operations having just a single failure.Spark, in this case, applies Resilient Distributed Datasets (RDDs), fault-tolerant collections of elements that can be operated side by side. References can be provided by RDDs in the form of datasets in an external storage system like shared filesystems, HDFS, HBase, or whatever available data source. This results in allowing a Hadoop InputFormat and Spark can create RDDs from every storage source that is backed by Hadoop. That covers local filesystems or one of those listed earlier.Below-mentioned is five main properties that an RDD possesses:A list of partitionsA function for computing each splitA list of dependencies on other RDDsA Partitioner for key-value RDDs by choice (provided that the RDD is hash-partitioned)Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)The persistence of RDDs to cache a dataset in memory across operations enables the speeding up of future actions by possibly ten folds. The cache of Spark is fault-tolerant, it will recomputed automatically by making use of the original transformations provided any partition of an RDD is lost.ScalabilityIn terms of scaling up, both Hadoop MapReduce and Spark are on equal terms in using the HDFS. Reports say that Yahoo holds a 42,000 node Hadoop cluster with no bounds while the most comprehensive Spark cluster holds 8,000 nodes. However, in order to support output expectations, the cluster sizes are expected to grow along with that of big data.SecurityKerberos authentication, considered to be quite hectic to manage is supported by Hadoop. Nevertheless, companies have been assisted by third-party vendors to leverage Active Directory Kerberos and LDAP for authentication and also allow data encrypt for in-flight and data at rest. Access control lists (ACLs) a traditional file permissions model are supported by Hadoop while it provides Service Level Authorization for user control in job submission, resulting in clients having the right permissions without any fail.For Spark though, it presently offers somewhat inadequate security as it provides authentication via shared secret (password authentication). However, if the user runs Spark on HDFS, then it can utilise HDFS ACLs and file-level permissions. Moreover, running Spark on YARN will enable the latter to have the capacity of using Kerberos authentication. That is the security takeaway from using Spark.  ConclusionApache Spark and Apache Hadoop form the perfect combination for business applications. Where Hadoop MapReduce has been a revelation in the big data market for businesses requiring huge datasets to be brought under control by commodity systems, Apache Spark’s speed and comparative ease of use compliments the low-cost operation involving Hadoop MapReduce.Like we discussed at the beginning of this article that neither of these two can replace one another, Spark and Hadoop form a lethal and effective symbiotic partnership. While Hadoop has features like a distributed file system that Spark does not have, the latter presents real-time, in-memory processing for the required data sets. Both Hadoop and Spark form the perfect combination for the ideal big data scenario. Rest assured, in this situation, both working in the same team is what goes in favour of big data professionals.You would be interested to know that Knowledgehut offers world-class training for Apache Spark and Hadoop. Feel free to check these courses to enhance your knowledge about both Apache Spark and Hadoop.
Rated 4.5/5 based on 12 customer reviews

Apache Spark Vs Hadoop - Head to Head Comparison

7K
Apache Spark Vs Hadoop - Head to Head Comparison

Over the past few years, data science has been one of the most sought-after multidisciplinary fields in the world today. It has established itself as an essential component of numerous industries such as marketing optimisation, risk management, marketing analytics. fraud detection, agriculture, etc. Understandably, this has lead to increasing demand for resorting to different approaches to data.

When we talk about Apache Spark and Hadoop, it is really difficult to compare them with each other. We should be aware that both possess important features in the world of data science and big data. Hadoop excels over Apache Spark in some business applications, but when processing speed and ease of use is taken into account, Apache Spark has its own advantages that make it unique. The most important thing to note is, neither of these two can replace each other. However, since they are compatible with each other, they can be used together to produce very effective results for many big data applications.

To analyse how important these two platforms are, there is a set of parameters with which we can discuss their efficiencies such as performance, ease of use, cost, data processing, compatibility, fault tolerance, scalability, and security. In this article, we will talk about Apache Spark and Hadoop individually for a bit, followed by stressing these parameters to better understand their significance in data science and big data.

What is Hadoop?

Hadoop

Hadoop, also known as Apache Hadoop, is a project formed by Apache.org that includes a software library and a framework that enables the usage of simple programming models to distributed processing of large data sets (big data) across computer clusters. Hadoop is quite efficient in scaling up from single computer systems to a lot of commodity system, offering substantial local storage. Due to this, Hadoop is considered as an omnipresent heavyweight in the big data analytics space. 

There are modules that work together to form the Hadoop framework. Here are the main Hadoop framework modules:

  • Hadoop Common
  • Hadoop Distributed File System (HDFS)
  • Hadoop YARN
  • Hadoop MapReduce

Hadoop’s core is based on the above four modules followed by many others like Ambari, Avro, Cassandra, Hive, Pig, Oozie, Flume, and Sqoop. These are responsible for improving and extending Hadoop’s power to big data applications and large data set processing.

Hadoop is utilised by numerous companies using big data sets and analytics and is the de facto model for big data applications. Initially, it was designed to take care of crawling and searching billions of web pages and collecting their information into a database, This resulted in Hadoop Distributed File System (HDFS), a distributed file system designed to run on commodity hardware and Hadoop MapReduce, a processing technique and a program model for distributed computing based on java.

Hadoop comes handy when companies find data sets too large and complex to not being able to process the information in reasonably sufficient time. Since crawling and searching the web are text-based tasks, Hadoop MapReduce comes in handy as it is an exceptional text processing engine.

An Overview of Apache Spark

Overview of Apache Spark

An open-source distributed general-purpose cluster-computing framework, Apache Spark is considered as a fast and general engine for large-scale data processing. Compared to heavyweight Hadoop’s Big Data framework, Spark is very lightweight and faster by nearly 100 times. Although the facts say so, in fact, Spark runs up to 10 times faster on disk. Apart from that, it can perform batch processing but it really is good at streaming workloads, interactive queries, and machine-based learning.

✓Streaming workloads

✓Interactive queries

✓Machine-based learning.

Spark engine’s real-time data processing capability has a clear edge over Hadoop MapReduce’s disk-bound, batch processing one. Not only is Spark compatible with Hadoop and its modules, but it is also listed as a module on Hadoop’s project page. And because Spark can run in Hadoop clusters through YARN (Yet Another Resource Negotiator), it has its own page and a standalone mode. It can run as a Hadoop module and as a standalone solution which makes it difficult to make direct comparisons.

Despite these facts, Spark is expected to diverge and might even replace Hadoop, especially in terms of faster access to processed data. Spark’s cluster computing feature enables it to compete with only Hadoop MapReduce and not the entire Hadoop ecosystem. That is why it can use HDFS despite not having its own distributed file system. To be concise, Hadoop MapReduce uses persistent storage whereas Spark uses Resilient Distributed Datasets (RDDs). What is RDD? This will be stressed in the Fault Tolerance section.

The differences between Apache Spark and Hadoop

Let us have a look at the parameters using which we can compare the features of Apache Spark with Hadoop.

Apache Spark vs Hadoop in a nutshell

Apache Spark
Parameters
Hadoop
Processes everything in memoryPerformance-wiseHadoop MapReduce uses batch processing
Has user-friendly APIs for multiple programming languagesEase of UseHas add-ons such as Hive and Pig
Spark systems cost moreCostsHadoop MapReduce systems cost lesser
Shares every Hadoop MapReduce compatibilityCompatibilityCompliments Apache Spark seamlessly
Has GraphX, its own graph computation libraryData ProcessingHadoop MapReduce operates in sequential steps
Spark uses Resilient Distributed Datasets (RDDs)Fault ToleranceUtilises TaskTrackers to keep the JobTracker ticking
Comparatively lesser scalabilityScalabilityLarge Scalability
Provides authentication via shared secret (password authentication)SecuritySupports Kerberos authentication
  • Performance-wise

Spark is definitely faster when compared to Hadoop MapReduce. However, they cannot be compared because they perform processing in different styles. Spark is way faster because it processes everything in memory, even using disk for data that does not all fit into memory. 

The in-memory processing of Spark performs near real-time analytics for data from machine learning, log monitoring, marketing campaigns, Internet of Things sensors, security analytics, and social media sites. Hadoop MapReduce, on the other hand, utilises the batch-processing method so it understandably was never created for mesmerising speed. As a matter of fact, it was initially created to continuously gather information from websites during the times when data in or near real-time were not required.

  • Ease of Use

Spark does not only have a good reputation for its excellent performance, but it is also relatively easy to use along with providing additional support for languages like user-friendly APIs for Scala, Java, Python, and Spark SQL. Since Spark SQL is quite comparable to SQL 92, the user requires no additional knowledge to use it.

Supported Languages:

  • APIs for Scala
  • Java
  • Python
  • Spark SQL.

Ease of Use

Additionally, Spark is armed with an interactive mode to allow developers and users get instant feedback for questions and other actions. Hadoop MapReduce makes up for the lack of any interactive mode with add-ons like Hive and Pig, thus easing the workflow of Hadoop MapReduce.

  • Costs

Apache Spark and Apache Hadoop MapReduce are both free open-source software.

However, because Hadoop MapReduce’s processing is disk-based, it utilises standard volumes of memory. This results in companies buying faster disks with a lot of disk space to run Hadoop MapReduce. In stark contrast to this, Spark requires a lot of memory but compensates by settling with a standard amount of disk space running at standard speeds.

  • Apache Spark and Apache Hadoop Compatibility

Both Spark and Hadoop MapReduce are compatible with each other. Moreover, Spark shares every Hadoop MapReduce compatibility for data sources, file formats, and business intelligence tools via JDBC and ODBC.

Apache Spark and Apache Hadoop Compatibility

  • Data Processing

Hadoop MapReduce is a batch-processing engine. So how does it work? Well, it works in sequential steps.

Step 1: Reads data from the cluster

Step 2: Performs its operation on the data

Step 3: Writes the results back to the cluster

Step 4: Reads updated data from the cluster

Step 5: Performs the next data operation

Step 6: Writes those results back to the cluster

Step 7: Repeat.

Spark performs in a similar manner, but the process doesn’t go on. It includes a single step and then to memory.

Step 1: Reads data from the cluster

Step 2: Performs its operation on the data

Step 3: Writes it back to the cluster.

Moreover, Spark has GraphX, its own graph computation library. GraphX presents the same data as graphs and collections. Users have the option to use Resilient Distributed Datasets (RDDs) to transform and join graphs. This will be further addressed below in the Fault Tolerance section.

  • Fault Tolerance

There are two different ways in which Hadoop MapReduce and Spark resolve the fault tolerance issue. Hadoop MapReduce utilises nodes like TaskTrackers to keep the JobTracker ticking. On the process being interrupted, the JobTracker reassigns every pending and in-progress operation to another TaskTracker. Although this process effectively provides fault tolerance, the completion times might get majorly affected even for operations having just a single failure.

Spark, in this case, applies Resilient Distributed Datasets (RDDs), fault-tolerant collections of elements that can be operated side by side. References can be provided by RDDs in the form of datasets in an external storage system like shared filesystems, HDFS, HBase, or whatever available data source. This results in allowing a Hadoop InputFormat and Spark can create RDDs from every storage source that is backed by Hadoop. That covers local filesystems or one of those listed earlier.

Below-mentioned is five main properties that an RDD possesses:

  1. A list of partitions
  2. A function for computing each split
  3. A list of dependencies on other RDDs
  4. A Partitioner for key-value RDDs by choice (provided that the RDD is hash-partitioned)
  5. Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)

The persistence of RDDs to cache a dataset in memory across operations enables the speeding up of future actions by possibly ten folds. The cache of Spark is fault-tolerant, it will recomputed automatically by making use of the original transformations provided any partition of an RDD is lost.

  • Scalability

In terms of scaling up, both Hadoop MapReduce and Spark are on equal terms in using the HDFS. Reports say that Yahoo holds a 42,000 node Hadoop cluster with no bounds while the most comprehensive Spark cluster holds 8,000 nodes. However, in order to support output expectations, the cluster sizes are expected to grow along with that of big data.

  • Security

Kerberos authentication, considered to be quite hectic to manage is supported by Hadoop. Nevertheless, companies have been assisted by third-party vendors to leverage Active Directory Kerberos and LDAP for authentication and also allow data encrypt for in-flight and data at rest. Access control lists (ACLs) a traditional file permissions model are supported by Hadoop while it provides Service Level Authorization for user control in job submission, resulting in clients having the right permissions without any fail.

For Spark though, it presently offers somewhat inadequate security as it provides authentication via shared secret (password authentication). However, if the user runs Spark on HDFS, then it can utilise HDFS ACLs and file-level permissions. Moreover, running Spark on YARN will enable the latter to have the capacity of using Kerberos authentication. That is the security takeaway from using Spark.  

Conclusion

Apache Spark and Apache Hadoop form the perfect combination for business applications. Where Hadoop MapReduce has been a revelation in the big data market for businesses requiring huge datasets to be brought under control by commodity systems, Apache Spark’s speed and comparative ease of use compliments the low-cost operation involving Hadoop MapReduce.

Like we discussed at the beginning of this article that neither of these two can replace one another, Spark and Hadoop form a lethal and effective symbiotic partnership. While Hadoop has features like a distributed file system that Spark does not have, the latter presents real-time, in-memory processing for the required data sets. Both Hadoop and Spark form the perfect combination for the ideal big data scenario. Rest assured, in this situation, both working in the same team is what goes in favour of big data professionals.

You would be interested to know that Knowledgehut offers world-class training for Apache Spark and Hadoop. Feel free to check these courses to enhance your knowledge about both Apache Spark and Hadoop.

KnowledgeHut

KnowledgeHut

Author

KnowledgeHut is a fast growing Management Consulting and Training firm that is a source of Intelligent Information support for businesses and professionals across the globe.


Website : https://www.knowledgehut.com/

Join the Discussion

Your email address will not be published. Required fields are marked *

Suggested Blogs

Apache Spark Vs Apache Storm - Head To Head Comparison

In today’s world, the need for real-time data streaming is growing exponentially due to the increase in real-time data. With streaming technologies leading the world of Big Data, it might be tough for the users to choose the appropriate real-time streaming platform. Two of the most popular real-time technologies that might consider for opting are Apache Spark and Apache Storm. One major key difference between the frameworks Spark and Storm is that Spark performs Data-Parallel computations, whereas Storm occupies Task-Parallel computations. Read along to know more differences between Apache Spark and Apache Storm, and understand which one is better to adopt on the basis of different features. Comparison Table: Apache Spark Vs. Apache StormSr. NoParameterApache SparkApache Storm1.Processing  ModelBatch ProcessingMicro-batch processing2.Programming LanguageSupports lesser languages like Java, Scala.Support smultiple languages, such as Scala, Java, Clojure.3.Stream SourcesHDFSSpout4.MessagingAkka, NettyZeroMQ, Netty5.Resource ManagementYarn and Meson are responsible.Yarn and Mesos are responsible.6.Low LatencyHigher latency as compared to SparkBetter latency with lesser constraints7.Stream PrimitivesDStreamTuple, Partition8.Development CostSame code can be used for batch and stream processing.Same code cannot be used for batch and stream processing.9.State ManagementSupports State ManagementSupports State Management as well10.Message Delivery GuaranteesSupports one message processing mode: ‘at least once’.Supports three message processing mode: ‘at least once’, ‘at most once’, ‘exactly once’.11.Fault ToleranceIf a process fails, Spark restarts workers via resource managers. (YARN, Mesos)If a process fails, the supervisor process starts automatically.12.Throughput100k records per node per second10k records per node per second13.PersistenceMapStatePer RDD14.ProvisioningBasic monitoring using GangliaApache Ambaripache Spark: Apache Spark is a general-purpose, lighting fast, cluster-computing technology framework, used for fast computation on large-scale data processing. It can manage both batch and real-time analytics and data processing workloads.  Spark was developed at UC Berkeley in the year 2009. Apache Storm:Apache Storm is an open-source, scalable fault-tolerant, and real-time stream processing computation system. It is a framework for real-time distributed data processing, which focuses on stream processing or event processing. It can be used with any programming language and can be integrated using any queuing or database technology.  Apache Storm was developed by a team led by Nathan Marz at BackType Labs. Apache Spark Vs. Apache StormProcessing Model: Apache Storm supports micro-batch processing, while Apache Spark supports batch processing. Programming Language:Storm applications can be created using multiple languages like Java, Scala and Clojure, while Spark applications can be created using Java and Scala.Stream Sources:For Storm, the source of stream processing is Spout, while that for Spark is HDFS. Messaging:Storm uses ZeroMQ and Netty as its messaging layer while Spark is using a combination of Nettu and Akka for distributing the messages throughout the executors. Resource Management:Yarn and Meson are responsible for resource management in Spark, while Yarn and Mesos are responsible for resource management in Storm. Low Latency: Spark provides higher latency as compared to Apache Storm, whereas Storm can provide better latency with fewer restrictions.Stream Primitives:Spark provides with stream transforming operators which transform DStream into another, while Storm provides with various primitives which perform tuple level of processing at the stream level (functions, filters). Development Cost:It is possible for Spark to use the same code base for both stream processing and batch processing. Whereas for Storm, the same code base cannot be used for both stream processing and batch processing.  State Management: The changing and maintaining state in Apache Spark can be updated via UpdateStateByKey, but no pluggable strategy can be applied in the external system for the implementation of state. Whereas Storm does not provide any framework for the storage of any intervening bolt output as a state. Hence, each application has to create a state for itself whenever required. Message Delivery Guarantees (Handling the message level failures):Apache Spark supports only one message processing mode, viz, ‘at least once’, whereas Storm supports three message processing modes, viz, ‘at least once’ (Tuples are processed at least one time, but can be processed more than once), ‘at most once’  and ‘exactly once’ (T^uples are processed at least once). Storm’s reliability mechanisms are scalable, distributed and fault-tolerant. Fault-Tolerant:Apache Spark and Apache Storm, both are fault tolerant to nearly the same extent. If a process fails in Apache Storm, then the supervisor process will restart it automatically, as the state management is managed by Zookeeper, while Spark restarts its workers with the help of resource managers, who may be Mesos, YARN or its separate manager.Ease of Development: In the case of Storm, there are effective and easy to use APIs which show that the nature of topology is DAG. The Storm tuples are written dynamically. In the case of Spark, it consists of Java and Scala APIs with practical programming, making topology code a bit difficult to understand. But since the API documentation and samples are easily available for the developers, it is now easier. Summing Up: Apache Spark Vs Apache StormApache Storm and Apache Spark both offer great solutions to solve the transformation problems and streaming ingestions. Moreover, both can be a part of a Hadoop cluster to process data. While Storm acts as a solution for real-time stream processing, developers might find it to be quite complex to develop applications due to its limited resources. The industry is always on a lookout for a generalized solution, which has the ability to solve all types of problems, such as Batch processing, interactive processing, iterative processing and stream processing. Keeping all these points in mind, this is where Apache Spark steals the limelight as it is mostly considered as a general-purpose computation engine, making it a highly demanding tool by IT professionals. It can handle various types of problems and provides a flexible environment to in. Moreover, developers find it to be easy and are able to integrate it well with Hadoop. 
Rated 4.5/5 based on 19 customer reviews
6591
Apache Spark Vs Apache Storm - Head To Head Compar...

In today’s world, the need for real-time data st... Read More

Apache Spark Vs MapReduce

Why we need Big Data frameworksBig data is primarily defined by the volume of a data set. Big data sets are generally huge – measuring tens of terabytes – and sometimes crossing the threshold of petabytes. It is surprising to know how much data is generated every minute. As estimated by DOMO:Over 2.5 quintillion bytes of data are created every single day, and it’s only going to grow from there. By 2020, it’s estimated that 1.7MB of data will be created every second for every person on earth.You can read DOMO's full report, including industry-specific breakdowns, here.To store and process even only a fraction of this amount of data, we need Big Data frameworks as the traditional Databases would not be able to store so much of data nor traditional processing systems would be able to process this data quickly. Here comes the frameworks like Apache Spark and MapReduce to our rescue and help us to get deep insights into this huge amount of structured, unstructured and semi-structured data and make more sense out of it.Market Demands for Spark and MapReduceApache Spark was originally developed in 2009 at UC Berkeley by the team who later founded Databricks. Since its launch Spark has seen rapid adoption and growth. Most of the cutting-edge technology organizations like Netflix, Apple, Facebook, Uber have massive Spark clusters for data processing and analytics. The demand for Spark is increasing at a very fast pace. According to marketanalysis.com report forecast, the global Apache Spark market will grow at a CAGR of 67% between 2019 and 2022. The global Spark market revenue is rapidly expanding and may grow up $4.2 billion by 2022, with a cumulative market valued at $9.2 billion (2019 – 2022).MapReduce has been there for a little longer after being developed in 2006 and gained industry acceptance during the initial years. But at last, 5 years or so with Apache Spark gaining more ground, demand for MapReduce as the processing engine has reduced. But, it cannot be said in black and white that MapReduce will be completely replaced by Apache Spark in the coming years. Both the technologies have their own pros and cons as we will see them below. One solution cannot fit at all the places, so MapReduce will have its own takers depending on the problem to be solved.Also, Spark and MapReduce do complement each other on many occasions.Both these technologies have made inroads in all walks of common man’s life. You name the industry and its there. Be it telecommunication, e-commerce, banking, insurance, healthcare, medicine, agriculture, biotechnology, etc.What is Spark?As per Apache, “Apache Spark is a unified analytics engine for large-scale data processing”. Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.Spark, instead of just “map” and “reduce” functions, defines a large set of operations called transformations and actions for the developers and which are ultimately transformed to map/reduce by the spark execution engine and these operations are arbitrarily combined for highly optimized performance.Spark is developed in Scala language and it can run on Hadoop in standalone mode using its own default resource manager as well as in Cluster mode using YARN or Mesos resource manager. It is not mandatory to use Hadoop for Spark, it can be used with S3 or Cassandra also. But, in the majority of the cases, Hadoop is the best fit as Spark’s data storage layer.Features of SparkSpeed: Spark enables applications running on Hadoop to run up to 100x faster in memory and up to 10x faster on disk. Spark achieves this by minimising disk read/write operations for intermediate results and storing in memory and perform disk operations only when essential. Spark achieves this using DAG, query optimizer and highly optimized physical execution engine.Fault Tolerance: Apache Spark achieves fault tolerance using spark abstraction layer called RDD (Resilient Distributed Datasets), which are designed to handle worker node failure.Lazy Evaluation: All the processing(transformations) on Spark RDD/Datasets are lazily evaluated, i.e. the output RDD/datasets are not available right away after transformation but will be available only when an action is performed.Dynamic nature: Spark offers over 80 high-level operators that make it easy to build parallel apps.Multiple Language Support: Spark provides multiple programming language support and you can use it interactively from the Scala, Python, R, and SQL shells.Reusability: Spark code can be used for batch-processing, joining streaming data against historical data as well as run ad-hoc queries on streaming state.Machine Learning: Apache Spark comes with out of the box support for machine learning called MLib which can be used for complex, predictive data analytics.Graph Processing: GraphX is Apache Spark's API for graphs and graph-parallel computation. You can view the same data as both graphs and collections, transform and join graphs with RDDs efficiently, and write custom iterative graph algorithms using the Pregel API.Real-Time Stream Processing: Spark Streaming brings Apache Spark's language-integrated API to stream processing, letting you write streaming jobs the same way you write batch jobs.Where is Spark usually used?Spark is used by 1000+ organizations in Production. Many of these organizations are known to run Spark clusters of 1000+ nodes. In terms of data size, Spark has been shown to work well up to petabytes. It has been used to sort 100 TB of data 3X faster than Hadoop MapReduce (which sorted 100 TB of data in 23 min, using 2100 machines) using 10X fever machines, winning the 2014 Daytona GraySort Benchmark, as well as to sort 1 PB. Several production workloads use Spark to do ETL and data analysis on PBs of data. Below are some examples where Spark is used across industries:AsiaInfo: Uses Spark Core, Streaming, MLlib and Graphx and Hadoop to build cost-effective data centre solution for our customers in the telecom industry as well as other industrial sectors.Atp: Predictive models and learning algorithms to improve the relevance of programmatic marketing.Credit Karma: Creates personalized experiences using SparkeBay Inc: Using Spark core for log transaction aggregation and analyticsKelkoo: Using Spark Core, SQL, and Streaming. Product recommendations, BI and analytics, real-time malicious activity filtering, and data mining.More examples can be found on Apache’s  Powered By pageSpark Example in Scala (Spark shell can be used for this)// “sc” is a “Spark context” – this transforms the file into an RDD val textFile = sc.textFile("data.txt") // Return number of items (lines) in this RDD; count() is an action textFile.count() // Demo filtering.  Filter is a transform.  By itself this does no real work val linesWithSpark = textFile.filter(line => line.contains("Spark")) // Demo chaining – how many lines contain “Spark”?  count() is an action. textFile.filter(line => line.contains("Spark")).count() // Length of line with most words.  Reduce is an action. textFile.map(line => line.split(" ").size).reduce((a, b) => if (a > b) a else b) // Word count – traditional map-reduce.  collect() is an action val word Counts = text File.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b) word Counts.collect()Sample Spark Transformationsmap(func): Return a new distributed dataset formed by passing each element of the source through a function func.filter(func): Return a new dataset formed by selecting those elements of the source on which func returns trueunion(other Dataset): Return a new dataset that contains the union of the elements in the source dataset and the argument.Sample Spark Actionsreduce(func): Aggregate the elements of the dataset using a function func (which takes two arguments and returns one). The function should be commutative and associative so that it can be computed correctly in parallel.collect(): Return all the elements of the dataset as an array at the driver program. This is usually useful after a filter or other operation that returns a sufficiently small subset of the data.count(): Return the number of elements in the dataset.The data is referred from the RDD Programming guide.What is MapReduce?MapReduce is a programming model for processing and generating large data sets with a parallel, distributed algorithm on a cluster.Programmers have been writing parallel programs for a long time in different languages like C++, Java, C#, Python. But, they have their own nuances and maintaining these, is the programmer's responsibility. There are chances of application crashing, performance hit, incorrect results. Also, such systems if grows very large is not very fault tolerant or difficult to maintain.MapReduce has simplified all these. Fault tolerance, parallel execution, resources management is all responsibility of the Resource manager and the framework. Programmers have to only concentrate on business logic by writing only map and reduce functions.Brief Description of MapReduce ArchitectureA MapReduce application has broadly two functions called map and reduce.Map: Mapper process takes input as key/value pair, processes them i.e. performs some computation and then produces intermediate results as key/value pairsi.e. map(k1,v1) ---> list(k2,v2)Reduce: Reducer process receives an intermediate key and a set of values in sorted order. It processes these and generates output key/value pairs by grouping values for each key.i.e. reduce(k2, list(v2)) ---> list(v3)Can also define an option function “Combiner” (to optimize bandwidth)If defined, runs after Mapper & before Reducer on every node that has run a map taskCombiner receives as input all data emitted by the Mapper instances on a given nodeCombiner output sent to the Reducers, instead of the output from the MappersIs a "mini-reduce" process which operates only on data generated by one machineHow does MapReduce work?MapReduce is usually applied to huge datasets. A MapReduce job splits the input data into smaller independent chunks called partitions and then processes them independently using map tasks and reduce tasks. Below is an example.MapReduce Word Count (Pseudocode)map(String input_key, String input_value): // input_key: document name // input_value: document contents for each word w in input_value: EmitIntermediate(w, "1"); reduce(String output_key, Iterator intermediate_values): // output_key: a word // output_values: a list of counts int result = 0; for each v in intermediate_values: result += ParseInt(v); Emit(AsString(result));MapReduceApply a function to all the elements of Listlist1=[1,2,3,4,5]; square x = x * x list2=Map square(list1) print list2 -> [1,4,9,16,25]0Combine all the elements of list for a summarylist1 = [1,2,3,4,5]; A = reduce (+) list1 Print A -> 15Apache Spark vs MapReduceAfter getting off hangover how Apache Spark and MapReduce works, we need to understand how these two technologies compare with each other, what are their pros and cons, so as to get a clear understanding which technology fits our use case.As we can see, MapReduce involves at least 4 disk operations whereas Spark only involves 2 disk operations. This is one reason for Spark to be much faster than MapReduce. Spark also caches intermediate data which can be used in further iterations helping Spark improve its performance further. The more iterative the process the better is the Spark performance due to in-memory processing and caching. This is where MapReduce performance not as good as Spark due to disk read/write operations for every iteration.Let’s see a comparison between Spark and MapReduce on different other parameters to understand where to use Spark and where to use MapReduceAttributesMapReduceApache SparkSpeed/PerformanceMapReduce is designed for batch processing and is not as fast as Spark. It is used for gathering data from multiple sources and process it once and store in a distributed data store like HDFS. It is best suited where memory is limited and processing data size is so big that it would not fit in the available memory.Spark is 10-100 times faster because of in-memory processing and its caching mechanism. It can deliver near real-time analytics. It is used in Credit Card Processing, Fraud detection, Machine learning and data analytics, IoT sensors etcCostAs it is part of Apache Open Source there is no software cost.Hardware cost is less in MapReduce as it works with smaller memory(RAM) as compared to Spark. Even commodity hardware is sufficient.Spark also is Apache Open Source so no license cost.Hardware cost is more than MapReduce as even though Spark can work on commodity hardware it needs a lot more memory(RAM) as compared to MapReduce since it should be able to fit all the data in Memory for optimal performance. Cluster needs little high-end commodity hardware with lots of RAM else performance gets hitEase of UseMapReduce is a bit complex to write. MapReduce is written in Java and the APIs are a bit complex to code for new programmers, so there is a steep learning curve involved. The Pig has SQL like syntax and it is easier for SQL developers to get onboard easily. Also, there is no interactive mode available in MapReduceSpark has APIs in Scala, Java, Python, R for all basic transformations and actions. It also has rich Spark SQL APIs for SQL savvy developers and it covers most of the SQL functions and is adding more functions with each new release. Also, Spark has scope for writing User Defined Analytical Functions and Functions (UDF/UDAF) for anyone who would like to have custom functions.CompatibilityMapReduce is also compatible with all data sources and file formats Hadoop supports. But MapReduce needs another Scheduler like YARN or Mesos to run, it does not have any inbuilt Scheduler like Spark’s default/standalone scheduler.Apache Spark can in standalone mode using default scheduler. It can also run on YARN or Mesos. It can run on-premise or on the cloud. Spark supports most of the data formats like parquet, Avro, ORC, JSON, etc. It also supports multiple languages and has APIs for Java, Scala, Python, R.Data ProcessingMapReduce can only be used for batch processing where throughput is more important and latency can be compromised.Spark supports Batch as well as Stream processing, so fits both use cases and can be used for Lambda design where applications need both Speed layer and slower layer/data processing layerSecurityMapReduce has more security features.MapReduce can enjoy all the Hadoop security benefits and integrate with Hadoop security projects, like Knox Gateway and Sentry.Spark is a bit bare at the moment. Spark currently supports authentication via a shared secret. Spark can integrate with HDFS and it can use HDFS ACLs and file-level permissions. Spark can also run on YARN leveraging the capability of Kerberos.Fault ToleranceMapReduce uses replication for fault tolerance. If any slave daemon fails, master daemons reschedule all pending and in-progress operations to another slave. This method is effective, but it can significantly increase the completion times for operations with a single failure alsoIn Spark, RDDs are the building blocks and Spark also uses it RDDs and DAG for fault tolerance. If an RDD is lost, it will automatically be recomputed by using the original transformations.LatencyMapReduce has high latencySpark provides low latency performanceInteractive ModeMapReduce does not have any interactive mode of operation.Spark can be used interactively also for data processing. It has out-of-the-box support for spark shell for scala/python/RMachine Learning/Graph ProcessingNo support for these. A mahout has to be used for MLSpark has dedicated modules for ML and Graph processingBoth these technologies MapReduce and Spark have pros and cons:MapReduce is best suited for Analysis of archived data where data size is huge and it is not going to fit in memory, and if the instant results and intermediate solutions are not required. MapReduce also scales very well and the cluster can be horizontally scaled with ease using commodity machines.Offline Analytics is a good fit for MapReduce like Top Products per month, Unique clicks per banner.MapReduce is also suited for Web Crawling as well as Crawling tweets at scale and NLP like Sentiment Analysis.Another use case for MapReduce is de-duplicating data from social networking sites, job sites and other similar sites.MapReduce is also heavily used in Data mining for Generating the model and then classifying.Spark is fast and so can be used in Near Real Time data analysis.A lot of organizations are moving to Spark as their ETL processing layer from legacy ETL systems like Informatica. Spark as very good and optimized SQL processing module which fits the ETL requirements as it can read from multiple sources and can also write to many kinds of data sources.Spark can also handle Streaming data so its best suited for Lambda design.Most graph processing algorithms like page rank perform multiple iterations over the same data and this requires a message passing mechanism. Spark has great support for Graph processing using GraphX module.Almost all machine learning algorithms work iteratively. Spark has a built-in scalable machine learning library called MLlib which contains high-quality algorithms that leverage iterations and yields better results than one pass approximations sometimes used on MapReduce.Hadoop MapReduce is more mature as it has been there for a longer time and its support is also better in the open source community. It can be beneficial for really big data use case where memory is limited and data will not fit the RAM. Most of the time, Spark use case will involve Hadoop and other tools like Hive, Pig, Impala and so when these technologies complement each other it will be a win for both Spark and MapReduce.Conclusion:Hadoop Mapreduce is more mature as it has been there for a longer time and its support is also better in the open source community. It can be beneficial for really big data use case where memory is limited and data will not fit the RAM. Most of the time, Spark use case will involve Hadoop and other tools like Hive, Pig, Impala and so when these technologies complement each other it will be a win for both Spark and Mapreduce.
Rated 4.5/5 based on 1 customer reviews
8720
Apache Spark Vs MapReduce

Why we need Big Data frameworksBig data is primari... Read More

Best ways to learn Apache Spark

If you ask any industry expert what language should you learn for Big Data? You will get an obvious reply to learn Apache Spark. Apache Spark is widely considered as the future of the Big Data industry. Since Apache Spark has stepped into Big data market, it has gained a lot of recognition for itself. Today, most of the cutting-edge companies like Apple, Facebook, Netflix, and Uber, etc. have deployed Spark at massive scale. In this blog post, we will understand why one should learn Apache Spark? And several ways to learn it. Apache Spark is a powerful open-source framework for the processing of large datasets. It is the most successful projects in the Apache software foundation. Apache Spark basically designed for fast computation, also which runs faster than Hadoop. Apache Spark can collectively process huge amount of data present in clusters over multiple nodes. The main feature of Apache Spark is its in-memory cluster computing that increases the processing speed of an application.Why You Should Learn Apache SparkApache Spark has become the most popular unified analytics engine for Big Data and Machine Learning. Enterprises are widely utilizing Spark which in turn is increasing demand for Apache Spark developers. Apache Spark developers are the ones earning the highest salary. IT professionals can leverage this upcoming skill set gap by pursuing a certification in Apache Spark. A developer with expertise in Apache Spark skills can earn an average salary of $78K as per Payscale. It is the right time for you to learn Apache Spark as there is a very high demand for Spark developers chances of getting a job is high.Here are the reasons why you should learn Apache Spark today:In order to go with the growing demand for Apache SparkTo fulfill the demands for Spark developersTo get benefits of existing big data investmentsResources to learn ReactTo learn Spark, you can refer to Spark’s website. There are multiple resources you will find to learn Apache Spark, from books, blogs, online videos, courses, tutorials, etc. With these multiple resources available today, you might be in the dilemma of choosing the best resource, especially in this fast-paced and swiftly evolving industry.BooksCertificationsVideosTutorials, Blogs, and TalksHands-on Exercises 1. BooksWhen was the last time you read a book? Do you have reading habits? If not, it’s the time to read the books. Reading has a significant number of benefits. Those aren’t fans of books might miss out the importance of Apache Spark. To learn Apache Spark, you can skim through the best Apache Spark books given below.Apache Spark in 24 hours is a perfect book for beginners which comprises 592 pages covering various topics. An excellent book to learn in a very short span of time. Apart from this, there are also books which will help you master.Here is the list of top books to learn Apache Spark:Learning Spark by Matei Zaharia, Patrick Wendell, Andy Konwinski, Holden KarauAdvanced Analytics with Spark by Sandy Ryza, Uri Laserson, Sean Owen and Josh WillsMastering Apache Spark by Mike FramptonSpark: The Definitive Guide – Big Data Processing Made SimpleSpark GraphX in ActionBig Data Analytics with SparkThese are the various Apache Spark books meant for you to learn. These books include for beginners and others for the advanced level professionals.2. Apache Spark Training and CertificationsOne more way to learn Apache Spark is through taking up training. Apache Spark Training will boost your knowledge and also help you learn from experience. You will be certified once you are done with training. Getting this certification will help you stand out of the crowd. You will also gain hands-on skills and knowledge in developing Spark applications through industry-based real-time projects.3. Videos:Videos are really good resources to help you learn Apache Spark. Following are the few videos will help you understand Apache Spark.Overview of SparkIntro to Spark - Brian ClapperAdvanced Spark Analytics - Sameer FarooquiSpark Summit VideosVideos from Spark Summit 2014, San Francisco, June 30 - July 2, 2013Full agenda with links to all videos and slidesTraining videos and slidesVideos from Spark Summit 2013, San Francisco, Dec 2-3-2013Full agenda with links to all videos and slidesYouTube playist of all KeynotesYouTube playist of Track A (Spark Applications)YouTube playist of Track B (Spark Deployment, Scheduling & Perf, Related projects)YouTube playist of the Training Day (i.e. the 2nd day of the summit)You can learn more on Apache Spark YouTube Channel for videos from Spark events. 4. Tutorials, Blogs, and TalksUsing Parquet and Scrooge with Spark — Scala-friendly Parquet and Avro usage tutorial from Ooyala's Evan ChanUsing Spark with MongoDB — by Sampo Niskanen from WellmoSpark Summit 2013 — contained 30 talks about Spark use cases, available as slides and videosA Powerful Big Data Trio: Spark, Parquet and Avro — Using Parquet in Spark by Matt MassieReal-time Analytics with Cassandra, Spark, and Shark — Presentation by Evan Chan from Ooyala at 2013 Cassandra SummitRun Spark and Shark on Amazon Elastic MapReduce — Article by Amazon Elastic MapReduce team member Parviz DeyhimSpark, an alternative for fast data analytics — IBM Developer Works article by M. Tim Jones 5. Hands-on ExercisesHands-on exercises from Spark Summit 2014 - These exercises will guide you to install Spark on your laptop and learn basic concepts.Hands-on exercises from Spark Summit 2013 - These exercises will help you launch a small EC2 cluster, load a dataset, and query it with Spark, Spark Streaming, and MLlib.So these were the best resources to learn Apache Spark. Hope you found what you were looking for. Wish you a Happy Learning!
Rated 4.5/5 based on 19 customer reviews
8593
Best ways to learn Apache Spark

If you ask any industry expert what language shoul... Read More