Search

Trending blog posts

How Big is ‘Big Data’, Anyway?

When I got introduced to the data-world with my first corporate induction training, about 10 years ago. I was then still processing the difference between Data and Information. The following helped me understand the same:Data: It is raw information (unprocessed facts and figures) without any context for e.g. Number 20Information: structured Data grouped together which can have interpretation. E.g $20 for a toy.Knowledge: combination of information, experience and insight that may benefit the individual for the organisation. E.g. $20 for a toy in Black Friday Sale in a mall.Wisdom: Knowledge becomes wisdom when one can assimilate and apply this knowledge to make the right decisions. E.g. One who wants to buy a toy will wait for the Black Friday Sale to get it at a cheaper price.By the time I started understanding above differences, ‘Big data’ term was already making it big and then the obvious question in mind was,” When to call ‘data’ à ‘ Big data’? “I then made an attempt to understand ‘how big is a data to be called  big data?’ and here, I have a big revelation to make, for all of you reading this article, that ‘Big Data’ is actually misleading term and it is irrelevant with “Bigness of data” but it is to be used in relevance. In fact, it is a term which needs to be understood, only in perspective.The simplest one I could find relevant is,  Big data is the data that cannot be stored with traditional storages, cannot be processed with traditional methods/ways and within a short period of time (and these references would still be valid as time advances.) but this is not textbook or only definition of big data. Interestingly, One who finds one set of data as big data can be traditional data for others so truly it cannot be bounded in words but loosely can be described through numerous examples. I am sure by the end of the article you will be able to answer the question for yourself. Let’s start.Do you know? - NASA researchers Michael Cox and David Ellsworth use the term “big data” for the first time to describe a familiar challenge in the 1990s supercomputers generating massive amounts of information - in Cox and Ellsworth’s case, simulations of airflow around aircraft - that cannot be processed and visualized.If you go through a  brief history of big data, you would know data which is not fitting into memory or disk was called ‘Big data problem’ back in 1997.As the years passed by innovations were on rising and disruptions were made so the data universe is growing all the time. Let’s understand a few widely available and stated statistics for ‘big data’ (Collected around 2017 or before) >>On average, people  send about 500 million tweets per day.Snapchat users share 527,760 photos in a minute Instagram users post 46,740 photos in a minute More than 120 professionals join LinkedIn in a minute Users watch 4,146,600 YouTube videos in a minuteThe average U.S. customer  uses 1.8 gigabytes of data per month on his or her cell phone plan.Amazon sells  600 items per second.On average, each person who uses email  receives 88 emails per day and send 34. That adds up to more than 200 billion emails each day.MasterCard processes  74 billion transactions per year.Commercial airlines  make about 5,800 flights per day.You might be interested to read through   Domo’s Data Never Sleeps 5.0 report, for the numbers generated every minute of the day.Understanding that the above stats are probably about 1.5-2 years older and data is ever-growing, it helps to establish the fact that ‘big data‘ is a moving target and…. In short,“Today’s big data is tomorrow’s small data.”Now that we have some knowledge about transactions/tweets/snaps in a day, Let’s also understand how much data, all these “One-minute Quickies” are generating. Let’s talk about some volumes too. Afterall volumes are one of the characteristics of big data but mind you, not only characteristic of big data. It is believed that, In a single day, the world produces 2.5 quintillion bytes (2.3 trillion gigabytes) of data, in layman's terms, this is the equivalent of everyone in the world downloading 60 episodes of Breaking Bad, in HD, 20 times! [Source:  VCloud 2012] and According to estimates, the volume of data worldwide doubles every 1.2 years.IDC predicts that the collective sum of the world's data will grow from 33 zettabytes this year to a 175ZB by 2025, for a compounded annual growth rate of 61 per cent. The 175ZB figure represents a 9 per cent increase over last year's prediction of data growth by 2025 – As per the report published in Dec’2018.But, do you know: how much would be 1 zettabyte of data? Let’s understand. One zettabyte is equal to one sextillion bytes or 1021 (1,000,000,000,000,000,000,000) bytes or, one zettabyte is roughly equal to a trillion gigabytes.Fun Fact: There is a legit term coined as The Zettabyte Era (Today’s Era).The Zettabyte Era can also be understood as an age of growth of all forms of digital data that exist in the world which includes the public Internet, but also all other forms of digital data such as stored data from security cameras or voice data from cell-phone calls.You must check out this  infographic by economywatch (taken from  SearchEngineJournal) to understand how much data zettabyte consists of, putting it into context with current data storage capabilities and usage.Today’s ‘big data’ is generated from majority 3 sources i.e. People Generated: Social media uploads, Mails etc. Machine Generated: M2M (machine to machine) interactions, IOT devices etc. Business Generated: Data generated and stored into today’s OLTPs, OLAPs, Data warehouses, data marts, reports, operational data throughout the enterprise/organization.Various analytics tools available in the market today, help in solving big data challenges by providing ways for storing this data, process this data and make valuable insights from this data.As we discussed, big data is moving target as time advances, it is also interesting to know even today, data which is not of huge size but is difficult to process and of relatively smaller volume would still be categorized as Big Data. For example, unstructured data in emails, from social media platforms, data which is required to process with real-time/near real-time etc. all the examples we have seen so far, all of it is big data.   But, It would be a mistake to assume that, Big Data only as data that is analyzed using Hadoop, Spark or another complex analytics platform. As big data is moving the target and it’s ever-growing, also with various disruptive sources of data are being introduced every day, to process this data newer tools would be invented, and hence big data cannot just remain a function of tools being used to analyze it. To conclude, as discussed at the starting of the article, it would still be appropriate and reasonable to say, this moving target of big data which would always be challenged for storage, processing methods and process it within a short period as well. So big data is a function of volume and/or time and/or storage and/or variety. It was fun and exciting to know what different aspects are hidden in ‘BIG DATA’ word and I thoroughly enjoyed solving this mystery.Did you enjoy solving it too?Do let us know how was experience through comments below.Happy Learning!!!
How Big is ‘Big Data’, Anyway?
Shruti
Rated 4.5/5 based on 23 customer reviews

How Big is ‘Big Data’, Anyway?

When I got introduced to the data-world with my first corporate induction training, about 10 years ago. I was then still processing the difference between Data and Information. The following helped me understand the same:Data: It is raw information (unprocessed facts and figures) without any context for e.g. Number 20Information: structured Data grouped together which can have interpretation. E.g $20 for a toy.Knowledge: combination of information, experience and insight that may benefit the individual for the organisation. E.g. $20 for a toy in Black Friday Sale in a mall.Wisdom: Knowledge becomes wisdom when one can assimilate and apply this knowledge to make the right decisions. E.g. One who wants to buy a toy will wait for the Black Friday Sale to get it at a cheaper price.By the time I started understanding above differences, ‘Big data’ term was already making it big and then the obvious question in mind was,” When to call ‘data’ à ‘ Big data’? “I then made an attempt to understand ‘how big is a data to be called  big data?’ and here, I have a big revelation to make, for all of you reading this article, that ‘Big Data’ is actually misleading term and it is irrelevant with “Bigness of data” but it is to be used in relevance. In fact, it is a term which needs to be understood, only in perspective.The simplest one I could find relevant is,  Big data is the data that cannot be stored with traditional storages, cannot be processed with traditional methods/ways and within a short period of time (and these references would still be valid as time advances.) but this is not textbook or only definition of big data. Interestingly, One who finds one set of data as big data can be traditional data for others so truly it cannot be bounded in words but loosely can be described through numerous examples. I am sure by the end of the article you will be able to answer the question for yourself. Let’s start.Do you know? - NASA researchers Michael Cox and David Ellsworth use the term “big data” for the first time to describe a familiar challenge in the 1990s supercomputers generating massive amounts of information - in Cox and Ellsworth’s case, simulations of airflow around aircraft - that cannot be processed and visualized.If you go through a  brief history of big data, you would know data which is not fitting into memory or disk was called ‘Big data problem’ back in 1997.As the years passed by innovations were on rising and disruptions were made so the data universe is growing all the time. Let’s understand a few widely available and stated statistics for ‘big data’ (Collected around 2017 or before) >>On average, people  send about 500 million tweets per day.Snapchat users share 527,760 photos in a minute Instagram users post 46,740 photos in a minute More than 120 professionals join LinkedIn in a minute Users watch 4,146,600 YouTube videos in a minuteThe average U.S. customer  uses 1.8 gigabytes of data per month on his or her cell phone plan.Amazon sells  600 items per second.On average, each person who uses email  receives 88 emails per day and send 34. That adds up to more than 200 billion emails each day.MasterCard processes  74 billion transactions per year.Commercial airlines  make about 5,800 flights per day.You might be interested to read through   Domo’s Data Never Sleeps 5.0 report, for the numbers generated every minute of the day.Understanding that the above stats are probably about 1.5-2 years older and data is ever-growing, it helps to establish the fact that ‘big data‘ is a moving target and…. In short,“Today’s big data is tomorrow’s small data.”Now that we have some knowledge about transactions/tweets/snaps in a day, Let’s also understand how much data, all these “One-minute Quickies” are generating. Let’s talk about some volumes too. Afterall volumes are one of the characteristics of big data but mind you, not only characteristic of big data. It is believed that, In a single day, the world produces 2.5 quintillion bytes (2.3 trillion gigabytes) of data, in layman's terms, this is the equivalent of everyone in the world downloading 60 episodes of Breaking Bad, in HD, 20 times! [Source:  VCloud 2012] and According to estimates, the volume of data worldwide doubles every 1.2 years.IDC predicts that the collective sum of the world's data will grow from 33 zettabytes this year to a 175ZB by 2025, for a compounded annual growth rate of 61 per cent. The 175ZB figure represents a 9 per cent increase over last year's prediction of data growth by 2025 – As per the report published in Dec’2018.But, do you know: how much would be 1 zettabyte of data? Let’s understand. One zettabyte is equal to one sextillion bytes or 1021 (1,000,000,000,000,000,000,000) bytes or, one zettabyte is roughly equal to a trillion gigabytes.Fun Fact: There is a legit term coined as The Zettabyte Era (Today’s Era).The Zettabyte Era can also be understood as an age of growth of all forms of digital data that exist in the world which includes the public Internet, but also all other forms of digital data such as stored data from security cameras or voice data from cell-phone calls.You must check out this  infographic by economywatch (taken from  SearchEngineJournal) to understand how much data zettabyte consists of, putting it into context with current data storage capabilities and usage.Today’s ‘big data’ is generated from majority 3 sources i.e. People Generated: Social media uploads, Mails etc. Machine Generated: M2M (machine to machine) interactions, IOT devices etc. Business Generated: Data generated and stored into today’s OLTPs, OLAPs, Data warehouses, data marts, reports, operational data throughout the enterprise/organization.Various analytics tools available in the market today, help in solving big data challenges by providing ways for storing this data, process this data and make valuable insights from this data.As we discussed, big data is moving target as time advances, it is also interesting to know even today, data which is not of huge size but is difficult to process and of relatively smaller volume would still be categorized as Big Data. For example, unstructured data in emails, from social media platforms, data which is required to process with real-time/near real-time etc. all the examples we have seen so far, all of it is big data.   But, It would be a mistake to assume that, Big Data only as data that is analyzed using Hadoop, Spark or another complex analytics platform. As big data is moving the target and it’s ever-growing, also with various disruptive sources of data are being introduced every day, to process this data newer tools would be invented, and hence big data cannot just remain a function of tools being used to analyze it. To conclude, as discussed at the starting of the article, it would still be appropriate and reasonable to say, this moving target of big data which would always be challenged for storage, processing methods and process it within a short period as well. So big data is a function of volume and/or time and/or storage and/or variety. It was fun and exciting to know what different aspects are hidden in ‘BIG DATA’ word and I thoroughly enjoyed solving this mystery.Did you enjoy solving it too?Do let us know how was experience through comments below.Happy Learning!!!
Rated 4.5/5 based on 23 customer reviews
How Big is ‘Big Data’, Anyway?
Blogs
14028
How Big is ‘Big Data’, Anyway?

When I got introduced to the data-world with my fi... Read More

Apache Spark Pros and Cons

Apache Spark:  The New ‘King’ of Big DataApache Spark is a lightning-fast unified analytics engine for big data and machine learning. It is the largest open-source project in data processing. Since its release, it has met the enterprise’s expectations in a better way in regards to querying, data processing and moreover generating analytics reports in a better and faster way. Internet substations like Yahoo, Netflix, and eBay, etc have used Spark at large scale. Apache Spark is considered as the future of Big Data Platform.Pros and Cons of Apache SparkApache SparkAdvantagesDisadvantagesSpeedNo automatic optimization processEase of UseFile Management SystemAdvanced AnalyticsFewer AlgorithmsDynamic in NatureSmall Files IssueMultilingualWindow CriteriaApache Spark is powerfulDoesn’t suit for a multi-user environmentIncreased access to Big data-Demand for Spark Developers-Apache Spark has transformed the world of Big Data. It is the most active big data tool reshaping the big data market. This open-source distributed computing platform offers more powerful advantages than any other proprietary solutions. The diverse advantages of Apache Spark make it a very attractive big data framework. Apache Spark has huge potential to contribute to the big data-related business in the industry. Let’s now have a look at some of the common benefits of Apache Spark:Benefits of Apache Spark:SpeedEase of UseAdvanced AnalyticsDynamic in NatureMultilingualApache Spark is powerfulIncreased access to Big dataDemand for Spark DevelopersOpen-source community1. Speed:When comes to Big Data, processing speed always matters. Apache Spark is wildly popular with data scientists because of its speed. Spark is 100x faster than Hadoop for large scale data processing. Apache Spark uses in-memory(RAM) computing system whereas Hadoop uses local memory space to store data. Spark can handle multiple petabytes of clustered data of more than 8000 nodes at a time. 2. Ease of Use:Apache Spark carries easy-to-use APIs for operating on large datasets. It offers over 80 high-level operators that make it easy to build parallel apps.The below pictorial representation will help you understand the importance of Apache Spark.3. Advanced Analytics:Spark not only supports ‘MAP’ and ‘reduce’. It also supports Machine learning (ML), Graph algorithms, Streaming data, SQL queries, etc.4. Dynamic in Nature:With Apache Spark, you can easily develop parallel applications. Spark offers you over 80 high-level operators.5. Multilingual:Apache Spark supports many languages for code writing such as Python, Java, Scala, etc.6. Apache Spark is powerful:Apache Spark can handle many analytics challenges because of its low-latency in-memory data processing capability. It has well-built libraries for graph analytics algorithms and machine learning.7. Increased access to Big data:Apache Spark is opening up various opportunities for big data and making As per the recent survey conducted by IBM’s announced that it will educate more than 1 million data engineers and data scientists on Apache Spark. 8. Demand for Spark Developers:Apache Spark not only benefits your organization but you as well. Spark developers are so in-demand that companies offering attractive benefits and providing flexible work timings just to hire experts skilled in Apache Spark. As per PayScale the average salary for  Data Engineer with Apache Spark skills is $100,362. For people who want to make a career in the big data, technology can learn Apache Spark. You will find various ways to bridge the skills gap for getting data-related jobs, but the best way is to take formal training which will provide you hands-on work experience and also learn through hands-on projects.9. Open-source community:The best thing about Apache Spark is, it has a massive Open-source community behind it. Apache Spark is Great, but it’s not perfect - How?Apache Spark is a lightning-fast cluster computer computing technology designed for fast computation and also being widely used by industries. But on the other side, it also has some ugly aspects. Here are some challenges related to Apache Spark that developers face when working on Big data with Apache Spark.Let’s read out the following limitations of Apache Spark in detail so that you can make an informed decision whether this platform will be the right choice for your upcoming big data project.No automatic optimization processFile Management SystemFewer AlgorithmsSmall Files IssueWindow CriteriaDoesn’t suit for a multi-user environment1. No automatic optimization process:In the case of Apache Spark, you need to optimize the code manually since it doesn’t have any automatic code optimization process. This will turn into a disadvantage when all the other technologies and platforms are moving towards automation.2. File Management System:Apache Spark doesn’t come with its own file management system. It depends on some other platforms like Hadoop or other cloud-based platforms.3. Fewer Algorithms:There are fewer algorithms present in the case of Apache Spark Machine Learning Spark MLlib. It lags behind in terms of a number of available algorithms.4. Small Files Issue:One more reason to blame Apache Spark is the issue with small files. Developers come across issues of small files when using Apache Spark along with Hadoop. Hadoop Distributed File System (HDFS) provides a limited number of large files instead of a large number of small files.5. Window Criteria:Data in Apache Spark divides into small batches of a predefined time interval. So Apache won't support record-based window criteria. Rather, it offers time-based window criteria.6. Doesn’t suit for a multi-user environment:Yes, Apache Spark doesn’t fit for a multi-user environment. It is not capable of handling more users concurrency.Conclusion:To sum up, in light of the good, the bad and the ugly, Spark is a conquering tool when we view it from outside. We have seen a drastic change in the performance and decrease in the failures across various projects executed in Spark. Many applications are being moved to Spark for the efficiency it offers to developers. Using Apache Spark can give any business a boost and help foster its growth. It is sure that you will also have a bright future!
Rated 4.5/5 based on 19 customer reviews
Apache Spark Pros and Cons
Blogs
8688
Apache Spark Pros and Cons

Apache Spark:  The New ‘King’ of Big DataApac... Read More

Fundamentals of Apache Spark

IntroductionBefore getting into the fundamentals of Apache Spark, let’s understand What really is ‘Apache Spark’ is? Following is the authentic one-liner definition.Apache Spark is a fast and general-purpose, cluster computing system.One would find multiple definitions when you search the term Apache Spark. All of those give similar gist, just different words. Let’s understand these special keywords which describe Apache Spark. Fast: As spark uses in-memory computing it’s fast. It can run queries 100x faster. We will get to details of architecture later to understand this aspect better little later in the article. One would find the keywords ‘Fast’ and/or ‘In-memory’ in all the definitions. General Purpose: Apache spark is a unified framework. It provides one execution model for all tasks and hence very easy for developers to learn and they can work with multiple APIs easily. Spark offers over 80 high-level operators that make it easy to build parallel apps and one can use it interactively from the Scala, Python, R, and SQL shells.Spark powers a stack of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application.Cluster Computing: Efficient processing of data on Set of computers (Refer commodity hardware here) or distributed systems. It’s also called a Parallel Data processing Engine in a few definitions. Spark is utilized for Big data analytics and related processing. One more important keyword associated with Spark is Open Source. It was open-sourced in 2010 under a   BSD license.Spark (and its RDD) was developed(earliest version as it’s seen today), in 2012, in response to limitations in the   MapReduce cluster computing paradigm. Spark is commonly seen as an in-memory replacement of MapReduce.Since its release, Apache Spark has seen rapid adoption due to its characteristics briefly discussed above.Who should go for Apache SparkBefore trying to find out whether Apache spark is for me? Or whether I have the right skill set, It's important to focus on the generality characteristic in further depth.Apache Spark consists of Spark Core and a set of libraries. The core is the distributed execution engine and the Java, Scala, and Python APIs offer a platform for distributed ETL application development. Additional libraries, built atop the core, allow diverse workloads for streaming, SQL, and machine learning.As Spark provides these multiple components, it’s evident that Spark is developed and widely utilized for big data and analytics.  Professionals who should learn Apache SparkIf one is aspiring to be landed into the following professions or anyone who has an interest in data and insights, Knowledge of spark will prove useful:Data ScientistsData EngineersPrerequisites of learning Apache SparkMost of the students looking for big data training, Apache spark is number one framework in big data. So most of the knowledge seekers looking for spark training, it is important to note that there are few prerequisites to learn apache spark.Before getting into Big data, you must have minimum knowledge on:Anyone of the programming languages >> Core   Python or Scala.Spark installations can be done on any platform but its framework is similar to Hadoop and hence having knowledge of HDFS and YARN is highly recommended. Having knowledge of Hive is an added advantage but is not mandatory.Basic knowledge of SQL. In SQL mainly select * from, joins and group by these three commands highly recommended.Optionally, knowing any cloud technology like AWS. Recommended for those who want to work with production-like environments.System requirements of Apache SparkOfficial site for  Apache Spark gives following recommendation (Traverse link for further details)Storage System: There are few ways to set this up as follows: Spark can run on the same node as HDFS. Spark standalone node cluster can be installed on the same nodes and configure Spark and Hadoop memory and CPU usage accordingly to avoid any interference.Or,1. Hadoop and Spark can execute on common Resource Manager ( Ex. Yarn etc)Or,2. Spark will be executing in same Local Area Network as HDFS but on separate nodes.Or3. If a requirement is a quick response and low latency from data stores then execute compute jobs on separate nodes than that of storage nodes.Local Disks: Typically 4-8 disks per node, configured without RAID.If underline OS is Linux then mount the disk with noatime option and in Spark environment configure spark.local.dir variable to be a comma-separated list of local disks.Note: For HDFS, it can be the same disk as HDFS.Memory: Minimum 8GB - 100s of GBs of memory per machine.A recommendation is the allocation of 75% of the memory to Spark.Network: 10GB or faster speed network.CPU cores: 8-16 Cores per machineHowever, for Training and Learning purpose and just to taste Spark, following are two available options: Run it locally Use AWS EMR (Or any cloud computing service)For learning purposes, minimum 4gb ram system with minimum 30gb disk may prove enough.History of Apache SparkSpark was primarily developed to Overcome the Limitations of MapReduce.Versioning: Spark initial version was version 0, version 1.6 is assumed to be a stable version and is being used in multiple commercial corporate projects. Version 2.3 is the latest available version. MapReduce is cluster computing  paradigm, which forces a particular linear  data flow structure on distributed programs: MapReduce programs read input data from disk,  map a function across the data,  reduce the results of the map, and store reduction results on disk. Due to multiple copies of data and multiple I/O as described, MapReduce takes lots of time to process the volume of data. MapReduce can do only batch time processing and is unsuitable for real-time data processingIt is unsuitable for trivial join like transformations. It’s unfit for large data on a network and also with OLTP data.Also, not suitable for graphics and interactive data.Spark overcomes all these limitations and able to do faster processing too on the local disk as well.Why Apache Spark?Numerous advantages of Spark have made its a market favorite.Let’s discuss one by one.Speed: Extends MapReduce Model to support computations like stream processing and interactive queries.Single Combination for processes and multiple tools:  Covers multiple workloads ( in a traditional system, it used to require different distributed systems), which makes combining different processing types and ease of tool management.Unification: Developers have to learn only one platform unlike multiple languages and tools in a traditional system.Support to different Resource Managers: Spark supports Hadoop HDFS system, and YARN for resource management but it’s not the only resource manager it supports. It works on MESOS and on any standalone scheduler like spark resource manager.Support for cutting-edge Innovation: Spark provides capabilities and support for an array of new-age technologies ranging from built-in machine learning libraries,   visualization tools, support for near processing (which was in a way the biggest challenge pre- spark era) and supports seamless integration with other deep learning frameworks like TensorFlow. This enables Spark to provide an innovative solution for new age use-cases.Spark can access diverse data sources and make sense of them all and hence it’s trending in the market over any other cluster computing software available. Who uses Apache SparkListing a few use cases of Apache spark below :1. Analytics - Spark can be very useful when building real-time analytics from a stream of incoming data.2. E-commerce - Information about the real-time transaction can be passed to streaming clustering algorithms like alternating least squares or K-means clustering algorithm. The results can be combined with data from other sources like social media profiles, product reviews on forums, customer comments, etc. to enhance the recommendations to customers based on new trends.Shopify: At Shopify, we underwrite credit card transactions, exposing us to the risk of losing money. We need to respond to risky events as they happen, and a traditional ETL pipeline just isn’t fast enough. Spark Streaming is an incredibly powerful real-time data processing framework based on Apache Spark. It allows you to process real-time streams like Apache Kafka using Python with incredible simplicity.Alibaba: Alibaba Taobao operates one of the world’s largest e-commerce platforms. We collect hundreds of petabytes of data on this platform and use Apache Spark to analyze these enormous amounts of data.3. Healthcare Industry –Healthcare has multiple use-cases of unstructured data to be processed in real-time. It has data ranging from image formats like scans etc to specific medical industry standards and wearable tracking devices. Many healthcare providers are keen on using spark for patient’s records to build 360 degrees view of the patient to do accurate diagnosis.MyFitnessPal: MyFitnessPal needed to deliver a new feature called “Verified Foods.” The feature demanded a faster pipeline to execute a number of highly sophisticated algorithms. Their legacy non-distributed Java-based data pipeline was slow, did not scale, and lacked flexibility.Here are a few other examples from industry leaders:Regeneron: Future of Drug Discovery with Genomics at Scale powered by SparkZeiss: Using Spark Structured Streaming for Predictive MaintenanceDevon Energy: Scaling Geographic Analytics with Spark GraphXYou can also learn more about use cases of Apache Spark  here.Career Benefits:Career Benefits of Spark for you as an individual:Apache Spark developers earn the highest average salary among all other programmers. According to its  2015 Data Science Salary Survey, O’Reilly found strong correlations between those who used Apache Spark and those who were paid more money. In one of its models, using Spark added more than $11,000 to the median salary.If you’re considering switching to this extremely in-demand career then taking up the  Apache Spark training will be an added advantage. Learning Spark will give you a steep competitive edge and can land you up in market best-paying jobs with top companies. Spark has gained enough adherents over the years to place it high on the list of fastest-growing skills; data scientists and sysadmins have evaluated the technology and clearly seen what they liked.  April’s Dice Report explored the fastest-growing technology skills, based on an analysis of job postings and data from Dice’s annual salary survey. The results are below; percentages are based on year-over-year growth in job postings:Benefits of Spark implementing Spark in your organization:Apache spark is now a decade older but still going strong. Due to lightning-fast processing and numerous other advantages discussed so far, Spark is still the first choice of many organizations.Spark is considered to be the most popular open-source project on the planet, with more than 1,000 contributors from 250-plus organizations, according to Databricks.ConclusionTo sum up, Spark helps to simplify the computationally intensive task of processing high volumes of real-time or batch data. It can seamlessly integrate with complex capabilities such as machine learning and graph algorithms. In short, Spark brings exclusive Big Data processing (which earlier was only for giant companies like Google) to the masses.Do let us know how your learning experience was, through comments below.Happy Learning!!!
Rated 4.5/5 based on 13 customer reviews
Fundamentals of Apache Spark
Blogs
9805
Fundamentals of Apache Spark

IntroductionBefore getting into the fundamentals o... Read More

Apache Kafka Vs Apache Spark: Know the Differences

A new breed of ‘Fast Data’ architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. - Dean Wampler (Renowned author of many big data technology-related books)Dean Wampler makes an important point in one of his webinars. The demand for stream processing is increasing every day in today’s era. The main reason behind it is, processing only volumes of data is not sufficient but processing data at faster rates and making insights out of it in real time is very essential so that organization can react to changing business conditions in real time.And hence, there is a need to understand the concept “stream processing “and technology behind it. So, what is Stream Processing?Think of streaming as an unbounded, continuous real-time flow of records and processing these records in similar timeframe is stream processing. AWS (Amazon Web Services) defines “Streaming Data” is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of Kilobytes). This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows and used for a wide variety of analytics including correlations, aggregations, filtering, and sampling.In stream processing method, continuous computation happens as the data flows through the system.Stream processing is highly beneficial if the events you wish to track are happening frequently and close together in time. It is also best to utilize if the event needs to be detected right away and responded to quickly.There is a subtle difference between stream processing, real-time processing (Rear real-time) and complex event processing (CEP). Let’s quickly look at the examples to understand the difference. Stream Processing: Stream processing is useful for tasks like fraud detection and cybersecurity. If transaction data is stream-processed, fraudulent transactions can be identified and stopped before they are even complete.Real-time Processing: If event time is very relevant and latencies in the second's range are completely unacceptable then it’s called Real-time (Rear real-time) processing. For ex. flight control system for space programsComplex Event Processing (CEP): CEP utilizes event-by-event processing and aggregation (for example, on potentially out-of-order events from a variety of sources, often with large numbers of rules or business logic).We have multiple tools available to accomplish above-mentioned Stream, Realtime or Complex event Processing. Spark Streaming, Kafka Stream, Flink, Storm, Akka, Structured streaming are to name a few. We will try to understand Spark streaming and Kafka stream in depth further in this article. As historically, these are occupying significant market share. Apache Kafka Stream: Kafka is actually a message broker with a really good performance so that all your data can flow through it before being redistributed to applications. Kafka works as a data pipeline.Typically, Kafka Stream supports per-second stream processing with millisecond latency.  Kafka Streams is a client library for processing and analyzing data stored in Kafka. Kafka streams can process data in 2 ways. Kafka -> Kafka: When Kafka Streams performs aggregations, filtering etc. and writes back the data to Kafka, it achieves amazing scalability, high availability, high throughput etc.  if configured correctly. It also does not do mini batching, which is “real streaming”.Kafka -> External Systems (‘Kafka -> Database’ or ‘Kafka -> Data science model’): Typically, any streaming library (Spark, Flink, NiFi etc) uses Kafka for a message broker. It would read the messages from Kafka and then break it into mini time windows to process it further. Representative view of Kafka streaming: Note:Sources here could be event logs, webpage events etc. etc. DB/Models would be accessed via any other streaming application, which in turn is using Kafka streams here. Kafka Streams is built upon important stream processing concepts such as properly distinguishing between event time and processing time, windowing support, and simple (yet efficient) management of application state. It is based on many concepts already contained in Kafka, such as scaling by partitioning.Also, for this reason, it comes as a lightweight library that can be integrated into an application.The application can then be operated as desired, as mentioned below: Standalone, in an application serverAs a Docker container, or Directly, via a resource manager such as Mesos.Why one will love using dedicated Apache Kafka Streams?Elastic, highly scalable, fault-tolerantDeploy to containers, VMs, bare metal, cloudEqually viable for small, medium, & large use casesFully integrated with Kafka securityWrite standard Java and Scala applicationsExactly-once processing semanticsNo separate processing cluster requiredDevelop on Mac, Linux, WindowsApache Spark Streaming:Spark Streaming receives live input data streams, it collects data for some time, builds RDD, divides the data into micro-batches, which are then processed by the Spark engine to generate the final stream of results in micro-batches. Following data flow diagram explains the working of Spark streaming. Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data. DStreams can be created either from input data streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs. Think about RDD as the underlying concept for distributing data over a cluster of computers. Why one will love using Apache Spark Streaming?It makes it very easy for developers to use a single framework to satisfy all the processing needs. They can use MLib (Spark's machine learning library) to train models offline and directly use them online for scoring live data in Spark Streaming. In fact, some models perform continuous, online learning, and scoring.Not all real-life use-cases need data to be processed at real real-time, few seconds delay is tolerated over having a unified framework like Spark Streaming and volumes of data processing. It provides a range of capabilities by integrating with other spark tools to do a variety of data processing.  Spark Streaming Vs Kafka StreamNow that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Following table briefly explain you, key differences between the two. Sr.NoSpark streamingKafka Streams1Data received form live input data streams is Divided into Micro-batched for processing.processes per data stream(real real-time)2Separated processing Cluster is requriedNo separated processing cluster is requried.3Needs re-configuration for Scaling Scales easily by just adding java processes, No reconfiguration requried.4At least one semanticsExactly one semantics5Spark streaming is better at processing group of rows(groups,by,ml,window functions etc.)Kafka streams provides true a-record-at-a-time processing capabilities. it's better for functions like rows parsing, data cleansing etc.6Spark streaming is standalone framework.Kafka stream can be used as part of microservice,as it's just a library.Kafka streams Use-cases:Following are a couple of many industry Use cases where Kafka stream is being used: The New York Times: The New York Times uses Apache Kafka and Kafka Streams to store and distribute, in real-time, published content to the various applications and systems that make it available to the readers.Pinterest: Pinterest uses Apache Kafka and the Kafka Streams at large scale to power the real-time, predictive budgeting system of their advertising infrastructure. With Kafka Streams, spend predictions are more accurate than ever.Zalando: As the leading online fashion retailer in Europe, Zalando uses Kafka as an ESB (Enterprise Service Bus), which helps us in transitioning from a monolithic to a micro services architecture. Using Kafka for processing event streams enables our technical team to do near-real time business intelligence.Trivago: Trivago is a global hotel search platform. We are focused on reshaping the way travellers search for and compare hotels while enabling hotel advertisers to grow their businesses by providing access to a broad audience of travellers via our websites and apps. As of 2017, we offer access to approximately 1.8 million hotels and other accommodations in over 190 countries. We use Kafka, Kafka Connect, and Kafka Streams to enable our developers to access data freely in the company. Kafka Streams powers parts of our analytics pipeline and delivers endless options to explore and operate on the data sources we have at hand.Broadly, Kafka is suitable for microservices integration use cases and have wider flexibility.Spark Streaming Use-cases:Following are a couple of the many industries use-cases where spark streaming is being used: Booking.com: We are using Spark Streaming for building online Machine Learning (ML) features that are used in Booking.com for real-time prediction of behaviour and preferences of our users, demand for hotels and improve processes in customer support. Yelp: Yelp’s ad platform handles millions of ad requests every day. To generate ad metrics and analytics in real-time, they built the ad event tracking and analyzing pipeline on top of Spark Streaming. It allows Yelp to manage a large number of active ad campaigns and greatly reduce over-delivery. It also enables them to share ad metrics with advertisers in a timelier fashion.Spark Streaming’s ever-growing user base consists of household names like Uber, Netflix, and Pinterest.Broadly, spark streaming is suitable for requirements with batch processing for massive datasets, for bulk processing and have use-cases more than just data streaming. Dean Wampler explains factors to evaluation for tool basis Use-cases beautifully, as mentioned below: Sr.NoEvaluation CharacteristicResponse Time windowTypical Use Case Requirement1.Latency tolerancePico to Microseconds (Real Real time)Flight control system for space programs etc.Latency tolerance< 100 MicrosecondsRegular stock trading market transactions, Medical diagnostic equipment outputLatency tolerance< 10 millisecondsCredit cards verification window when consumer buy stuff onlineLatency tolerance< 100 millisecondshuman attention required Dashboards, Machine learning modelsLatency tolerance< 1 second to minutesMachine learning model trainingLatency tolerance1 minute and abovePeriodic short jobs(typical ETL applications)2.Evaluation CharacteristicTransaction/events frequencyTypical Use Case RequirementVelocity1M per secondNest Thermostat, Big spikes during specific time period.3Evaluation CharacteristicTypes of data processingNAData Processing Requirement1. SQLNA2. ETL3. Dataflow4. Training and/or Serving Machine learning modelsData Processing Requirement1. Bulk data processingNA2. Individual Events/Transaction processing4.Evaluation CharacteristicUse of toolNAFlexibility of implementation1. Kafka : flexible as provides library.NA2. Spark: Not flexible as it’s part of a distributed frameworkConclusionKafka Streams is still best used in a ‘Kafka -> Kafka’ context, while Spark Streaming could be used for a ‘Kafka -> Database’ or ‘Kafka -> Data science model’ type of context.Although, when these 2 technologies are connected, they bring complete data collection and processing capabilities together and are widely used in commercialized use cases and occupy significant market share. 
Rated 4.5/5 based on 19 customer reviews
Apache Kafka Vs Apache Spark: Know the Differences
Blogs
9974
Apache Kafka Vs Apache Spark: Know the Differences

A new breed of ‘Fast Data’ architectures has e... Read More

Apache Spark Vs Apache Storm - Head To Head Comparison

In today’s world, the need for real-time data streaming is growing exponentially due to the increase in real-time data. With streaming technologies leading the world of Big Data, it might be tough for the users to choose the appropriate real-time streaming platform. Two of the most popular real-time technologies that might consider for opting are Apache Spark and Apache Storm. One major key difference between the frameworks Spark and Storm is that Spark performs Data-Parallel computations, whereas Storm occupies Task-Parallel computations. Read along to know more differences between Apache Spark and Apache Storm, and understand which one is better to adopt on the basis of different features. Comparison Table: Apache Spark Vs. Apache StormSr. NoParameterApache SparkApache Storm1.Processing  ModelBatch ProcessingMicro-batch processing2.Programming LanguageSupports lesser languages like Java, Scala.Support smultiple languages, such as Scala, Java, Clojure.3.Stream SourcesHDFSSpout4.MessagingAkka, NettyZeroMQ, Netty5.Resource ManagementYarn and Meson are responsible.Yarn and Mesos are responsible.6.Low LatencyHigher latency as compared to SparkBetter latency with lesser constraints7.Stream PrimitivesDStreamTuple, Partition8.Development CostSame code can be used for batch and stream processing.Same code cannot be used for batch and stream processing.9.State ManagementSupports State ManagementSupports State Management as well10.Message Delivery GuaranteesSupports one message processing mode: ‘at least once’.Supports three message processing mode: ‘at least once’, ‘at most once’, ‘exactly once’.11.Fault ToleranceIf a process fails, Spark restarts workers via resource managers. (YARN, Mesos)If a process fails, the supervisor process starts automatically.12.Throughput100k records per node per second10k records per node per second13.PersistenceMapStatePer RDD14.ProvisioningBasic monitoring using GangliaApache AmbariApache Spark: Apache Spark is a general-purpose, lighting fast, cluster-computing technology framework, used for fast computation on large-scale data processing. It can manage both batch and real-time analytics and data processing workloads.  Spark was developed at UC Berkeley in the year 2009. Apache Storm:Apache Storm is an open-source, scalable fault-tolerant, and real-time stream processing computation system. It is a framework for real-time distributed data processing, which focuses on stream processing or event processing. It can be used with any programming language and can be integrated using any queuing or database technology.  Apache Storm was developed by a team led by Nathan Marz at BackType Labs. Apache Spark Vs. Apache Storm1. Processing Model: Apache Storm supports micro-batch processing, while Apache Spark supports batch processing. 2. Programming Language:Storm applications can be created using multiple languages like Java, Scala and Clojure, while Spark applications can be created using Java and Scala.3. Stream Sources:For Storm, the source of stream processing is Spout, while that for Spark is HDFS. 4. Messaging:Storm uses ZeroMQ and Netty as its messaging layer while Spark is using a combination of Nettu and Akka for distributing the messages throughout the executors. 5. Resource Management:Yarn and Meson are responsible for resource management in Spark, while Yarn and Mesos are responsible for resource management in Storm. 6. Low Latency: Spark provides higher latency as compared to Apache Storm, whereas Storm can provide better latency with fewer restrictions.7. Stream Primitives:Spark provides with stream transforming operators which transform DStream into another, while Storm provides with various primitives which perform tuple level of processing at the stream level (functions, filters). 8. Development Cost:It is possible for Spark to use the same code base for both stream processing and batch processing. Whereas for Storm, the same code base cannot be used for both stream processing and batch processing.  9. State Management: The changing and maintaining state in Apache Spark can be updated via UpdateStateByKey, but no pluggable strategy can be applied in the external system for the implementation of state. Whereas Storm does not provide any framework for the storage of any intervening bolt output as a state. Hence, each application has to create a state for itself whenever required. 10. Message Delivery Guarantees (Handling the message level failures):Apache Spark supports only one message processing mode, viz, ‘at least once’, whereas Storm supports three message processing modes, viz, ‘at least once’ (Tuples are processed at least one time, but can be processed more than once), ‘at most once’  and ‘exactly once’ (T^uples are processed at least once). Storm’s reliability mechanisms are scalable, distributed and fault-tolerant. 11. Fault-Tolerant:Apache Spark and Apache Storm, both are fault tolerant to nearly the same extent. If a process fails in Apache Storm, then the supervisor process will restart it automatically, as the state management is managed by Zookeeper, while Spark restarts its workers with the help of resource managers, who may be Mesos, YARN or its separate manager.12. Ease of Development: In the case of Storm, there are effective and easy to use APIs which show that the nature of topology is DAG. The Storm tuples are written dynamically. In the case of Spark, it consists of Java and Scala APIs with practical programming, making topology code a bit difficult to understand. But since the API documentation and samples are easily available for the developers, it is now easier. Summing Up: Apache Spark Vs Apache StormApache Storm and Apache Spark both offer great solutions to solve the transformation problems and streaming ingestions. Moreover, both can be a part of a Hadoop cluster to process data. While Storm acts as a solution for real-time stream processing, developers might find it to be quite complex to develop applications due to its limited resources. The industry is always on a lookout for a generalized solution, which has the ability to solve all types of problems, such as Batch processing, interactive processing, iterative processing and stream processing. Keeping all these points in mind, this is where Apache Spark steals the limelight as it is mostly considered as a general-purpose computation engine, making it a highly demanding tool by IT professionals. It can handle various types of problems and provides a flexible environment to in. Moreover, developers find it to be easy and are able to integrate it well with Hadoop. 
Rated 4.5/5 based on 2 customer reviews
Apache Spark Vs Apache Storm - Head To Head Comparison
Blogs
26680
Apache Spark Vs Apache Storm - Head To Head Compar...

In today’s world, the need for real-time data st... Read More

Best ways to learn Apache Spark

If you ask any industry expert what language should you learn for Big Data? You will get an obvious reply to learn Apache Spark. Apache Spark is widely considered as the future of the Big Data industry. Since Apache Spark has stepped into Big data market, it has gained a lot of recognition for itself. Today, most of the cutting-edge companies like Apple, Facebook, Netflix, and Uber, etc. have deployed Spark at massive scale. In this blog post, we will understand why one should learn Apache Spark? And several ways to learn it. Apache Spark is a powerful open-source framework for the processing of large datasets. It is the most successful projects in the Apache software foundation. Apache Spark basically designed for fast computation, also which runs faster than Hadoop. Apache Spark can collectively process huge amount of data present in clusters over multiple nodes. The main feature of Apache Spark is its in-memory cluster computing that increases the processing speed of an application.Why You Should Learn Apache SparkApache Spark has become the most popular unified analytics engine for Big Data and Machine Learning. Enterprises are widely utilizing Spark which in turn is increasing demand for Apache Spark developers. Apache Spark developers are the ones earning the highest salary. IT professionals can leverage this upcoming skill set gap by pursuing a certification in Apache Spark. A developer with expertise in Apache Spark skills can earn an average salary of $78K as per Payscale. It is the right time for you to learn Apache Spark as there is a very high demand for Spark developers chances of getting a job is high.Here are the reasons why you should learn Apache Spark today:In order to go with the growing demand for Apache SparkTo fulfill the demands for Spark developersTo get benefits of existing big data investmentsResources to learn ReactTo learn Spark, you can refer to Spark’s website. There are multiple resources you will find to learn Apache Spark, from books, blogs, online videos, courses, tutorials, etc. With these multiple resources available today, you might be in the dilemma of choosing the best resource, especially in this fast-paced and swiftly evolving industry.BooksCertificationsVideosTutorials, Blogs, and TalksHands-on Exercises 1. BooksWhen was the last time you read a book? Do you have reading habits? If not, it’s the time to read the books. Reading has a significant number of benefits. Those aren’t fans of books might miss out the importance of Apache Spark. To learn Apache Spark, you can skim through the best Apache Spark books given below.Apache Spark in 24 hours is a perfect book for beginners which comprises 592 pages covering various topics. An excellent book to learn in a very short span of time. Apart from this, there are also books which will help you master.Here is the list of top books to learn Apache Spark:Learning Spark by Matei Zaharia, Patrick Wendell, Andy Konwinski, Holden KarauAdvanced Analytics with Spark by Sandy Ryza, Uri Laserson, Sean Owen and Josh WillsMastering Apache Spark by Mike FramptonSpark: The Definitive Guide – Big Data Processing Made SimpleSpark GraphX in ActionBig Data Analytics with SparkThese are the various Apache Spark books meant for you to learn. These books include for beginners and others for the advanced level professionals.2. Apache Spark Training and CertificationsOne more way to learn Apache Spark is through taking up training. Apache Spark Training will boost your knowledge and also help you learn from experience. You will be certified once you are done with training. Getting this certification will help you stand out of the crowd. You will also gain hands-on skills and knowledge in developing Spark applications through industry-based real-time projects.3. Videos:Videos are really good resources to help you learn Apache Spark. Following are the few videos will help you understand Apache Spark.Overview of SparkIntro to Spark - Brian ClapperAdvanced Spark Analytics - Sameer FarooquiSpark Summit VideosVideos from Spark Summit 2014, San Francisco, June 30 - July 2, 2013Full agenda with links to all videos and slidesTraining videos and slidesVideos from Spark Summit 2013, San Francisco, Dec 2-3-2013Full agenda with links to all videos and slidesYouTube playist of all KeynotesYouTube playist of Track A (Spark Applications)YouTube playist of Track B (Spark Deployment, Scheduling & Perf, Related projects)YouTube playist of the Training Day (i.e. the 2nd day of the summit)You can learn more on Apache Spark YouTube Channel for videos from Spark events. 4. Tutorials, Blogs, and TalksUsing Parquet and Scrooge with Spark — Scala-friendly Parquet and Avro usage tutorial from Ooyala's Evan ChanUsing Spark with MongoDB — by Sampo Niskanen from WellmoSpark Summit 2013 — contained 30 talks about Spark use cases, available as slides and videosA Powerful Big Data Trio: Spark, Parquet and Avro — Using Parquet in Spark by Matt MassieReal-time Analytics with Cassandra, Spark, and Shark — Presentation by Evan Chan from Ooyala at 2013 Cassandra SummitRun Spark and Shark on Amazon Elastic MapReduce — Article by Amazon Elastic MapReduce team member Parviz DeyhimSpark, an alternative for fast data analytics — IBM Developer Works article by M. Tim Jones 5. Hands-on ExercisesHands-on exercises from Spark Summit 2014 - These exercises will guide you to install Spark on your laptop and learn basic concepts.Hands-on exercises from Spark Summit 2013 - These exercises will help you launch a small EC2 cluster, load a dataset, and query it with Spark, Spark Streaming, and MLlib.So these were the best resources to learn Apache Spark. Hope you found what you were looking for. Wish you a Happy Learning!
Rated 4.5/5 based on 1 customer reviews
Best ways to learn Apache Spark
Blogs
8703
Best ways to learn Apache Spark

If you ask any industry expert what language shoul... Read More