Big Data Interview Questions and Answers

In the early days of Big Data, it was accessible only to the Big businesses. As the technologies related to Big Data were costly, so the small businesses were not able to make use of it. But with the growth of   cloud and allied technologies now even the small enterprises are tapping the potential of big data and making big out of it.

More and more businesses are now turning to predictive analytics to drive sales and growth. There is also an increase in the number of connected devices. So, a large amount of data is being generated which contains insights that when harnessed can prove to be a boon for the enterprises. The trend is now to make use of machine learning and AI to gain an extra edge and remain competitive in the market. 

The trend is now shifting from on-premise processing to online/cloud processing.  It relieves businesses from heavy upward investments. They are now able to make use of the latest technologies and tools with a minimal/affordable cost. Because of these-  cost per usage basis- trends, nowadays even the small enterprises are now able to  have access to the Big Data Tools and Technologies and increasing efficiencies across all levels.

Data preparation involves collecting, combining, organizing and structuring data so that it can be analyzed for patterns, trends, and insights. The Big Data needs to be preprocessed, cleansed, validated and transformed. For this, the required data is pulled in from different sources internal or external.  One of the major focuses of data preparation is that the data under consideration for analysis is consistent and accurate. It so because accurate data will only produce valid results.

When the data is collected, it is not complete. It may have some missing values, outliers, etc. Data preparation is the major and very important activity in any Big Data project. Only good data will produce good results. Most of the time, the data resides in silos, in different databases. It is also in different formats. So it needs to be reconciled. There are five D's associated with the process of data preparation. These are :

1. Discover
2. Detain
3. Distill
4. Document
5. Deliver

The process of data preparation is automated. Various machine learning algorithms can be used in data preparation like filling missing values, fields renaming, ensuring consistency, removing redundancy, etc. There are various terminologies related to the process of data preparation such as data cleansing, transforming variables, removing outliers, data curation, data enrichment, data structuring and modeling, etc. These terminologies are actually the various processes or activities that are done under the process of data preparation.

It is seen that the time spent on data preparation is generally more than the time required for data analysis.

Though the methods used for data preparation are automated, it takes a lot of time to prepare the data as the volume of data is very large in quantity and it tends to grow continuously.

By data cleansing, you can identify which of your data records or entries are incomplete, inaccurate, incorrect or irrelevant. In other words, we can say that data cleansing is nothing but identifying the inaccuracies and redundancies in the dataset. There remain certain issues with the data we collect. These issues must be resolved or rectify before we can apply any kind of processing or analysis on the data. If the data remains unclean, it will give the wrong insights. To have good results, it is expected that the input data must also be good. For this data, cleansing is required and it is very important and a necessary step in any Big Data project.

Without cleansing data, you should not proceed further. Otherwise, you may end up with all the incorrect information. The various issues that our input dataset may contain are outlined as follows:

1. Invalid Values
2. Different Formats
3. Attribute Dependencies
4. Uniqueness
5. Missing Values
6. Misspellings
7. Wrongly Classified Values etc

There are various methods to identify these issues:

1. Visualization
2. Outlier Analysis
3. Validation Code

By visualization method, we mean we can take a random sample of the data and see whether it is correct or not.

By the Outlier Analysis method, we mean to find out an extreme or odd value that is not expected in that particular feature. For example in the 'age' column, we can not expect a value like 200 or 350 etc.

By the Validation Code method, we mean creating such a  code that can identify whether the data or values under consideration are right or not. Once we can identify the issues, we can apply the corresponding methods to correct them.

Cleansing a Big Data can become a time consuming and cumbersome process. So it is always suggested to start with a small random sample of the data.

Developing rules on a small valid sample of the data will speed-up your time required to get the required insights. This is so because it reduces the latency that is associated with the iterative analysis/exploration of Big Data.

One obvious question is- why do we need data transformation?

Several reasons make it compulsory to transform the data. In a Big Data kind of environment, we need to make use of every type of data available and from every possible source to draw useful insights out of it that will help the business to grow. These reasons can be:

1. Making it compatible with the other data
2. Moving it to the other systems
3. Joining it with other data
4. Aggregating the information present in the data.

Several steps can be followed to have a successful data transformation. These steps are:

1. Data Interpretation or Data Discovery
2. Data Quality Check - Pre-Translation
3. Data Translation or Data Mapping
4. Data Quality Check - Post-Translation.

There are many ways that you can perform the data transformation.

• You can use scripting to transform the data. i.e. you have to manually write a code to perform the required transformation.
• You can use automation tools on-premises.
• Or you can opt for cloud-based automation tools

The process of data transformation tends to be slow, costly and time-consuming. You have to design an optimized strategy to have a successful data transformation to take place considering all the aspects, business needs, objectives, data governance,  regulatory requirements, security, scalability, etc.

The different methods that can be used for data transformation are:

• Data binning: It is also called as data bucketing. It is a technique used for data pre-processing. It reduces the effects of small observational errors. In the process of data binning, the sample is divided into some intervals and then replaced by the categorical values.
• Indicator variables: The technique of indicator variables is used for the conversion of categorical data into the Boolean values. It is done by creating the indicator variables. If there are more than two values-'n', we are required to create 'n-1' columns.
• Centering & Scaling: The value of one feature can be centred by subtracting the mean of all values. For scaling the data, the centered feature is divided by the standard deviation.
• Other techniques:  We can use some other techniques for data transformation, such as making a group of outliers having the same values. We can also decide to replace the value with the number of times it appears in the column.

Dimensionality reduction means reducing the number of dimensions or variables that are under consideration. Big Data contains a large number of variables. Most of the time, some of these variables are correlated.   So there is always room to select only the major/distinct variables that contribute in a big way to produce the result. Such variables are also called Principal Components.

In most cases, some features are redundant. We can always reduce the features where we observe a high correlation.  Dimensionality Reduction technique is also known as 'Low Dimensional Embedding'.

When the number of variables is huge, it becomes difficult to draw inferences from the given data set.  Visualization also becomes too difficult. So, it is always desirable in such situations to reduce the number of features and utilize only the more significant features. Thus the technique of  Dimensionality Reduction helps a lot in such situations by allowing us to reduce the number of dimensions and speed up our analytics. There are several obvious advantages of Dimensionality Reduction such as:

1. Reduced storage due to data compression.
2. Reduced computation time.
3. Removal of redundant features
4. Visualization becomes easier.

Dimensionality Reduction may cause some loss of data but the advantages gain is more.

There are two approaches to do Dimensionality Reduction:

1. Feature Selection
2. Feature Extraction

Following are the different ways by which we can perform 'Feature Selection':

1. Filter Method
2. Wrapper Method
3. Embedded Method

In 'Feature Extraction' we reduce the data from a 'high dimensional space' to a lesser number of dimensions or 'lower-dimensional space'. The process of 'Dimensionality Reduction' can be linear or nonlinear. Several methods are used with  Dimensionality Reduction.

Some of these are:

1. PCA (Principal Component Analysis)
2. LCA (Linear Discriminant Analysis)
3. DCA (Generalized Discriminant Analysis)

When we are using the 'Principal Component Analysis', there is a requirement that the variance of the data which is in the 'lower-dimensional space' should be 'maximum'.  When it is being mapped to a 'lower-dimensional space' from a 'higher dimensional space'. The following steps are followed in the process of Principal Component

Analysis:

1. Constructing the Covariance Matrix of the given data.
2. Computing the EigenVectors of the computed matrix.
3. Reconstructing the variance of the original data by using the Eigen Vectors corresponding to the largest eigenvalue.

By using 'Linear Discriminant Analysis' we try to find such a linear combination of features that can separate the two or more classes of objects/events.

The 'Generalized  Discriminant Analysis' method is used to provide a mapping of the given 'input vectors' into a 'high dimensional feature space’.

Big Data is not a business jargon now. It is becoming a necessity. To remain competitive in the market, you have to make use of Big Data. Now, you can not ignore it. So, we see a positive upward trend in the adoption of Big Data across different verticals.

The use cases may be different for different industries. Previously it was only the private industries that were utilizing the power of Big Data. But nowadays we see the government organizations are also adopting Big Data.  The Big Data adoption in different enterprises is for different reasons.

Some of these are mentioned below:

1. To improve control over waste management, fraud detection, and abuse
2. To make efficient resource management
3. To formulate future strategies and budgeting
4. To predict equipment failures etc.

There are certain challenges in the adoption of Big Data that needs to be properly addressed. Some of these challenges are:

1. Initial capital investment
2. Change in culture
3. Change in business practices
4. Change in operations etc.

Many governments, as well as private organizations, have already invested heavily in their IT infrastructure before the emergence of Big Data. So making a sudden adoption of Big Data is reluctant. Many organizations have their data stored in silos. There were also no strategies and established practices for the extraction and processing of that data. So making full use of the data was also not possible due to different formats and protocols. The awareness regarding Big Data integration was little and the reluctance to change was high. Most of them want to adopt a wait and watch strategy. So, initially, we saw a slow adoption in Big Data and allied technologies. But nowadays the trend has shifted and we see wants to have some kind of Big Data adoption in their enterprises.

The Big Data solutions that are available also vary widely. A specific Big Data solution that is suitable for one enterprise may be completely unsuitable for the other. So considering these various challenges, it becomes imperative to see the best practices in the adoption of Big Data solutions. Certain factors are conducive to the adoption of Big Data:

1. Leadership that is open-minded and holistic
2. Business needs and objectives must be clearly stated before the implementation of any Big Data technology.
3. Specific business use cases should be identified and aligned with the business objectives.
4. A clear strategy and plan to utilize existing IT resources.
5. A holistic view of the integration plan with the legacy systems.
6. Emphasis on data governance.
7. Employees Awareness and training etc.

Attention should be given on the overall Big Data infrastructure and not just the present required application. While planning for the adoption of Big Data, care should also be taken regarding the proper size of the cluster, requirement of good commodity hardware, storage and network architecture, security and compliance considerations.

The overall data management including its availability, integrity, usability, security, etc. is termed as data governance. For effective data governance, there should be a council about data governance, well-defined procedures and an effective plan for the implementation of those procedures and practices.

When it is ensured that there is an integrity of the given data and also it is trustworthy, we get the expected business benefits out of that data. As businesses depend more and more on data for making business decisions, data governance becomes very important and more critical.

A uniform and consistent data access across different business applications should be ensured.  A team of data governance is responsible for implementing these policies and defined procedures regarding the handling of data. This team may include data managers, business managers and other staff related to the handling of data. Some Associations are dedicated to the promotion of best practices in data governance. These are:

1. Data Governance Institute
2. Data Management Association
3. Data Governance Professionals Organization etc.

There can be many use cases where we see the data governance can play a crucial role. Some of these use cases can be as enlisted below:

1. Mergers and Acquisitions,
2. Business Process Management,
3. Modernization of legacy systems
4. Financial and Regulatory Compliance,
5. Credit Risk Management
6. Business Intelligence Applications etc.

Some various strategies and steps need to be incorporated to have good data governance in place.

1. You need to decide the data ownership.
2. Define the policies regarding data storage, availability, backup, security, etc.
3. Define the standard procedures for authentication and usage of data by the different users in the enterprise.
4. Ensure good policies regarding data audit and various government compliances.
5. Ensure data consistency at various levels and across various departments and applications within the enterprise.

Thus by implementing data governance, we ensure data integrity, consistency, accuracy, accessibility and quality. Defining data ownership is considered to be the first step in data governance. Then the different processes regarding data storage, back-up, archival, security etc needs to be defined. Procedures and standards regarding data access and authorization need to be defined. A set of policies and audit controls regarding compliance with the different regulations and company policies should be defined.

Data stewardship means owning accountability regarding data availability, accessibility, accuracy,   consistency, etc. To ensure data stewardship, a team of different people is formed. This team includes data managers, data engineers, business analysts, policymakers, etc. A Data Steward is responsible for data proficiency and the management of an organization's data. He is also expected to critically handle almost all the things that are related to data policies, processing, data governance and look over the organization’s information assets in compliance with the different policies and the other regulatory obligations.

A Data Steward is supposed to answer the following questions:

1. What is the importance of this particular data to an organization?
2. How long the data should be stored?
3. What could be the improvements in the quality of the data insights?

He also looks after the data protection, data authorization and access depending upon the defined roles. Any breach would be immediately noted and brought to the notice of the management. He has to ensure that the practices regarding data retention, archival as well as disposal requirements have complied with the organizational policies and the various regulations in place. While ensuring transparency, he also has to check that the data privacy and security is not breached.  A Data Steward should ensure the quality of data and should take different measures to keep it intact in consultation with the various stakeholders.

He acts as an intermediary between the business side/ management of the organization and the IT department. Depending on the company culture and the kind of Big Data project is concerned,  we can have different models or forms of Data Stewards. There can be:

1. Subject Area wise Data Stewards
2. Functional Data Stewards
3. Process wise Data Stewards
4. System Data Stewards
5. Project Data Stewards

As far as skills are concerned, a Data Steward should have the following skills:

1. Programming skills
2. Database & Warehousing Proficiency
3. Technical Writing
4. Business Acumen & Foresight etc.

These are the models that are designed to measure an organization's maturity to Big Data. Big Data Maturity Models provide the required tools for an enterprise to assess its Big Data capabilities.  It assists an enterprise to formulate its goals and strategies concerning Big Data. Using Big Data Maturity Model, an enterprise can have clear communication about its Big Data strategy and policy among the various departments and at various levels within the enterprise.

A Big Data Maturity Model can also be used as a means to monitor the progressive journey of an enterprise into the world of Big Data. It also helps in identifying the weak areas and the areas that require more attention to fit into the Big Data arena. A Big Data Maturity Model gives a direction as to how an organization can make the efficient use of its Big Data to achieve the anticipated benefits out of the Big Data. It can also be implied from the use of the Big Data Maturity Model that the more mature a model, the more benefits/revenue an organization can expect. It also helps in curtailing the overall operational expenses.

Yes, there are several categories of Big Data Maturity Model. Mostly it is categorized into three main levels:

1. Descriptive Models
2. Comparative Models
3. Prescriptive Models.

The descriptive model helps in assessing the maturity level of an enterprise at various stages. It is described in qualitative terms. It does not provide any recommendation regarding the improvement in the maturity of an organization's Big Data capability. However, it helps you to understand the value that is generated out of your investments in Big Data. From it, you can identify the probable steps that can be taken to improve your Big Data potential. The following maturity levels are described in the descriptive model :

1. Ad-hoc
2. Foundational
3. Competitive Differentiating
4. Breakaway.

The Comparative model given an idea about the status of your organization concerning your competitors as far as Big Data capability is concerned. It provides a kind of benchmarking that you can use to know your position in the Big Data market.

It consists of quantitative as well as qualitative information to gauge your status/position when compared to your peers. Comparative models consist of various stages/levels in terms of maturity. These are as follows:

1. Nascent
2. Pre-adoption
3. Early-adoption
4. Corporate adoption
5. Mature/Visionary

The Prescriptive model first assesses the present situation and then the various stages. It then suggests the probable paths by which the improvement can be made to increase your Big Data capability/maturity. The  prescriptive model also has several phases which are mentioned as follows:

1. Undertake Big Data Education
2. Measure Big Data Readiness
3. Identify a significant Big Data use case
4. Structure a Proof of Concept Big Data Project.

The criteria to evaluate Big Data Maturity Model can be stated as follows:

1. Model structure completeness
2. The quality of development and evaluation of the model
3. Ease of use
4. Value creation

By model structure completeness we mean the completeness of the model in all respect taking into consideration the Big Data needs of the organization. The model should also exhibit consistency across all levels. The second criterion evaluates the model in terms of quality of development and evaluation. The model should exhibit trustworthiness. It should also be stable. In the third criterion, we see the ease of use of the model. The model should be easily applicable to the situations under consideration. It should also be comprehensible.

The fourth criterion state the value creation of the model concerning Big Data. We evaluate whether the model brings any value addition to the enterprise as far as Big Data initiatives are concerned. We also see how close the model to reality is. We also evaluate its relevance and performance.

We evaluate these Big Data Maturity model taking into consideration the various aspects of the business. These aspects can be stated as follows:

1. Business Strategy
2. Information
3. Analytics
4. Organizational Culture and Execution
5. Architecture
6. Governance

Big Data Maturity Model assists in a big way to plan and go ahead with the Big Data initiatives. First, it helps in formulating the organizational goals and strategies concerning Big Data. It equips the organization with the necessary tools to assess its  Big Data capabilities and plan accordingly. Big Data Maturity Models make an organization to ensure its maturity level concerning Big Data.

An organization can better assess itself on various aspects using Big Data Maturity Model.  It will help an organization to have clear communication with all the staff and across all the domains about the strategies, policies, and initiatives concerning Big Data. An  organization can also monitor itself in its implementation of Big Data initiatives and compare with the other players in the market who are in the

Big Data space. It will help an organization to revisit its goals and make corresponding changes in the implementation and strategic moves as far as the adoption of Big Data is concerned.

By using the Big Data Maturity Model, an organization can identify the areas where improvements are needed to have uniformity and cohesion in the implementation of Big Data initiative. In this way, by ensuring the correctness of strategy and guiding as well as gauging the implementation details of the Big Data initiatives, the Big Data Maturity Model plays a vital role in not just planning but also in monitoring the overall Big Data journey of an enterprise.

Yes, there are certain tools to assess the Big Data Maturity Model. We have tools like:

1. IBM's 'Big Data & Analytics Maturity Model'
2. 'TDWI Big Data Maturity Model Assessment Tool'
3. 'Knowledgent Big Data Maturity Assessment'
4. 'Info-Tech Big Data Maturity Assessment Tool'
5. 'CSC Big Data Maturity Tool'
6. 'Radcliffe Big Data Maturity Model'
7. 'Booz & Company's Model'
8. 'Van Veenstra's Model'

These tools give benchmark based on surveys. The survey contains around 50 questions across the various aspects of an organization. These aspects are:

1. Business Strategy
2. Information
3. Analytics
4. Organizational Culture and Execution
5. Architecture
6. Governance

These are also called as dimensions. There is a range of benchmark which determines the maturity level.

For example, the following table gives a glimpse of the maturity level Benchmarking:

Scores per Dimension Maturity Level

• <15 Nascent
• 16–25 Pre-Adoption
• 26–35 Early Adoption
• 36–45 Corporate Adoption
• 46–49 Mature
• 50 Visionary

Its interpretation can be as follows: If you get a score of 19 for a particular aspect/dimension,  it means you are in the pre-adoption level/stage for the dimension which is under consideration. These assessment tools guide you regarding the progress of your Big Data journey. It may be likely that this progress is not uniform across all the dimensions. A variation in score may be observed for different dimensions.

It may be likely that you are in the corporate adoption stage at data management but as far as analytics is concerned, you are in the pre-adoption stage. Thus the tools to assess Big Data Maturity Model are very helpful and give you the understanding of your maturity in the Big Data space.

There are various options to use as a messaging system with Big Data. These kinds of systems are widely known as the 'Distributed Commit Log Technologies'. Apart from delivery guarantees and flexibility, theses messaging systems are more inclined towards Throughput and Scalability. Some of the most used Big Data messaging systems are :

1. Apache Kafka
2. Microsoft Event Hubs
3. Amazon Kinesis
4. Google Pub/Sub
5. Rabbit MQ

'Apache Kafka' is a messaging system that is distributed in nature. It allows having publish-subscribe kind of messages in a data pipeline. It is a highly scalable and fast messaging system. It centralizes communication between large data systems. It is mostly used as a 'Central Messaging System'.  Kafka is fault-tolerant and so is more reliable. It gives high performance. Originally it was developed at 'LinkedIn'. 'Microsoft Event Hubs' are event investors. They receive and process millions of events/second. Producers of the events send them to an event hub through AMQP/HTTPS. 'Amazon Kinesis' is a cloud-based messaging service. It's processing in real-time. Google Pub/Sub is also a cloud-based messaging service. Here the 'Consumers' subscribe to a topic and the 'Publishers' send the messages to a topic. Rabbit MQ is a 'Message Queuing System'. It serves as a middleware between the 'Producers' and the 'Consumers.'

Lambda architecture is a Big Data processing architecture. To handle the enormous quantities of data, the lambda architecture makes use of batch as well as stream processing methods. It is a fault-tolerant architecture and achieves a balance between latency and throughput. Lambda architecture makes use of the model of data that has an append-only, immutable data source which serves as a system of record.

In this architecture, new events are appended to the existing events. The new events do not overwrite existing events. The lambda architecture is designed for ingesting and the processing of timestamp-based events. The state can be determined from the 'natural', 'time-based' ordering of the data.

In Lambda architecture, we have a system that consists of three layers:

1. Batch processing
2. Real-time processing
3. Serving layer

The third layer is to respond to queries. The data is ingested to the processing layers from a master copy of the entire data set. This master copy is immutable.  The real-time processing layer processes the data streams in real-time. It does not require completeness or any fix-ups.

This layer provides real-time views on the most recent data. So the latency is minimized but the throughput is sacrificed.   The real-time processing is also termed as speed processing.

As there is a lag by the batch layer in providing the views on the most recent data, we can say that the speed layer does the work of filling this gap. The benefit which we get from the speed layer is that the view is immediately available once we receive the data. This view may not be complete or we can say when compared with the view generated by the batch layer. However, there is always a choice with you to replace the view produced by the speed layer with the batch layer's view when that data made available to the batch layer. The output obtained from the batch layer and the speed layer is stored in the serving layer.  In response to the 'ad-hoc queries', this serving layer returns the views that are pre-computed or building the views from processed data.

Data enrichment is a process to improve, refine or enhance the data. It is something like adding some additional details to the existing data. It also includes adding external data from some trusted sources to the existing data. Data enrichment helps you to have complete and accurate data. More informed decisions can be made by having enriched data. As data is the most valuable asset in the Big Data world, it must be ensured that the data is in good condition. It should not be incomplete, missing, redundant or inaccurate. If we do not have good data, we can not expect good results out of it. 

What we mean by good data is that it should be complete and accurate. The process of data enrichment helps us to add more details to the existing data so that it becomes a complete data.

Incomplete or little data can not give a bigger or complete picture of your customer. If you have insufficient information about your customers, you may not be able to give the expected service or customized offerings. This affects the business conversion rate and ultimately the business revenue. So having data in a  good and complete condition is a must for Big Data analytics to give the correct insights and hence produce the expected results. Data enrichment involves data refinement that may be insufficient, inaccurate or may have small errors. Extrapolating data is also a kind of data enrichment. Here we produce more data from the available raw data.  There are several types of data enrichment methods. Out of these, the two significant methods are:

1. Demographic Data Enrichment
2. Geographic Data Enrichment

It is up to you to decide what kind of data enrichment you need depending on your business requirements and objectives. Data enrichment is not a one time process, it is to be done continuously because the customer data tends to change with time. There are several data enrichment tools available. Some of these are:

1. Clearbit
2. Datanyze
3. LeadGenius
4. Leadspace
5. Reachforce
6. FullContact
7. RingLead
8. DemandGen
9. ZoomInfo etc.

Outliers are observations that appear far away from the group. They diverge from the overall pattern outlined by the given sample. Due to the presence of outliers in the dataset, we can observe a drastic change in the results. There are various unfavourable effects of outliers in the data set. Some of the

impacts can be stated as follows:

1. It may increase the error variance.
2. Normality may get decreased.
3. It may decrease the power of various statistical tests.
4. We may get biased estimates.

Outliers must not be ignored and should be properly treated as their presence may change the basic assumptions in statistical modelling. The results may get skewed due to the presence of outliers. Before applying procedures to deal with the outliers, we should always try to reason out the presence of outliers.

If we know the reason for the presence of outliers in our dataset, we can use the methods accordingly, to deal with the outliers. The reasons for having outliers in the dataset can be as follows:

1. Non-natural (Data Errors)
2. Natural (True Outliers)

The non-natural reasons for outliers can be :

1. Data Entry Errors
2. Measurement Error
3. Sampling Error
4. Experimental Error
5. Data Processing Error etc.

Natural or true outliers can be originally present in the dataset. To deal with outliers, the following approaches can be used:

1. Deleting observations
2. Transformation
3. Binning
4. Imputing values
5. Treating as a separate group
6. Other statistical methods.

Trimming can also be used at the extremes/both ends to remove the outliers. Weights can also be assigned to different observations. Mean, Mode and Median can also be used to remove outliers. Before imputing values, we should analyze if it is a natural outlier or artificial. If the outliers present are significantly large in number, it is advisable to treat them as separate groups. We can then build corresponding models for both the groups. The output is then combined.

Using the Cloud for Big Data Development is a good choice. It will help the businesses to increase their operational efficiencies with a minimal initial investment. They just have to pay only for the facilities they are using. Furthermore, they can upgrade or downgrade the facilities as per the changing business requirements.

For some enterprises, deploying Big Data technologies on their premises prove to be a costly affair.  Most of the time, they do not possess the required expertise to deal with Big Data deployment. Furthermore, the initial investments in these technologies are more. Their business requirements are also changing with market conditions. The tools and technologies related to Big Data also tend to evolve with the changing requirements. So keeping updated with the latest versions/tools prove to be costly for the enterprises. So, the cloud seems to be a better alternative to start with the Big Data initiative. There are several players in the cloud space. The major Big Data Cloud providers are:

1. Amazon Web Services
2. Microsoft Azure
3. Google Cloud Platform
4. Rackspace
5. Qubole etc.

Developing a cloud-based solution for Big Data involves a lot of awareness regarding the various Big Data offerings by different cloud providers. At the very first a business should be very, very clear in its requirements regarding Big Data. These requirements can be something like:

1. Kind of insights needed
2. Data Sources
3. Storage needs
4. Processing requirements (batch/real-time) etc.

Once you are clear in your Big Data requirements and strategies for future developments, you can choose a better combination of the storage solutions, processing platforms and the analytical tools to get the required results from your Big Data initiatives.

Depending on the business constraints and the state regulations, you can decide to opt some Big Data solutions from the cloud and some tools can be employed within the enterprise to have a better tradeoff. This way you can ensure confirming the various regulations as well as make efficient use of the available resources and the budgetary provisions.

In Big Data projects, one of the greatest concerns is data availability and accessibility. The cloud providers assure 99.9 % uptime. They also employ various data checking and security mechanisms to ensure data availability all the time. Making such provisions at an enterprise-level requires heavy investments in not just capital but also in tackling the operational challenges. It is observed mostly that despite having sound planning, the demand becomes difficult to anticipate.  This results in under or over-allocation of resources which ultimately affects your investments. This will enable the new services, products and projects to initiate on a small scale basis with minimal or at very low costs. This gives a lot of room for innovations. So, opting for cloud seems to be a better choice as far as the initial journey into the world of Big Data is concerned.