Let us go and see what will be the first step that you need to take. the first step would be to analyze yourself that how challenging or easy, it will be for you depending upon which technology you are in. So if you have a good hold on programming languages like Java, Scala, Python, R, or SQL. It's going to be pretty easy for you and on to that if you have an idea on data warehousing, client-server architecture, Linux shell scripting, it will be very easy for you to get into big data.

On the other hand, if your current experience is on some different technologies in the IT industry, for example like sap, web development, web servers applications, or java scripts then also it is not completely impossible. However, yeah learning curve will be a bit on a higher side. So now, let us see what could be the first step that you need to take the first thing. you need to understand the use of big data, I mean why these companies are using big data, what challenges the industry used to face when they were not using big data, I mean this basic idea you should have that why you are going to learn big data. Is it that big or it is just a market gimmick?

I am talking just the basic idea on what is batch data, what is streaming or live data, structured, semi-structured or unstructured data, what are eating concepts, what is Hadoop, what is spark hive, NoSQL databases, HBase and data warehousing. I have mentioned it in one-liner points because you just need to have very basic information like what it is, so that when you go ahead and deep dive into Big Data, these terms should not sound new to you. So just the basic idea of what these things are.

So let us say you start learning today. You can finish it in somewhat around one month. I have given one month because many of you I believe will be working with some of the other IT companies. so I'm not sure how much time you will be able to give it. However, yes, as far as these topics are concerned one month should be more than enough to get the basic idea. Then once you have the basic idea, you need to see what is Hadoop, once you get into the Big Data world, Hadoop should be one term that should be popping in your mind all the time. Therefore, you should have at least all this basic idea, which I have mentioned in these bullet points.

Hadoop architecture what is name node and data nodes, file system resource, managers cluster, resource planning, that means how you can plan a Hadoop cluster, important configuration files important processes and demons, high availability, and fault tolerance. So let us give near about 10 to 12 days for understanding these basic points. Then once you are done with this, the next topic will be ETL. You need to understand what ETL is and what its use is in big data. You need to see some important ETL concepts and important ETL tools. Let us give this near about three to five days.

See friends in this article I am just telling you the bullet points, that you should keep on your fingertips if you are going into the big data world. I am NOT going into the deep what ideal concepts will be, what ETL tools. I am just giving you a road map that will lead you to a good position in the big data industry. The next step would be you have to understand spark. Spark is one of the most used technology in the big data world. So you need to understand what is a spark, why spark required in big data, how are they related, you need to clear this up. Spark data structures and their importance. Spark has three data structures, so you need to have a basic idea of all of them. Then important spark libraries spark components like driver worker executor etc. important spark functions and spark program execution lifecycle. Therefore, for spark, it has a bit complicated so getting the basic idea itself will give it seven to ten days.

Then your next step would be to understand hive, what is a hive, why is hive required in the big data world, SQL. Then basic hive commands, performance, tuning concepts, which are most important tuning concepts like partitioning bucketing, then input-output file formats. Therefore, this will suffice for a basic idea in the hive. So let us give it three to five days so after understanding these concepts.

ETL Hadoop spark and hive you can say that you have achieved a beginner level than what next. Whatever concepts you have learned obviously in the programming world, you have to do a lot of hands-on. For which, I will tell you a very basic setup which is very easy to set up on your laptop or desktop and that will provide you basically all the tools which we have discussed so far and you can get your hands dirty on it. Therefore, for this, you have to download Oracle VirtualBox. Then once you have downloaded Oracle VirtualBox, the second step would be to download Cloudera QuickStart VM, this VirtualBox image would be somewhere around 150 to 200 MB but this Cloudera QuickStart VM will be somewhere around 5 to 6 GB. This Cloudera QuickStart VM includes big data tools like hive HBase Spark Scala Eclipse yarn zookeeper and a complete Hadoop setup.

This is based on Linux and OS operating system so for all this setup, you will not need more than one day and once you have all this set up as a beginner you can start doing your hands-on. First thing, you should practice the use case scenarios you will find most of the use case scenarios on Google, initially, you should practice like file input-output and HDFS how to load data in the hive, brush up some of your skills on Scala programming and basic data operations in SPARK. Therefore, friends for these hands-on use case scenarios will go on and on. I hope you like this article if you really did please hit that like button and please subscribe to our website. Thank you.

What is big data? Big data is extremely large or complex set of data and it is so large that it is difficult to process it using traditional database and software techniques. Every day we are creating approximately 2.5 quintillion bytes of data. So where is this huge amount of data being generated? from earlier we had mobile phones with the functionality of calling and text messages or clicking some pictures maybe but with the new technologies like smartphones we have a lot of applications for music, sports, social media like Facebook, Twitter, LinkedIn, and many more also.

Data is being generated when we shop online. So why does it need attention as the data is growing companies are capturing the data that streams into their businesses. They can apply analytics and get significant value from it with better speed and efficiency. Companies are leveraging the benefits of big data by analyzing the patterns and trends and predicting something useful out of it, for example, companies like Amazon and Netflix use big data to improve the customer experience as we see here from the statistics shown by 2020.

1.7 megabytes of data will be created every second for each human. This needs immediate attention because this data cannot be just thrown away it is going to give profit to the businesses big data challenges. Big data is not just about the volume of data it poses. Other challenges as well like velocity and variety as a volume 40 zettabytes of data will be created by 2020. this huge volume of data is either human-generated like from social media YouTube or candy machine-generated like through sensors and personal health trackers and canal so be generated with organizations like card details commercial transactions and medical records another challenge is the velocity the speed at which data is coming into the system.

The data needs to be processed with faster speed and then there is a variety of data is not only structured but also unstructured and semi-structured data like images videos and tweets. so how our enterprise is using this big data today. Let us see Big Data popular use cases Internet of Things. These are numerous ways in which analytics can be applied to the Internet of Things for example sensors are used to collect data. That can be analyzed to achieve actionable insights tracking customer or product movement etc. Many enterprises are creating a dashboard application that provides a 360-degree view of the customers that pulls data from a variety of sources analyzes it and presents it to customer service.

Therefore, this allows them to gather rich insights about businesses' big data. Popular use cases are related information security and data where have optimizations big data tools are being used to remove some of the burdens from the data warehouses. Even the healthcare industry is looking for patterns and treatments that lead to the best outcomes for patients. The main challenge of Big Data is storing and processing the data at a specified time span. The traditional approach is not efficient in doing that. Therefore, Hadoop technology and various big data tools have emerged to solve the challenges faced in the Big Data environment. Therefore, there are many big data tools and all of them help the user in some or another way in saving time money and uncovering business insights. These can be divided into the following categories like data storage and management.

The next broad category is data cleaning. Data needs to be cleaned up and well structured. Examples of such tools, which help in defining and reshaping the data into useable data sets, are Microsoft Excel. Data mining is a process of discovery in sites within a database some of the popular tools used for data mining are Terra data and rapid miner. Data visualization tools are a useful way of conveying and complex data insights in a pictorial way that is easy to understand. For example, Tableau and IBM Watson analytics and Plotly are the common tools for data reporting. Data ingestion is the process of getting the data into Hadoop, which can be done using scooped flume or storm data analysis requires asking questions and finding the answers and data.

The popular tools used for data analysis are Hive, Pig, MapReduce, and Spark. Data acquisition is also used for acquiring the data for which scoop flume or storm tools are quite popular. The popular Big Data tools offer many advantages, which can be summarized; as follows, they provide the analyst with advanced analytics algorithms and models. They help the user to run on big data platforms such as Hadoop or any high-performance analytic systems. They help the user to work with not only structured data but also unstructured and semi-structured data coming from multiple sources. It is quite easy to visualize and analyze data in a form that helps in conveying the complex data insights in a pictorial way, which is easy to understand by users big data tools help you to integrate with other technologies very easily.

In this session, let us try to understand what big data is. Big data refers to the huge volume of data that cannot be stored and processed using the traditional approach within the given period. The next big question that comes to our mind is how huge this data needs to be in order to be classified as big data. There is a lot of misconception while referring to the term Big Data. We usually use the term Big Data to refer to the data that is either in Gigabytes or Terabytes or Petabytes or Exabyte or anything that is larger than this in size. This does not define the term Big Data completely even a small amount of data can be referred to as big data depending on the context it is being used.

Let me take an example and try to explain it to you for instance. If we try to attach a document that is of 100 megabytes in size to an email, we would not be able to do so as the email system would not support an attachment of this size. Therefore, these 100 megabytes of attachment with respect to email can be referred to as Big Data.

Let me take another example and try to explain the term Big Data. Let us say we have around 10 terabytes of image files upon which certain processing needs to be done. For instance, we may want to resize and enhance these images within a given period. Suppose if we make use of the traditional system to perform this task we would not be able to accomplish this task within the given period, as the computing resources of the traditional system would not be efficient to accomplish this task on time. Therefore, these 10 terabytes of image files can be referred to as big data.

Now let us try to understand big data using some real-world examples. I believe you all might be aware of some of the popular social networking sites such as Facebook, Twitter, LinkedIn, Google+, and YouTube. Each of these sites receives a huge volume of data on a daily basis. It has been reported on some of the popular tech blocks that Facebook alone receives around 100 terabytes of data each day. Whereas Twitter processes around 400 million tweets each day as far as LinkedIn and Google+ are concerned each of their sites receives tens of terabytes of data on a daily basis. 

Finally coming to YouTube, it has been reported that each minute around 48 hours off lash videos are uploaded to YouTube you can just imagine how much volume of data is being stored and processed on these sites. However, as the number of users keeps growing on these sites storing and processing this data becomes a challenging task. Since this data holds a lot of valuable information. This data needs to be processed in a short span of time by using this valuable information. Companies can boost their sales and generate more revenue by making use of the traditional computing system. We would not be able to accomplish this task within the given period, as the computing resources of the traditional computing system would not be sufficient for processing and storing such a huge volume of data. This is where Hadoop comes into the picture we would be discussing Hadoop in more detail in the later sessions; therefore we can term this huge volume of data as big data.

Let me take another real-world example related to the airline industry and try to explain the term big data. For instance, the aircraft is while they are flying they keep transmitting data to the air traffic control located at the airports. The air traffic control uses this data to track and monitor the status and progress of the flight on a real-time basis. Since multiple aircraft would be transmitting this data simultaneously, a huge volume of data is accumulated at the air traffic control within a short span of time.

Therefore, it becomes a challenging task to manage and process this huge volume of data using the traditional approach. Hence, we can turn this huge volume of data into big data. I hope you all might have understood what big data is.