Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to self-study big data's technology". The content of the explanation in the article is simple and clear, and it is easy to learn and understand. Now please follow the editor's train of thought slowly and deeply. Let's study and learn "how big data's technology should be self-taught"!

Big data is also data in nature, but has new features, including a wide range of data sources, diversified data formats (structured data, unstructured data, Excel files, text files, etc.), large amount of data (at least TB level, maybe even PB level), fast data growth, and so on.

Learn which technologies we should learn through a few questions:

With a wide range of data sources, how to collect and summarize? corresponding to the emergence of tools such as Sqoop,Cammel,Datax.

After data acquisition, how to store it? corresponding to the emergence of distributed file storage systems such as GFS,HDFS,TFS.

After the data is stored, how to quickly calculate the results you want? The corresponding distributed computing framework such as MapReduce solves this problem; but writing MapReduce requires a large amount of Java code, so there are parsing engines such as Hive,Pig that convert SQL into MapReduce; ordinary MapReduce processing data can only be processed in batches, and the time delay is too long, in order to get the result for each piece of data input, there is a low-delay streaming computing framework such as Storm/JStorm. However, if you need both batch and stream processing, you have to build two clusters, Hadoop cluster (including HDFS+MapReduce+Yarn) and Storm cluster, which is not easy to manage, so there is an one-stop computing framework such as Spark, which can process both batch and stream (in essence, micro-batch processing). Then the emergence of Lambda architecture and Kappa architecture provides a general architecture for business processing.

What tools should we master in order to improve productivity:

Ozzie,azkaban: a tool for scheduled task scheduling.

Hue,Zepplin: graphical task execution management, result viewing tool.

Scala language: the best language to write Spark programs, of course, you can choose to use Python.

Python language: used when writing some scripts.

Allluxio,Kylin, etc.: tools to speed up the operation speed by preprocessing the stored data.

Thank you for your reading. the above is the content of "how to self-study big data's technology". After the study of this article, I believe you have a deeper understanding of how big data's technology should be taught by yourself. the specific use of the situation also needs to be verified by practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.