Shulou(Shulou.com)06/03 Report--

This article shares the basic learning materials of big data, and introduces big data's basic knowledge and development trend in detail, which is very suitable for beginners. Interested friends can refer to it.

The following is the text of big data's novice learning route:

Linux: because big data-related software runs on Linux, Linux should learn more solidly. Learning Linux well will be of great help for you to quickly master big data-related technology, and can make you better understand the running environment and network environment configuration of big data software such as hadoop, hive, hbase, spark, and so on. You can step on many holes less, and learn shell to understand scripts so that you can understand and configure big data cluster more easily. It will also make you learn faster about the new big data technology in the future.

Hadoop: this is now the popular big data processing platform has almost become synonymous with big data, so this is a must. Hadoop includes several components HDFS, MapReduce and YARN,HDFS are the places where data is stored, just like our computer's hard disk, files are stored on this, MapReduce is the data processing calculation, it has a characteristic is that no matter how big the data is, as long as you give it time, it can run the data, but the time may not be very fast, so it is called data batch processing. YARN is an important component that embodies the concept of Hadoop platform. With its big data ecosystem, other software can run on hadoop, so that we can make better use of the advantages of HDFS large storage and save more resources. For example, we no longer have to build a separate spark cluster, just let it run on the existing hadoop yarn. In fact, you can do big data's treatment by learning these components of Hadoop, but you may not have a clear concept of how big "big data" is right now. Listen to me and don't worry about this. In the future, when you work, there will be a lot of scenarios where you will encounter dozens of T / hundreds of T of large-scale data, and then you will not think that the big data is really good, and the bigger it is, the more you will have a headache. Of course, don't be afraid to deal with such a large scale of data, because this is your value, let those who do Javaee php html5 and DBA envy it.

Zookeeper: this is an one-size-fits-all oil. You will use it when installing Hadoop's HA, and you will use it in future Hbase. It is generally used to store some cooperative information, which is relatively small, generally no more than 1m, and the software that uses it depends on it. For us personally, we just need to install it correctly and make it normal run.

Mysql: we have finished learning big data's processing, and then we will learn the small data processing tool mysql database, because when we install hive, what layer does mysql need to master? You can install it on Linux, run it, configure simple permissions, change the root password, and create a database. The main thing here is to learn the syntax of SQL, because the syntax of hive is very similar to this.

Sqoop: this is used to import data from Mysql into Hadoop. Of course, you can not use this, directly export the Mysql data sheet to a file and then put it on HDFS, of course, the use of the production environment should pay attention to the pressure of Mysql.

Hive: this thing is an artifact for those who know SQL grammar. It makes it easy for you to deal with big data without having to write MapReduce programs. Some people say that Pig? It can be mastered almost as much as Pig.

Oozie: now that you've learned Hive, I'm sure you need this thing. It can help you manage your Hive or MapReduce or Spark scripts, check whether your program is executed correctly, send you an alarm if something goes wrong, help you retry the program, and most importantly, it can help you configure the dependencies of tasks. I'm sure you'll love it, otherwise you don't feel like shit when you look at that pile of scripts and the dense crond.

Hbase: this is the NOSQL database in the Hadoop ecosystem, its data is stored in the form of key and value, and key is unique, so it can be used for data weight, it can store a much larger amount of data than MYSQL. So he is often used as the storage destination after big data's processing is completed.

Kafka: this is an easy-to-use queuing tool. What is the queue for? Do you know to wait in line to buy tickets? If there is too much data, you also need to wait in line to deal with it, so that the other students you work with won't cry. Why do you give me so much data (such as hundreds of gigabytes of files)? how can I handle it? don't blame him because he doesn't work with big data. You can tell him that I put the data in the queue and you take it one by one when you use it, so that he will stop complaining and go to optimize his program immediately. Because it's his business that he can't handle it. Not the question you gave me. Of course, we can also use this tool to store or HDFS online real-time data, and you can use it with a tool called Flume, which is designed to provide simple data processing and write to a variety of data recipients (such as Kafka).

Spark: it is used to make up for the shortcomings of data processing speed based on MapReduce, which is characterized by loading data into memory for computing rather than reading slow, slow-evolving hard drives. It is especially suitable for iterative operations, so algorithm streams are particularly fond of it. It is written in scala. Either the Java language or Scala can operate on it because they all use JVM.

Follow-up improvement: big data combined with artificial intelligence to achieve a real data scientist, opened the second pulse of the governor of data science, in the company is a technical expert level, when the monthly salary doubled again and became the core backbone of the company.

Machine learning (Machine Learning, ML): is a multi-domain cross-discipline, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is the core of artificial intelligence and the fundamental way to make computers intelligent. It is widely used in all fields of artificial intelligence. It mainly uses induction, synthesis rather than deduction. The machine learning algorithm is basically fixed, and it is relatively easy to learn.

Deep learning (Deep Learning, DL): the concept of deep learning originates from the research of artificial neural network, and it has developed rapidly in recent years. Examples of deep learning applications include AlphaGo, face recognition, image detection and so on. It is a scarce talent at home and abroad, but deep learning is relatively difficult, and the algorithm is updated relatively fast, so we need to learn from experienced teachers.

These are the details of big data's basic learning materials. Have you gained anything after reading them? If you want to know more about it, you are welcome to follow the industry information!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.