Shulou(Shulou.com)06/01 Report--

What are the commonly used tools used by big data to develop? aiming at this problem, this article introduces the corresponding analysis and answer in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Java language and Linux operating system, which are the basis for learning big data.

Java: only need to know some basic knowledge, do not need to use deep Java technology to do big data, learning java SE is equal to learning the basics of big data.

Linux: because all the software related to big data runs on Linux, Linux should learn to be solid and learn that Linux can quickly master the technology related to big data, so that you can better understand the running environment and network environment configuration of big data software such as hadoop, hive, hbase, spark, and so on. You can take many detours less, and learning shell can more easily understand and configure big data cluster. At the same time, it can also let you know the future development of big data technology more quickly.

Hadoop: this is a popular big data processing platform, it has almost become synonymous with big data, so be sure to learn it. Hadoop contains HDFS, MapReduce and YARN these three components, HDFS is stored in these files like files on our computer hard disk, MapReduce is used to process data, and MapReduce is used to calculate data, one of its characteristics is that no matter how large the data is, as long as you give it time, MapReduce can run the data, but the time may not be too fast, so it is called batch processing of data.

Zookeeper: this is an one-size-fits-all oil. You can use it when you install Hadoop's HA, and Hbase can use it later. The software is usually used to store some collaborative information, which is generally no more than 1m, and all software that uses the software depends on it. For us, we only need to install the software correctly to make it work properly.

Mysql: we learned about big data processing, and then learned the tools of mysql database to deal with small data, because we are still using mysql, how many layers does mysql need to master? You can install and run it on Linux, configure simple permissions, change root passwords, and create databases. Here, we mainly learn the syntax of SQL, because the syntax of hive is very similar.

Sqoop: this file is used to import data from Mysql to Hadoop. Similarly, you can export Mysql datasheets directly as files into HDFS without it, and of course, be careful when using Mysql in a production environment.

Hive: this is a great tool for using SQL syntax, making it easy for you to work with large amounts of data without having to write MapReduce programs. Some people say Peeger, right? Mastered almost one of them like Pig.

Now that you've learned Hive, I'm sure you need this software to help you manage Hive or MapReduce,Spark scripts, check that your program is running correctly, alert you and retry the program if something goes wrong, and most importantly, it can also help you configure task dependencies. You're going to love it, or you'll be looking at a lot of scripts and writing crond.

Hbase: this is the NOSQL database in the Hadoop ecosystem, whose data is stored in the form of key and value, and key is unique, so it can be used for data rearrangement, and it can store a large amount of data compared to MYSQL. Therefore, he is often used to store destinations after dealing with big data.

Kafka: this is a better queuing tool, so why use queues? More data also needs to be queued, for example, how to deal with hundreds of gigabytes of files, you can take them out one by one when you put them in the queue, and of course, you can also use this tool to store or HDFS online real-time data, where you can collaborate with a tool called Flume, which is designed to provide simple processing of data and write it to various data receivers (such as Kafka).

Spark: it is used to make up for the lack of MapReduce-based data processing speed, it is characterized by loading data into memory for computing, rather than slow hard drives that can lead to crashes and slow evolution. It is especially suitable for iterative operation, in which the optimization of algorithm is the core. Either JAVA or Scala can manipulate it.

The answers to the questions about the tools commonly used by big data are shared here. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.