Shulou(Shulou.com)06/03 Report--

Big data is a series of processing methods for storing, calculating, statistics, analyzing and processing massive data, and the amount of data processed is usually TB-level, or even PB or EB-level data, which can not be accomplished by traditional data processing methods. It involves distributed computing, high concurrent processing, high availability processing, cluster, real-time computing and so on, which brings together the popular IT technology in the current IT field.

To get started with big data, you need to learn the following knowledge points:

Here I still want to recommend the big data Learning Exchange Group I built myself: 529867072, all of them are developed by big data. If you are studying big data, the editor welcomes you to join us. Everyone is a software development party. Irregularly share practical information (only related to big data software development), including the latest big data advanced materials and advanced development tutorials sorted out by myself. Welcome to join us if you want to go deep into big data.

1. Java programming technology.

Java programming technology is the basis of big data's learning. Java is a strongly typed language with high cross-platform ability. It can write desktop applications, Web applications, distributed systems and embedded system applications. It is big data's favorite programming tool. Therefore, if you want to learn big data well, it is essential to master the foundation of Java.

2. Linux command

The development of big data is usually carried out in the Linux environment. Compared with the Linux operating system, the Windows operating system is a closed operating system, and the open source big data software is very limited. Therefore, if you want to engage in big data development related work, you need to master the basic Linux operation commands.

3 、 Hadoop

Hadoop is an important framework developed by big data, its core is that HDFS and MapReduce,HDFS provide storage for massive data, MapReduce provides computing for massive data, so we need to focus on mastering, in addition, we also need to master Hadoop cluster, Hadoop cluster management, YARN and Hadoop advanced management and other related technologies and operations!

4 、 Hive

Hive is a data warehouse tool based on Hadoop, which can map structured data files to a database table, provide simple sql query function, and transform sql statements into MapReduce tasks to run, which is very suitable for statistical analysis of data warehouse. It is necessary to master the installation, application and advanced operation of Hive.

5. Avro and Protobuf

Both Avro and Protobuf are data serialization systems, which can provide rich data structure types, which is very suitable for data storage, and can also be used for data exchange between different languages. To learn from big data, you need to master its specific usage.

6 、 ZooKeeper

ZooKeeper is an important component of Hadoop and Hbase. It is a software that provides consistency services for distributed applications. The functions provided include configuration maintenance, domain name service, distributed synchronization, component services and so on. In the development of big data, it is necessary to master the common commands of ZooKeeper and the implementation methods of its functions.

7 、 HBase

HBase is a distributed, column-oriented open source database, which is different from the general relational database. It is more suitable for the database of unstructured data storage. It is a distributed storage system with high reliability, high performance, column-oriented and scalable. Big data's development needs to master the basic knowledge, application, architecture and advanced usage of HBase.

8 、 phoenix

Phoenix is an open source SQL engine based on JDBC API operation HBase written in Java, which has the characteristics of dynamic column, hash loading, query server, tracking, transaction, user-defined function, secondary index, namespace mapping, data collection, row time stamp column, paging query, jump query, view and multi-tenant. Big data development needs to master its principle and usage.

9 、 Redis

Redis is a key-value storage system, which largely compensates for the deficiency of key/value storage such as memcached, and can supplement the relational database on some occasions. It provides clients such as Java,C/C++,C#,PHP,Java,Perl,Object-C,Python,Ruby,Erlang, which is very convenient to use. Big data's development needs to master the installation, configuration and related usage of Redis.

10 、 Flume

Flume is a highly available, highly reliable and distributed massive log collection, aggregation and transmission system. Flume supports customizing various data senders in the log system for data collection; at the same time, Flume provides the ability to simply process data and write to various data receivers (customizable). The development of big data needs to master its installation, configuration and related usage.

11 、 SSM

The SSM framework is integrated by three open source frameworks, Spring, SpringMVC and MyBatis, and is often used as a framework for web projects with simple data sources. The development of big data needs to master the three frameworks of Spring, SpringMVC and MyBatis respectively, and then use SSM for integration operation.

12 、 Kafka

Kafka is a high-throughput distributed publish and subscribe messaging system. The purpose of big data's development and application is not only to unify online and offline message processing through Hadoop's parallel loading mechanism, but also to provide real-time messages through clusters. The development of big data needs to master the principle of Kafka architecture and the function and usage of each component as well as the realization of related functions.

13 、 Scala

Scala is a multi-paradigm programming language. Big data designed the important framework Spark using Scala language. If you want to learn the Spark framework well, it is essential to have the foundation of Scala. Therefore, big data needs to master the basic knowledge of Scala programming!

14 、 Spark

Spark is a fast and general computing engine specially designed for large-scale data processing. It provides a comprehensive and unified framework for managing the needs of big data processing of different datasets and data sources. Big data development needs to master the basic knowledge of Spark, SparkJob, Spark RDD, spark job deployment and resource allocation, Spark shuffle, Spark memory management, Spark broadcast variables, Spark SQL, Spark Streaming and Spark ML.

15 、 Azkaban

Azkaban is a batch workflow task scheduler, which can be used to run a set of workflows and processes in a specific order. Azkaban can be used to complete big data's task scheduling. Big data development needs to master the relevant configuration and syntax rules of Azkaban.

16. Python and data analysis

Python is an object-oriented programming language with rich libraries, easy to use and widely used. It is also used in the field of big data, mainly for data acquisition, data analysis and data visualization. Therefore, big data needs to learn some knowledge of Python.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.