Shulou(Shulou.com)06/03 Report--

Big data platform is a series of technical platforms for collection, storage, calculation, statistics, analysis and processing of massive structured, unstructured and semi-institutionalized data. The amount of data processed by big data platform is usually TB-level, or even PB-level or EB-level data, which can not be processed by traditional data warehouse tools. The technologies involved are distributed computing, high concurrency processing, high availability processing, cluster, real-time computing and so on. It integrates all kinds of popular technologies in the current IT field.

This paper sorts out some common open source tools of big data platform and classifies them according to their main functions so that big data learners and users can quickly find and refer to them.

Collection of some common tools of ▲ big data platform

It mainly includes: language tools, data acquisition tools, ETL tools, data storage tools, analysis and calculation, query applications and operation and maintenance monitoring tools. The following is a brief description of each tool.

I. language tool class

1. Java programming technology.

Java programming technology is one of the most widely used network programming languages at present, and it is the basis of big data's study.

Java has the characteristics of simplicity, object-oriented, distributed, robust, security, platform independence and portability, multithreading, dynamic and so on. It has high cross-platform ability and is a strongly typed language. Can write desktop applications, Web applications, distributed systems and embedded system applications, etc., is big data engineer's favorite programming tool.

Most importantly, Hadoop and many other big data processing technologies use Java. Therefore, if you want to learn big data well, it is essential to master the foundation of Java.

Here I still want to recommend the big data Learning Exchange Group I built myself: 251956502, all of them are developed by big data. If you are studying big data, the editor welcomes you to join us. Everyone is a software development party. Irregularly share practical information (only related to big data software development), including the latest big data advanced materials and advanced development tutorials sorted out by myself. Welcome to join us if you want to go deep into big data.

2. Linux command

A lot of big data development is usually carried out in the Linux environment, compared with the Linux operating system, the Windows operating system is a closed operating system, the open source big data software is very limited. Therefore, if you want to work related to the development of big data, you also need to master the basic operation commands of Linux.

3 、 Scala

Scala is a multi-paradigm programming language. On the one hand, it inherits the excellent features of many languages, on the other hand, it does not abandon the powerful platform of Java. Big data developed the important framework Spark is designed in Scala language. If you want to learn the Spark framework well, it is essential to have the foundation of Scala. Therefore, big data development needs to master the basic knowledge of Scala programming!

4. Python and data analysis

Python is an object-oriented programming language with rich libraries, easy to use and widely used. It is also used in the field of big data, mainly for data acquisition, data analysis and data visualization. Therefore, big data developers need to learn some knowledge of Python.

2. Data acquisition tools

1 、 Nutch

Nutch is an open source search engine implemented by Java. It provides us with all the tools we need to run our own search engine, including full-text search and Web crawlers.

2 、 Scrapy

Scrapy is an application framework written for crawling website data and extracting structural data, which can be used in a series of programs such as data mining, information processing or storing historical data. Big data's collection needs to master the crawler technology of Nutch and Scrapy.

III. ETL tools

1 、 Sqoop

Sqoop is a tool for transferring data between Hadoop and a relational database server. It is used to import data from relational databases (such as MySQL,Oracle) to Hadoop HDFS and export from the Hadoop file system to relational databases. Learning to use Sqoop is of great help in importing data from relational databases to Hadoop.

2 、 Kettle

Kettle is an ETL toolset that allows you to manage data from different databases by providing a graphical user environment to describe what you want to do, not what you want to do. As an important part of Pentaho, there are more and more domestic project applications, and its data extraction is efficient and stable.

IV. Data storage tools

1. Hadoop distributed storage and computing

Hadoop implements a distributed file system (Hadoop Distributed File System), referred to as HDFS. The core design of Hadoop's framework is: HDFS and MapReduce. HDFS provides storage for massive data, while MapReduce provides computing for massive data, so it needs to be mastered.

In addition, you also need to master Hadoop cluster, Hadoop cluster management, YARN and Hadoop advanced management and other related technologies and operations!

2 、 Hive

Hive is a data warehouse tool based on Hadoop, which can map structured data files to a database table, provide simple SQL query function, and transform SQL statements into MapReduce tasks to run. Compared to writing MapReduce in Java code, Hive has obvious advantages: rapid development, low personnel cost, scalability (free expansion of cluster size), and extensibility (support for custom functions). It is very suitable for statistical analysis of data warehouse. It is necessary to master the installation, application and advanced operation of Hive.

3 、 ZooKeeper

ZooKeeper is an open source distributed coordination service, which is an important component of Hadoop and HBase. It is a software that provides consistency services for distributed applications, including configuration maintenance, domain name services, distributed synchronization, component services and so on. In the development of big data, it is necessary to master the common commands and functions of ZooKeeper.

4 、 HBase

HBase is a distributed, column-oriented open source database, which is different from the general relational database. It is more suitable for the database of unstructured data storage. It is a distributed storage system with high reliability, high performance, column-oriented and scalable. Big data's development needs to master the basic knowledge, application, architecture and advanced usage of HBase.

5 、 Redis

Redis is a Key-Value storage system, which largely compensates for the deficiency of Key/Value storage such as Memcached, and can play a good complementary role to relational databases in some situations. It provides clients such as Java,C/C++,C#,PHP,Java,Perl,Object-C,Python,Ruby,Erlang. It is very convenient to use, big data development needs to master the installation, configuration and related use of Redis.

6 、 Kafka

Kafka is a high-throughput distributed publish and subscribe messaging system. The purpose of big data's development and application is not only to unify online and offline message processing through Hadoop's parallel loading mechanism, but also to provide real-time messages through clusters. The development of big data needs to master the principle of Kafka architecture and the function and usage of each component as well as the realization of related functions.

7 、 Neo4j

Neo4j is a high-performance NoSQL graphics database with large-scale network analysis capabilities for dealing with millions and T-level nodes and edges. It is an embedded, disk-based, fully transactional Java persistence engine, but it stores structured data on a network (mathematically called a graph) rather than in a table. Neo4j has attracted more and more attention because of its embedded, high-performance, lightweight and other advantages.

8 、 Cassandra

Cassandra is a hybrid non-relational database, similar to Google's BigTable, and its main functions are richer than Dynamo (distributed Key-Value storage system). This NoSQL database was originally developed by Facebook and has been used by more than 1500 business organizations, including Apple, CERN, Comcast, eBay, GitHub, GoDaddy, Hulu, Instagram, Intuit, Netflix, Reddit and so on. It is a popular distributed structured data storage scheme.

9 、 SSM

The SSM framework is integrated by three open source frameworks, Spring, Spring MVC and MyBatis, and is often used as a framework for Web projects with simple data sources. The development of big data needs to master the three frameworks of Spring, Spring MVC and MyBatis respectively, and then use SSM for integration operation.

Tools for analysis and calculation

1 、 Spark

Spark is a fast and general computing engine specially designed for large-scale data processing. It provides a comprehensive and unified framework for managing the needs of big data processing of different datasets and data sources. Big data development needs to master the basic knowledge of Spark, SparkJob, Spark RDD deployment and resource allocation, Spark Shuffle, Spark memory management, Spark broadcast variables, Spark SQL, Spark Streaming and Spark ML.

2 、 Storm

Storm is a free open source software, a distributed, fault-tolerant real-time computing system, can be very reliable to deal with large data streams, used to deal with Hadoop batch data. Storm supports many programming languages and has many applications: real-time analysis, online machine learning, non-stop computing, distributed RPC (remote procedure call protocol, a request for services from remote computer programs over the network), ETL, and so on.

The processing speed of Storm is amazing: it has been tested that each node can process 1 million data tuples per second.

3 、 Mahout

The purpose of Mahout is to "create an environment for rapidly creating scalable, high-performance machine learning applications". The main feature is to provide a scalable environment for scalable algorithms, novel Scala/Spark/H2O/Flink-oriented algorithms, Samsara (vector mathematical environment similar to R), and it also includes many algorithms for data mining on MapReduce.

4 、 Pentaho

Pentaho is the most popular open source business intelligence software in the world. It is a Java-based BI suite with workflow as the core and emphasizing solution-oriented rather than tool components. It includes a Web Server platform and several tool software: report, analysis, chart, data integration, data mining and so on, which can be said to include all aspects of business intelligence.

Pentaho's tools can connect to NoSQL databases. Big data developers need to know how to use it.

VI. Query application tools

1. Avro and Protobuf

Both Avro and Protobuf are data serialization systems, which can provide rich data structure types, which is very suitable for data storage, and can also be used for data exchange between different languages. To learn from big data, you need to master its specific usage.

2 、 Phoenix

Phoenix is an open source SQL engine based on JDBC API operation HBase written in Java, which has the characteristics of dynamic column, hash loading, query server, tracking, transaction, user-defined function, secondary index, namespace mapping, data collection, time stamp column, paging query, jump query, view and multi-tenant. Big data development needs to master its principle and usage.

3 、 Kylin

Kylin is an open source distributed analysis engine that provides SQL interfaces for Hadoop-based very large datasets (TB/PB level) and multi-dimensional OLAP distributed online analysis. Originally developed by eBay and contributed to the open source community. It can query huge Hive tables in subseconds.

4 、 Zeppelin

Zeppelin is a Web-based notebook that provides interactive data analysis. It is convenient for you to make exquisite documents that are data-driven, interactive and collaborative, and support a variety of languages, including Scala (using Apache Spark), Python (Apache Spark), SparkSQL, Hive, Markdown, Shell and so on.

5 、 ElasticSearch

ElasticSearch is a Lucene-based search server. It provides a distributed, multi-user full-text search engine based on RESTful Web interface. ElasticSearch is developed in Java and released as open source under the Apache license terms, and is currently a popular enterprise search engine. Designed for cloud computing, can achieve real-time search, stable, reliable, fast, easy to install and use.

6 、 Solr

Solr is based on Apache Lucene, is a highly reliable, highly expanded enterprise search platform, is a very excellent full-text search engine. Well-known users include eHarmony, Sears, StubHub, Zappos, Best Buy, AT&T, Instagram, Netflix, Bloomberg and Travelocity. The development of big data needs to understand its basic principle and usage.

7. Data management tools

1 、 Azkaban

Azkaban is an open source batch workflow task scheduler from linked, which is composed of three parts: Azkaban Web Server (Management Server), Azkaban Executor Server (execution Manager) and MySQL (Relational Database). It can be used to run a set of workflows in a specific order, and you can use Azkaban to complete big data's task scheduling. Big data development needs to master the relevant configuration and syntax rules of Azkaban.

2 、 Mesos

Mesos is an open source cluster management software first developed by AMPLab of the University of California, Berkeley, which supports Hadoop, ElasticSearch, Spark, Storm and Kafka architectures. For the data center, it is like a single resource pool, which removes CPU, memory, storage and other computing resources from physical or virtual machines. It is easy to establish and effectively run fault-tolerant and flexible distributed systems.

3 、 Sentry

Sentry is an open source real-time error reporting tool that supports Web front and back end, mobile applications and games, mainstream programming languages and frameworks such as Python, OC, Java, Go, Node, Django, RoR, and so on. It also provides the integration of common development tools such as GitHub, Slack, Trello, etc. Using Sentry is very helpful for data security management.

VIII. Monitoring tools for operation and maintenance

Flume is a highly available, highly reliable and distributed massive log collection, aggregation and transmission system. Flume supports customizing various data senders in the log system for data collection; at the same time, Flume provides the ability to simply process data and write to various data receivers (customizable). The development of big data needs to master its installation, configuration and related usage.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.