Shulou(Shulou.com)06/03 Report--

Project GitHub address: https://github.com/heibaiying/BigData-Notes

Big data technology stack mind map big data common software installation guide 1. Hadoop distributed file storage system-HDFS distributed computing framework-MapReduce cluster resource manager-YARNHadoop stand-alone pseudo-cluster environment build Hadoop cluster environment build HDFS common Shell commands HDFS Java API use Zookeeper to build Hadoop high availability cluster II. Introduction to HiveHive and Core Concepts installation and deployment of Hive in Linux Environment basic use of Hive CLI and Beeline Command Line Hive commonly used DDL Operation Hive Partition Table and Bucket Table Hive View and Index Hive Common DML Operation Hive data query detailed explanation

Spark Core:

Brief introduction to Spark Spark Development Environment Building Elastic datasets RDDRDD Common operators detailed explanation of Spark Operation Mode and Job submission Spark Accumulator and broadcast variables Building Spark High availability Cluster based on Zookeeper

Spark SQL:

The basic use of DateFrame and DataSetStructured API using Spark SQL external data sources Spark SQL common aggregate functions Spark SQL JOIN operations

Spark Streaming:

Brief introduction to Spark Streaming basic Operation Spark Streaming Integration FlumeSpark Streaming Integration Kafka IV, StormStorm and flow processing introduction to Storm Core Concepts detailed explanation of Storm stand-alone Environment Construction of Storm Cluster Environment Construction of Storm programming Model detailed Analysis of three Packaging methods of Storm Project detailed Analysis of Storm Integration Redis detail Storm Integration HDFS/HBaseStorm Integration Kafka V, Flink

TODO

Introduction to HBaseHbase system architecture and data structure HBase basic environment building (Standalone / pseudo-distributed mode) HBase cluster environment building HBase commonly used Shell commands HBase Java APIHbase filter detailed HBase coprocessor detailed explanation of HBase disaster recovery and backup HBase SQL middle layer-- PhoenixSpring/Spring Boot integration Mybatis + Phoenix VII, KafkaKafka introduction Kafka high availability cluster based on Zookeeper Kafka producer detailed Kafka consumer detailed understanding of Kafka replica mechanism 8, Introduction to ZookeeperZookeeper and core concepts Zookeeper stand-alone environment and cluster environment build Shell commands Zookeeper Java client commonly used in Zookeeper-- Apache CuratorZookeeper ACL permission Control 9, introduction to FlumeFlume and basic use of Flume installation and deployment in Linux Environment Flume Integration Kafka X, introduction to SqoopSqoop and basic use of Sqoop installation 11, AzkabanAzkaban introduction to Azkaban3.x compilation and deployment of Azkaban Flow 1.0 using Azkaban Flow 2.012. Introduction to ScalaScala and configuration of basic data types and operator flow control statement arrays in the development environment-- an overview of common collection types of Array collection types-- Map & Tuple classes and object inheritance and trait functions-- Map & Tuple classes and object inheritance and trait functions & closures & Coriarization pattern matching type parameters implicit conversion and implicit parameters XIII. Common content big data uses common packaging methods to share information and recommend development tools.

Project GitHub address: https://github.com/heibaiying/BigData-Notes

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.