Shulou(Shulou.com)06/03 Report--

If you are confident that you can keep on learning after reading, then take action now!

First, big data's technical foundation

1. Basic operation of linux

Introduction and installation of linux system

Common linux commands-File Operations

Linux Common commands-user Management and permissions

Linux Common commands-system Management

Common linux commands-Secret-free login configuration and network management

Installation of common software on linux

Linux local yum source configuration and yum software installation

Linux Firewall configuration

Linux advanced text processing commands cut, sed, awk

Linux scheduled tasks crontab

2. Shell programming

Shell programming-basic syntax

Shell programming-process Control

Shell programming-function

Shell programming-Integrated case-Automated deployment script

3. In-memory database redis

Introduction to redis and nosql

Redis client connection

Operation and Application of string Type data structure in redis-object Cache

Redis list type data structure operation and application case-task scheduling queue

Redis hash and set data structure Operation and Application case-Shopping cart

Redis sortedset data structure Operation and Application case-ranking

4. Layout Coordination Service zookeeper

Brief introduction and Application scenario of zookeeper

Zookeeper cluster installation and deployment

Data Node and Command Line Operation of zookeeper

Basic operation and event monitoring of java client of zookeeper

Zookeeper Core Mechanism and data Node

Zookeeper Application case-distributed shared Resource Lock

Zookeeper application case-dynamic awareness of server online and offline

The principle of data consistency of zookeeper and the election mechanism of leader

Or I would like to recommend the big data Learning Exchange I created by myself Qun: 710219868 there are bosses and materials, enter the Qun chat invitation code and fill in Nanfeng (required)

There is a sharing open class with learning routes. After listening to it, you will know how to learn from big data.

5. Enhanced advanced features of java

Basic knowledge of Java multithreading

Detailed explanation of Java synchronization keywords

Java concurrent Thread Pool and its Application in Open Source Software

Java concurrent package message team and its Application in Open Source Software

Java JMS technology

Java dynamic proxy reflection

6. Lightweight RPC framework development

RPC principle learning

Nio principle learning

Netty commonly used API learning

Requirement Analysis and principle Analysis of lightweight RPC Framework

Lightweight RPC framework development

2. Offline computing system

1. Getting started with hadoop

Background introduction of hadoop

Overview of distributed system

Introduction to the process of offline data analysis

Cluster building

Preliminary use of cluster

2. HDFS enhancement

The concept and characteristics of HDFS

Shell (command line client) operation of HDFS

The working Mechanism of HDFS

The working Mechanism of NAMENODE

Api operation of java

Case 1: developing shell capture script

3. Detailed explanation of MAPREDUCE

Customize the RPC framework of hadoop

Mapreduce programming Specification and sample programming

Mapreduce Program running Mode and debug method

The Internal Mechanism of the running Mode of mapreduce Program

The main Workflow of mapreduce Computing Framework

Serialization method of custom object

MapReduce programming case

4. MAPREDUCE enhancement

Mapreduce sorting

Custom partitioner

Combiner of Mapreduce

Detailed explanation of the working Mechanism of mapreduce

5. MAPREDUCE actual combat

Maptask parallelism mechanism-file slicing

Maptask parallelism setting

Inverted index

Mutual friend

6. Introduction of federation and use of hive

HA Mechanism of Hadoop

Installation and deployment of HA cluster

Datanode dynamic online and offline for cluster operation and maintenance testing

Namenode State switching Management for Cluster Operation and maintenance testing

Balance of data blocks tested by cluster operation and maintenance

HDFS-API change under HA

Introduction to hive

Hive architecture

Hive installation and deployment

The first use of hvie

7. Hive enhancements and flume introduction

Basic syntax of HQL-DDL

Basic syntax of HQL-DML

Join of HIVE

HIVE parameter configuration

HIVE custom functions and Transform

An example Analysis of HQL execution by HIVE

HIVE Best practices Note

HIVE optimization strategy

HIVE actual combat case

Flume introduction

Installation and deployment of Flume

Case: collect directory to HDFS

Case: collect files to HDFS

III. Flow calculation

1. Storm from beginner to proficiency

What is Storm?

Storm architecture analysis

Storm architecture analysis

Storm programming model, Tuple source code, concurrency analysis

Storm WordCount cases and Common Api Analysis

Storm Cluster deployment practice

Storm+Kafka+Redis service index calculation

Storm × × compile

Strom Cluster Startup and Source Code Analysis

Storm task submission and source code analysis

Analysis of Storm data sending process

Analysis of Storm Communication Mechanism

Fault-tolerant Mechanism and Source Code Analysis of Storm messages

Storm Multi-stream Project Analysis

Write your own streaming task execution framework

2. Storm upstream and downstream and architecture integration

What is a message queue?

Kakfa core components

Kafka Cluster deployment practice and Common commands

Kafka profile carding

Kakfa JavaApi learning

Analysis of Kafka File Storage Mechanism

Deployment of Redis Foundation and stand-alone Environment

Redis data structure and typical cases

Getting started with Flume

Flume+Kafka+Storm+Redis integration

Fourth, memory computing system Spark

1. Scala programming

Introduction to scala programming

Installation of scala related software

Basic syntax of scala

Scala methods and functions

Features of scala functional programming

Scala arrays and collections

Scala programming activity (stand-alone WordCount)

Scala object oriented

Scala pattern matching

Introduction to actor programming

Option and partial function

Actual combat: concurrent WordCount of actor

Corey

Implicit conversion

2. AKKA and RPC

Akka concurrent programming framework

Actual practice: RPC programming practice

3. Getting started with Spark

Spark introduction

Spark environment building

Introduction to RDD

Conversion and Action of RDD

Actual combat: RDD comprehensive exercise

RDD advanced operator

Custom Partitioner

Actual combat: the number of website visits

Broadcast variable

Actual combat: calculate the place of ownership according to IP

Custom sorting

Using JDBC RDD to realize data Import and Export

Detailed explanation of WorldCount execution process

4. Detailed explanation of RDD

RDD dependency relationship

RDD caching mechanism

Checkpoint checkpoint mechanism of RDD

Analysis of Spark Task execution process

Stage Partition of RDD

5. Spark-Sql application

Spark-SQL

Spark combined with Hive

DataFrame

Actual combat: cases of Spark-SQL and DataFrame

6. SparkStreaming application practice

Introduction to Spark-Streaming

Spark-Streaming programming

Actual combat: StageFulWordCount

Flume combined with Spark Streaming

Kafka combined with Spark Streaming

Window function

Introduction of ELK Technology Stack

Installation and use of ElasticSearch

Storm architecture analysis

Storm programming model, Tuple source code, concurrency analysis

Storm WordCount cases and Common Api Analysis

7. Spark core source code parsing

Spark source code compilation

Spark remote debug

Source code analysis of Spark task submission line process

Source Code Analysis of Spark Communication process

Source code analysis of SparkContext creation process

Source Code Analysis of Communication process between DriverActor and ClientActor

Source code analysis of Worker startup Executor process

Source Code Analysis of the Registration process of Executor to DriverActor

Source Code Analysis of the Registration process of Executor to Driver

DAGScheduler and TaskScheduler source code analysis

Source code analysis of Shuffle process

Source code analysis of Task execution process

Machine learning algorithm

1. Python and numpy library

A brief introduction to Machine Learning

Machine Learning and python

Python language-getting started

Python language-detailed explanation of data types

Python language-flow Control statement

Python language-function usage

Python language-modules and packages

Phthon language-object oriented

Python Machine Learning algorithm Library-numpy

Necessary Mathematical knowledge for Machine Learning-probability Theory

2. Implementation of common algorithms.

Knn Classification algorithm-algorithm principle

Knn Classification algorithm-Code implementation

Knn Classification algorithm-A case of handwriting recognition

Lineage regression Classification algorithm-algorithm principle

Lineage regression Classification algorithm-algorithm implementation and demo

Naive Bayesian Classification algorithm-algorithm principle

Naive Bayesian Classification algorithm-algorithm implementation

Naive Bayesian Classification algorithm-an Application case of Spam recognition

Kmeans clustering algorithm-algorithm principle

Kmeans clustering algorithm-algorithm implementation

Kmeans clustering algorithm-Geographic location clustering Application

Decision Tree Classification algorithm-algorithm principle

Decision Tree Classification algorithm-algorithm implementation

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.