Shulou(Shulou.com)06/03 Report--

The basis of learning big data

1. Java SE, EE (SSM)

90% of big data's frames are written by Java.

2 、 MySQL

SQL on Hadoop

3 、 Linux

Big data's framework is installed on the Linux operating system

What do you need to learn

. In the process of getting started learning big data, I have encountered learning, industry, lack of systematic learning route, systematic learning planning, welcome you to join my big data learning communication skirt: 251956502, skirt files have my big data learning manual, development tools, PDF documents and books, you can download them by yourself.

Big data offline analysis

General processing of Thum1 data (T: maybe 1 day, 1 week, 1 month, 1 year)

A, Hadoop: generally do not choose the latest version, stepping on the pit is difficult to solve

(common, HDES, MapReduce, YARN)

The idea of building environment and processing data

B, Hive: big data's data warehouse

Manipulate data by writing SQL, similar to sql in MySQL database

C, HBase: NOSQL database based on HDFS

Column-oriented storage

D. Collaboration framework:

Sqoop (Bridge: HDFS "=" RDBMS)

Flume: collect information from log files

E. Scheduling framework

Anzkaban

Understand: crotab (included with Linux)

Zeus (Alibaba)

Oozie (cloudera)

F, Frontier Framework extension:

Kylin, impala, ElasticSearch (ES)

Big data real-time analysis

Mainly based on spark framework

Scala:OOP (object-oriented programming) + FP (function is programming)

SparkCore: analogical MapReduce

SparkSQL: analogical hive

SparkStreaming: real-time data processing

Kafka: message queuing

Frontier Framework extension: flink

Alibaba: blink

Big data machine learning

Spark MLlib: machine Learning Library

Pyspark programming: the combination of Python and spark

Recommendation system

Python data analysis

Python machine learning

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.