Big data's basic learning

What is big data?

For example:

1. Commodity recommendation: question:

(1) how to store a large number of orders?

(2) how to calculate a large number of orders?

2. Weather forecast: question:

(1) how to store a large amount of weather data?

(2) how to calculate a large amount of weather data?

If you want to learn big data well, you'd better join a good learning environment. You can come to this Q Group 251956502 so that it is more convenient for everyone to learn, and you can also communicate and share materials together.

What is big data, the essence?

(1) data storage: distributed file system (distributed storage)

(2) data calculation: distributed computing

What is the relationship between Java and big data?

1. Hadoop: based on Java language development

2. Spark: based on Scala language, Scala based on Java language

The basis and route needed to learn from big data

1. The basis for learning big data:

Java fundamentals (JavaSE)-- > classes, inheritance, Igamot O, reflection, generics *

Linux Foundation (operation of Linux)-- > create files, directories, vi editors *

2. Learning route:

(1) Java Foundation and Linux Foundation

(2) Learning of Hadoop: architecture, principle, programming

(*) Phase I: HDFS, MapReduce, HBase (NoSQL database)

(*) Phase 2: data analysis engine-> Hive, Pig

Data acquisition engine-> Sqoop, Flume

(*) Phase III: HUE:Web management tools

ZooKeeper: HA for implementing Hadoop

Oozie: workflow engine

(3) study of Spark

(*) the first stage: Scala programming language

(*) second stage: Spark Core- > Computing based on memory and data

(*) the third stage: Spark SQL-> similar to the SQL statement in Oracle

(*) the fourth stage: Spark Streaming--- >

Carry out real-time calculation (flow calculation) such as waterworks

(4) Apache Storm: similar to Spark Streaming-> perform real-time calculation (flow calculation): for example: waterworks

(*) NoSQL:Redis memory-based database