Shulou(Shulou.com)06/03 Report--

First day

Introduction of 2.Hadoop Framework and search Technology system

1. Big data's typical characteristics and distributed Development difficulties

2.Hadoop framework introduction and search technology system introduction 3.Hadoop version and characteristics introduction HDFS distributed file system architecture of 4.Hadoop core module introduction of Yarn operating system architecture of 5.Hadoop core module introduction of 6.Linux security disable settings and JDK installation explanation of 7.Hadoop pseudo-distributed environment deployment HDFS part 8.Hadoop pseudo-distributed environment deployment Yarn and MR part common errors in the use of 9.Hadoop environment Explanation of general settings and auxiliary functions of collective 10.Hadoop environment (-)

Explanation of General Settings and Auxiliary functions in 11.Hadoop Environment (2) matters needing attention for deploying Eclipse plug-ins in 12.Windows Environment

Introduction of 2.Hadoop Framework and search Technology system

1.hadoop introduction

-"official website: http://hadoop.apache.org

-"three major distributions of hadoop Business

-"Apache -" apache

-"cloudera -" CDH

-"hostonwork -" HDP

-"distributed

-"crawler.

-"Storage (with hard disk, but a single machine is limited) & processing analysis

-"Quick query

-"calculate separately and merge the results

-"google-" Mapreduce thesis

-"map

-"reduce

-"HDFS file system is different from database

-"HBase

-"Technical system of search engine

-"data acquisition

-"(external network, Internet crawling data)

-"Database

-"data storage -" HDFS&Hbase

-"yarn operating system

-"data calculation

-"sql real-time query (message queuing, monitoring system)

-"Auxiliary frameworks such as zookeeper

-"generate index and search index (product recommendation is related to the information you usually search)

-"return a front-end user

-"offline system -" hadoop biosphere

-"data acquisition

-"(external network, Internet crawling data)

-"Cloud Stora

-"full or incremental import (synchronized to hbase, sql statement)

-"complex offline processing process (job operation, business logic, table join, field merging)

-"mapreduce (to update full or incremental data)

-"other frameworks implement real-time data updates

In this way, my entire data change can be updated to the search engine in seconds.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.