Shulou(Shulou.com)06/01 Report--

Hodoop in how to add online logs, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Step 1: receive the task of integrating online logs and start the analysis. The results are as follows:

1. Online logs are stored according to time, and at first you just think that you can add them directly. It was found that the actual business began in early September. In October, two fields were added and a log file was re-established. At present, there are two log files. The latter, in addition to adding two fields on the original basis, decided to enter the database first and store the data from the two logs into hive.

Add a scheduled command to synchronize the logs to the S3 server, using the following script:

#! / bin/sh

#

# put log to s3

# Author Lcm

# date 2016-10-27

#

# dateYMD= `date-d yesterday +% Ymure% mmury% d`

# dateYM= `date-d yesterday +% Ymuri% m`

# dateMD= `date-d yesterday +% mmurf% d`

DateYMD='2016-09 murmur 1

DateYM='2016-09'

DateMD='09-'$1

# 2016-09-2016-10-07

OLD_LOG_PATH='/home/centos/tools/apache-tomcat-8080/igp_logs/reg/'$dateYM'/reg1-'$dateMD'*.log'

# 2016-10-08

# OLD_LOG_PATH='/home/centos/tools/apache-tomcat-8080/igp_logs/reg1/'$dateYM'/reg1-'$dateMD'*.log'

OLD_LOG_PATH='/home/centos/tools/apache-tomcat-8080/igp_logs/reg1/'$dateYM'/reg1-'$dateMD'*.log'

NEW_LOG_PATH='/home/centos/tools/s3/s3cmd/s3cmddata/reglog'$dateMD'.log'

TAR_LOG_PATH='/home/centos/tools/s3/s3cmd/s3cmddata/reglog'$dateMD'.gz'

Cat $OLD_LOG_PATH > $NEW_LOG_PATH

If [- f "$NEW_LOG_PATH"]; then

Gzip-c $NEW_LOG_PATH > $TAR_LOG_PATH

/ home/centos/tools/s3/s3cmd put $TAR_LOG_PATH 's 3vlGemobi _ Gemobi _

Rm $NEW_LOG_PATH

Rm $TAR_LOG_PATH

Fi

The command to add a regular hive to the current server is mainly to store the log of the previous day in Hive. Before timing, you need to manually store the previous data into the database. The commands for operation are as follows: view the data source, download it from S3 to local, store the data of local files into hive, check whether the partition has been added successfully, and check whether the data has been added successfully.

1. / data/s3cmd/s3cmd ls-l s3://ngemobi/logs/igp_logs/reg/dt=2016-11-07 /

2. / data/s3cmd/s3cmd get s3://ngemobi/logs/igp_logs/reg/dt=2016-11-08/reglog11-08.gz

3. LOAD DATA LOCAL INPATH'/ data/s3data/reglog11-08' INTO TABLE appname_reg_logs partition (dt='2016-11-08')

4 、 show partitions appname_reg_logs

5. Select count (*) from appname_reg_logs where dt='2016-11-08'

Store the new and all Android ID (aid_all) added every day (aid_byday) into two Hive tables respectively. There is a sequential relationship here. The first step is to update the aid_all table after getting the new ones every day.

There's a logical relationship here.

New data for each day = all data for the day-existing in the aid_all table [all stored in the aid_ all table]

All Android id: update after all data processing today

Add manual code every day:

Insert into appname_androidid_byday partition (dt='2016-11-06') select distinct (tmp.reg_aid) from (selectappname_androidid_all.appname_androidid all_aid,appname_reg_logs.dt reg_dt, appname_reg_logs.appname_androidid reg_aidfromappname_reg_logsleft outer join appname_androidid_allonappname_reg_logs.appname_androidid=appname_androidid_all.appname_androidid) tmpwheretmp.reg_dt='2016-11-06' and tmp.all_aid is null

Every day, all aid_all is stored in the database:

Insert into appname_androidid_all partition (dt='2016-11-05') selectappname_androidid from appname_androidid_byday where dt='2016-11-05'

Get the summation file from the existing Hive table, and then read and store the data of the file into the Mysql table

1. Daily active users enter the database

Select count (*) as LOGS_GETERR_10001, errlog_logtype, errlog_plugin_version from get_error_logs where dt='seDate' and hour='seHour' group by errlog_logtype, errlog_plugin_version

2. The data added daily is stored in the database.

Select count (distinct tmp.reg_aid) as LOGS_APPNAMEADD_10002,tmp.reg_apkname, tmp.reg_country, tmp.reg_apkversioncode from (select appname_androidid_all.appname_androidid all_aid, appname_reg_logs.dt reg_dt, appname_reg_logs.appname_apkname reg_apkname, appname_reg_logs.appname_country reg_country, appname_reg_logs.appname_apkversioncode reg_apkversioncode) Appname_reg_logs.appname_androidid reg_aid from appname_reg_logs left outer join appname_androidid_all on appname_reg_logs.appname_androidid=appname_androidid_all.appname_androidid) tmp where tmp.reg_dt='2016-11-03 'tmp.all_aid is null group by tmp.reg_apkname,tmp.reg_country,tmp.reg_apkversioncode

3. Keep the data in storage every day.

Select count (distinct tmp.reg_aid) asLOGS_APPNAMEADD_10003,tmp.reg_apkname, tmp.reg_country, tmp.reg_apkversioncode from (select appname_androidid_all.appname_androidid all_aid, appname_androidid_all.dt all_dt, appname_reg_logs.appname_apknamereg_apkname, appname_reg_logs.appname_countryreg_country, appname_reg_logs.appname_apkversioncode reg_apkversioncode, appname_reg_logs.dt reg_dt Appname_reg_logs.appname_androidid reg_aid from appname_reg_logs left outer join appname_androidid_all on appname_androidid_all.dt='2016-11-02'and appname_reg_logs.appname_androidid=appname_androidid_all.appname_androidid) tmp wheretmp.reg_dt='2016-11-03' and tmp.all_aid is not nullgroup bytmp.reg_apkname,tmp.reg_country,tmp.reg_apkversioncode

Read the data from the mysql table and write the corresponding code for display and query

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.