Shulou(Shulou.com)06/03 Report--

Apache log segmentation

There are two ways of log segmentation, the first is the implementation of the rotatelogs segmentation tool that comes with apache, and the second is the third party tool cronnolog segmentation

Apache's own partition tool rotatelogs [root@client ~] # yum install bind httpd-y [root@client ~] # cd / usr/sbin/ [root@client sbin] # ls rotat*rotatelogs// installs apache to see the configuration apache master configuration file under the commands that can be used by the system Open the service [root@client sbin] # vim / etc/httpd/conf/httpd.conf Listen 192.168.136.128 Listen 80 / / listen to your local address # Listen 80 / / comment out the listening port of ipv6 # If your host doesn't have a registered DNS name, enter its IP address here.ServerName www.kgc.com:80 / / change the original to your defined domain name And open [root@client httpd] # systemctl stop firewalld.service [root@client httpd] # setenforce 0 [root@client httpd] # systemctl start httpd [root@client httpd] # ls / / access_log error_ log [root @ client httpd] # cat access_log / / access log is empty [root] # cat error_log / / error log files, which are used by PID processes. Does not represent some more errors [Wed Oct 23 21 18 pid 18 pid 24.917418 2019] [core:notice] [pid 4429] SELinux policy enabled Httpd running as context system_u:system_r:httpd_t: s0 [Wed Oct 23 21 pid 18 suexec:notice 24.918499 2019] [suexec:notice] [pid 4429] AH01232: suEXEC mechanism enabled (wrapper: / usr/sbin/suexec) [Wed Oct 23 21 18 suexec:notice 24.938959 2019] [lbmethod_heartbeat:notice] [pid 4429] AH02282: No slotmem from mod_ heartbroken monitor [Wed Oct 23 21 pid 1824.941240 2019] [mpm_prefork:notice] [pid 4429] AH00163: Apache / 2.4.6 (CentOS) configured-- resuming normal operations [Wed Oct 23 21 resuming normal operations 18bank 24.941268 2019] [core:notice] [pid 4429] AH00094: Command line:'/ usr/sbin/httpd-D FOREGROUND' go to the client segment to test it

Go back to the server and take a look at the access log. if there are more and more visitors, there will be more and more logs, and we can't see them, so we need to split [root@client httpd] # cat access_log 192.168.136.134-[23/Oct/2019:21:24:55 + 0800] "GET / favicon.ico HTTP/1.1" 404209 "-" Mozilla/5.0 (Windows NT 10.0; Win64). X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240 "192.168.136.134-[23/Oct/2019:21:24:55 + 0800]" GET / HTTP/1.1 "403 4897"-"" Mozilla/5.0 (Windows NT 10.0; Win64 X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240 "192.168.136.134-[23/Oct/2019:21:24:56 + 0800]" GET / noindex/css/bootstrap.min.css HTTP/1.1 "19341" http://192.168.136.128/"Mozilla/5.0 (Windows NT 10.0; Win64 X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240 "192.168.136.134-[23/Oct/2019:21:24:56 + 0800]" GET / images/apache_pb.gif HTTP/1.1 "2326" http://192.168.136.128/"Mozilla/5.0 (Windows NT 10.0; Win64 X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240 "192.168.136.134-[23/Oct/2019:21:24:56 + 0800]" GET / noindex/css/open-sans.cs Log Segmentation configuration [root@client httpd] # vim / etc/httpd/conf/httpd.conf # ErrorLog "logs/error_log" / / Let's comment out the original, online skills You can recover if you are wrong. ErrorLog "| / usr/sbin/rotatelogs-l logs/www.kgc.com.error_%Y%m%dlog 86400" / / find this line, / enter the pipe symbol "|" follow the absolute path of your system apache command, and give the name www.kgc.vom,%Y%m%d fixed format to represent the year, month and day. 86400 represents the time of day 86400 seconds CustomLog "| / usr/sbin/rotatelogs-l logs/www.kgc.com.access_%Y%m%dlog 86400" combined / / it's the same here. Find this line to verify log segmentation.

[root@client httpd] # systemctl stop httpd

[root@client httpd] # systemctl start httpd

[root@client httpd] # ls

Access_log error_log www.kgc.com.error_20191023log / / our log for the day was split out.

[root@client httpd] # date-s 10Universe 24

Thursday, 24 October 2019, 00:00:00 CST

[root@client httpd] # systemctl stop httpd

[root@client httpd] # systemctl start httpd

[root@client httpd] # ls

Access_log error_log www.kgc.com.error_20191023log www.kgc.com.error_20191024log / / We changed the time, and the log was split out.

The log of third-party tools is divided. I have a bag. Private message I [root@localhost httpd] # smbclient-L / / 192.168.100.3 / / remember that your host Vmnet1 network card should be set to 192.168.100.3Enter SAMBA\ root's password: OS= [Windows 10 Enterprise 17763] Server= [Windows 10 Enterprise 17763] Sharename Type Comment-ADMIN$ Disk Remote administration C$ Disk default share D$ Disk default share E$ Disk default share F$ Disk default share G$ Disk default share IPC$ IPC remote IPC LAMP Disk LAMP-C7 Disk share Disk Users Disk Connection to 192.168.100.3 failed (Error NT_STATUS_RESOURCE_NAME_NOT_FOUND) NetBIOS over TCP disabled-- no workgroup available [root@localhost httpd] # cd ~ [root@localhost ~] # mkdir / abc/ / create mount point [root@localhost] # mount.cifs / / 192.168.100.3/LAMP-C7 / abc/ hostel The files of the host are mounted to our mount point Password for root@//192.168.100.3/LAMP-C7: [root@localhost ~] # cd / abc/ [root@localhost abc] # lsapr-1.6.2.tar.gz Discuz_X2.5_SC_UTF8.zip mysql-5.6.26.tar.gzapr-util-1.6.0.tar.gz fiddler.exe Php-5.6.11.tar.bz2awstats-7.6.tar.gz httpd-2.4.29.tar.bz2cronolog-1.6.2-14.el7.x86_64.rpm LAMP-php5.6.txt [root@localhost abc] # rpm-ivh cronolog-1.6.2-14.el7.x86_64.rpm / / install this third-party partition toolkit warning: cronolog-1.6.2-14.el7.x86_64.rpm: header V3 RSA/SHA256 Signature Key ID 352c64e5: NOKEY in preparation. # # [100%] upgrading / installing... 1:cronolog-1.6.2-14.el7 # # # [root@localhost abc] # cd / usr/sbin/ [root@localhost sbin] # ls cronolog* / / check the command cronolog third-party tool log partition [root@localhost sbin] # vim / etc/httpd/conf/httpd.conf ErrorLog "| / usr/sbin/cronolog logs/www.kgc.comerror_%Y%m%dlog" / / do not add-l and no 86400CustomLog " | | / usr/sbin/ cronolog logs/www.kgc.comaccess_%Y%m%dlog "combined [root@localhost sbin] # systemctl stop httpd [root@localhost sbin] # systemctl start httpd [root@localhost sbin] # ls / var/log/httpd/access_log error_log www.kgc.comerror_ 2019191023 [root @ localhost sbin] # date-s 10 / 242019 Thursday, 24 October 2010 00:00:00 CST [root@localhost sbin] # ls / var/log/httpd/access_log www.kgc | Comerror _ 20191023logerror_log www.kgc.comerror_20191024log log analysis let's change to another virtual machine to do it. Configure DNS main configuration file [root@localhost ~] # yum install bind httpd-y / / install DNS,Apache package [root@localhost ~] # vim / etc/named.conf options {listen-on port 53 {any }; / / change to any listen-on-v6 port 53 {:: 1;}; directory "/ var/named"; dump-file "/ var/named/data/cache_dump.db"; statistics-file "/ var/named/data/named_stats.txt"; memstatistics-file "/ var/named/data/named_mem_stats.txt" Recursing-file "/ var/named/data/named.recursing"; secroots-file "/ var/named/data/named.secroots"; allow-query {any;}; / / change to any configuration DNS zone configuration file

[root@localhost ~] # vim / etc/named.rfc1912.zones

Zone "kgc.com" IN {/ / defines a domain name

Type master

File "kgc.com.zone"; / / defines the zone data configuration file

Allow-update {none;}

}

Zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0

Configure the DNS zone data configuration file [root@localhost ~] # cd / var/named/ [root@localhost named] # cp-p named.localhost kgc.com.zone [root@localhost named] # vim kgc.com.zone $TTL 1D @ IN SOA @ rname.invalid. (0; serial 1D; refresh 1H; retry 1W; expire 3H) Minimum NS @ A 127.0.0.1www IN A 192.168.136.135 / / add your local address configuration Apache main configuration file [root@localhost named] # vim / etc/httpd/conf/httpd.conf Listen 192.168.136.135:80#Listen 80ServerName www.kgc.com:80 [root@localhost named] # systemctl stop firewalld.service [root@localhost named] # setenforce 0 [root@localhost named] # systemctl start httpd to the client for testing

Log analysis and toolkit, private trust me if you need it. [root@localhost httpd] # mkdir / abc/ / create mount point [root@localhost httpd] # mount.cifs / / 192.168.100.3/LAMP-C7 / abc/ / mount to abcPassword for root@//192.168.100.3/LAMP-C7: [root@localhost httpd] # cd / abc/ [root@localhost abc] # lsapr-1.6.2.tar.gz Discuz_X2.5_SC_UTF8.zip mysql-5.6 .26.tar.gzapr-util-1.6.0.tar.gz fiddler.exe php-5.6.11.tar.bz2awstats-7.6.tar.gz / / this package is our log analysis toolkit httpd-2.4.29.tar.bz2cronolog-1.6.2-14.el7.x86_64.rpm LAMP-php5.6.txt decompression Move, using the script awstats that we just created To configure apache Let it load the log analysis of awstats in the statistical module [root@localhost abc] # tar zxvf awstats-7.6.tar.gz-C / opt/ jie'ya and decompress it to awstats-7.6/awstats-7.6/tools/awstats-7.6/tools/awstats_buildstaticpages.plawstats-7.6/tools/awstats_updateall.pl [root@localhost abc] # cd / opt/ [root@localhost opt] # lsawstats-7.6 rh [root@localhost opt] # under OPT Mv awstats-7.6/ / usr/local/awstats / / move this packet to / usr/local and name it awstats [root@localhost opt] # lsrh [root@localhost opt] # cd / usr/local/ View [root@localhost local] # lsawstats bin etc games include lib lib64 libexec sbin share src path and domain name for apache in the tool [root@localhost local] # cd awstats/ [root@localhost awstats] # lsdocs README.md tools wwwroot [root@localhost awstats] # cd tools/ [root@localhost tools] # lsawstats_buildstaticpages.pl dolibarr maillogconvert.pl xsltawstats_configure.pl geoip_generator.pl nginxawstats_exportlib.pl httpd_conf urlaliasbuilder.plawstats_updateall.pl logresolvemerge.pl webmin [root@localhost tools] #. / awstats_configure.pl > / etc/httpd/conf/httpd.conf / / write the apache path file (required if first install) [ycompn]? Y / / do you want to create a new apache file > www.kgc.com / / enter your domain name-> Define config file pathIn which directory do you plan to store your config file (s)? Default: / etc/awstats / / the configuration file related to your domain name will generate Directory path to store config file (s) (Enter for default): > / / directly enter Press ENTER to continue... under this path / / enter directly > http://localhost/awstats/awstats.pl?config=www.kgc.com / / this path is the log analysis page Press ENTER to finish... that it gives you. / / directly enter to check whether the configuration apache has loaded these modules [root@localhost tools] # cd / etc/httpd/conf [root@localhost conf] # vim httpd.conf Alias / awstatsclasses "/ usr/local/awstats/wwwroot/classes/" Alias / awstatscss "/ usr/local/awstats/wwwroot/css/" Alias / awstatsicons "/ usr/local/awstats/wwwroot/icon/" ScriptAlias / awstats/ "/ usr/local/awstats/wwwroot/cgi-bin/" # This is to permit URL access to scripts/files in AWStats directory. / / Directory site Options None AllowOverride None / / related access control # Order allow,deny / / comment out these two lines # Allow from all Require all granted / / add so that everyone can access the analyzed data and put it in the Apache access log

[root@localhost etc] # cd / etc/awstats/

[root@localhost awstats] # ls

This is the file awstats.www.kgc.com.conf / / generated just now.

[root@localhost awstats] # ls / var/log/httpd/ / /

Access_log error_log

[root@localhost awstats] # vim awstats.www.kgc.com.conf

LogFile= "/ var/log/httpd/access_log" / / change the original to access_log and analyze the log

DirData= "/ var/lib/awstats" / / this does not need to be changed. The data we analyzed are put in this file.

Create an awstats directory to make the system recognize that the analysis data is put in [root@localhost awstats] # cd / var/lib [root@localhost lib] # ls awstats*ls: cannot access awstats*: without that file or directory [root@localhost lib] # mkdir awstats [root@localhost lib] # lsAccountsService dav hyperv net-snmp rpcbind tpmalsa dbus initramfs NetworkManager rpm tunedalternatives Dhclient ipa-client nfs rpm-state udisks2authconfig dnsmasq iscsi ntp rsyslog upowerawstats / / this is flatpak libvirt [root@localhost lib] # systemctl restart httpd / / restart the service to test it on the client side. It's just an analysis page.

Enter this URL http://www.kgc.com/awstats/awstats.pl?config=www.kgc.com

Update data to count [root@localhost lib] # cd / usr/local/awstats/ [root@localhost awstats] # lsdocs README.md tools wwwroot [root@localhost awstats] # cd tools/ [root@localhost tools] # lsawstats_buildstaticpages.pl dolibarr maillogconvert.pl xsltawstats_configure.pl geoip_generator.pl nginxawstats_exportlib.pl httpd_conf urlaliasbuilder.plawstats_updateall.pl logresolvemerge.pl webmin [root@localhost tools] # . / awstats_updateall.pl now / / just execute it directly Remember to add now's latest Running'"/ usr/local/awstats/wwwroot/cgi-bin/awstats.pl"-update-config=www.kgc.com-configdir= "/ etc/awstats"'to update config www.kgc.comCreate/Update database for config "/ etc/awstats/awstats.www.kgc.com.conf" by AWStats version 7.6 (build 20161204) From data in log file "/ var/log/httpd/access_log"... Phase 1: First bypass old records Searching new record...Searching new records from beginning of log file...Phase 2: Now process new records (Flush history on disk after 20000 hosts)... Jumped lines in file: 0Parsed lines in file: 485Found 0 dropped records, Found 0 comments, Found 0 blank records, Found 1 corrupted records, Found 0 old records, Found 484new qualified records. Go back to the client and test if you can count the data.

All the data collection, we have to execute that script every time. Let's do a periodic scheduled task here and have it automatically execute [root@localhost tools] # crontab-eSplash 5 * / usr/local/awstats/tools/awstats_updateall.pl now// every five minutes per hour, every day, every week, every month, to execute this script [root@localhost tools] # systemctl start crond / / Open the optimized web address [root@localhost tools] # cd / var/www//html/ you are in / var/. There is an email [root@localhost html] # ls [root@localhost html] # vim aws.html in spool/mail/root to test the optimized URL on the client.

That's all we have. Thank you for watching.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.