Shulou(Shulou.com)06/02 Report--

Log segmentation

As the number of visits to the website increases, the single log file for Apache becomes larger and larger by default.

Log files take up a lot of disk space.

It is not convenient to view related information two kinds of log segmentation tools

Apache has its own rotatelogs segmentation tool.

Third-party tool cronolog Segmentation rotatelogs Segmentation tool experiment

(1) install httpd service.

[root@localhost] # yum install httpd-y. Omit the installation process

(2) after installation, we can check that the log file storage directory "/ var/log/httpd/" can see that there are no log files. When we start the service and check it again, the log file is generated.

[root@localhost ~] # ls / var/log/httpd/ View directory [root@localhost ~] # systemctl start httpd/ / enable services [root@localhost ~] # ls / var/log/httpd/ View directory access_log error_log / / log file [root@localhost ~] # systemctl stop firewalld.service / / turn off the firewall [root@localhost ~] # setenforce 0 / / turn off the enhanced security feature [root@localhost ~] #

(3) check the IP address.

[root@localhost] # ifconfig ens33: flags=4163 mtu 1500 inet 192.168.52.133 netmask 255.255.255.0 broadcast 192.168.52.255 inet6 fe80::3e1d:31ba:f66a:6f80 prefixlen 64 scopeid 0x20 ether 00:0c:29:27:1c:3f txqueuelen 1000 (Ethernet) RX packets 11004 bytes 14155417 RX errors 0 dropped 0 overruns 0 frame 0 TX packets 5374 bytes 360388 (351.9 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

(4) modify the configuration file (you can use "/" to query the keyword modification).

[root@localhost ~] # vim / etc/httpd/conf/httpd.confListen 192.168.52.133 logs/www.abc.com.error_%Y%m 80 / enable IPv4 snooping # Listen 80 / / comment IPv6 snooping ServerName www.abc.com:80 / / set domain name # ErrorLog "logs/error_log" ErrorLog | / usr/sbin/rotatelogs-l logs/www.abc.com.error_%Y%m % dlog 86400 "/ error log configuration # CustomLog" logs/access_log "combinedCustomLog" | / usr/sbin/rotatelogs-l logs/www.abc.com.access_%Y%m%dlog 86400 "combined / / access log configuration [root@localhost ~] # systemctl restart httpd / / restart service [root@localhost ~] #

(5) Let's first check the log file directory, and there is only the error log at this time. When we use the win10 host to visit the website and then view it, the access log is generated.

[root@localhost ~] # [root@localhost ~] # cd / var/log/httpd/ enter the log file storage directory [root@localhost httpd] # ls / / View access_log error_log www.abc.com.error_20191025log / / only error log [root@localhost httpd] # ls / / check again after accessing with the win10 host Access_log error_log www.abc.com.access_20191025log www.abc.com.error_20191025log / / access log generation [root@localhost httpd] #

(6) We use the "date-s" command to modify the system date one day later, restart the service to check the directory again, and generate a new day's log file.

[root@localhost httpd] # date-s 10 Universe 26 lsaccess_log error_log www.abc.com.access_20191025log www.abc.com.error_20191025log www.abc.com.error_20191026log Saturday, October 26, 192019, 00:00:00 CST [root@localhost httpd] # [root@localhost httpd] # lsaccess_log error_log www.abc.com.access_20191025log www.abc.com.error_20191025log [root@localhost httpd] # systemctl restart httpd [root@localhost httpd] # lsaccess_log error_log www.abc.com.access_20191025log www.abc.com.error_20191025log www.abc.com.error_20191026log [ Root@localhost httpd] # third-party tool cronolog experiment

(1) Let's install the httpd service first.

[root@localhost ~] # yum install httpd-y.mp / omit the installation process [root@localhost ~] #

(2) Mount the installation package of cronolog tools from the host to the Linux system through the Samba service.

[root@localhost ~] # mkdir / mnt/tools / / create a mount directory [root@localhost ~] # smbclient-L / / 192.168.100.50 / / check the shared Enter SAMBA\ root's password: OS= [Windows 10 Enterprise LTSC 2019 17763] Server= [Windows 10 Enterprise LTSC 2019 6.3] Sharename Type Comment -- IPC$ IPC remote IPC share Disk tools Disk Users Disk Connection to 192.168.100.50 failed (Error NT_STATUS_RESOURCE_NAME_NOT_FOUND) NetBIOS over TCP disabled-- no workgroup available [root@localhost ~] # mount.cifs / / 192.168.100.50/tools / mnt/tools / / Mount Password for root@ / / 192.168.100.50/tools: [root@localhost ~] # ls/ mnt/tools/ View awstats-7.6.tar.gz fiddler.exe john-1.8.0.tar.gzcronolog-1.6.2-14.el7.x86_64.rpm intellijideau2018.rar LAMP-C7extundelete-0.2.4.tar.bz2 jdk-8u191-windows-x64.zip [root@localhost ~] #

(3) install the cronolog tool directly with the "rpm" command.

[root@localhost ~] # cd / mnt/tools / / change the directory [root@localhost tools] # ls / / View awstats-7.6.tar.gz fiddler.exe john-1.8.0.tar.gzcronolog-1.6.2-14.el7.x86_64.rpm intellijideau2018.rar LAMP-C7extundelete-0.2.4.tar.bz2 jdk-8u191-windows -x64.zip [root@localhost tools] # rpm-ivh cronolog-1.6.2-14.el7.x86_64.rpm / / installation warning: cronolog-1.6.2-14.el7.x86_64.rpm: header V3 RSA/SHA256 Signature Key ID 352c64e5: NOKEY in preparation. # # [100%] upgrading / installing... 1:cronolog-1.6.2-14.el7 # # # [100%] [root@localhost tools] #

(4) check the command file storage directory.

[root@localhost tools] # which cronolog / / View the command file storage directory / usr/sbin/cronolog [root@localhost tools] #

(5) modify the configuration file of httpd service.

[root@localhost tools] # vim / etc/httpd/conf/httpd.confListen 192.168.52.133 etc/httpd/conf/httpd.confListen 80 / enable IPv4 snooping # Listen 80 / / comment IPv6 snooping ServerName www.abc.com:80 / / set the domain name # ErrorLog "logs/error_log" ErrorLog "| / usr/sbin/cronolog logs/www.abc.com.error_%Y%m%d.log" / / configure error log # CustomLog "logs/access_log" combined CustomLog | / usr/sbin/cronolog logs/www.abc.com.access_%Y%m%d.log "combined / / configure access log

(6) the test of the experimental results.

[root@localhost tools] # systemctl start httpd/ / enable httpd service [root@localhost tools] # systemctl stop firewalld.service / / turn off the firewall [root@localhost tools] # setenforce 0 / / turn off the enhanced security function [root@localhost tools] # ls / var/log/httpd/ View the log storage directory www.abc.com.error_20191025.log / / only error log [root@localhost tools] # ls / var/log/httpd/ View www.abc.com.access_20191025.log www.abc.com.error_20191025.log / / access log after access with win10 host to generate [root@localhost tools] # [root@localhost tools] # date-s 10-26-19 / / change system time Saturday, October 26, 2019, 00:00:00 CST [ Root@localhost tools] # systemctl restart httpd/ / restart Service [root@localhost tools] # ls / var/log/httpd/ check the log storage directory again www.abc.com.access_20191025.log www.abc.com.error_20191025.log www.abc.com.error_20191026.log / / generate a new log file [root@localhost tools] # AWStats log analysis system introduces the AWStats log analysis system

An open source log analysis system developed by Per language

It can be used to analyze the access log information of Apache, Samba, Vsftpd, IIS and other servers, combined with planned task services such as crond, to analyze and deploy AWStats log analysis system experiments on log contents on a regular basis.

(1) install httpd service and bind (named) service.

[root@localhost ~] # yum install bind httpd-y.mp / omit the installation process

(2) modify the main configuration file of the named service.

[root@localhost ~] # vim / etc/named.confoptions {listen-on port 53 {any;}; / / 127.0.0.1 change to any listen-on-v6 port 53 {:: 1;}; directory "/ var/named"; dump-file "/ var/named/data/cache_dump.db" Statistics-file "/ var/named/data/named_stats.txt"; memstatistics-file "/ var/named/data/named_mem_stats.txt"; recursing-file "/ var/named/data/named.recursing"; secroots-file "/ var/named/data/named.secroots"; allow-query {any;}; / / localhost changed to any

(3) modify the regional configuration file of the named service.

[root@localhost ~] # vim / etc/named.rfc1912.zoneszone "abc.com" IN {/ / add a domain name information type master; file "abc.com.zone"; allow-update {none;};}

(4) check the IP address.

[root@localhost named] # ifconfig ens33: flags=4163 mtu 1500 inet 192.168.52.133 netmask 255.255.255.0 broadcast 192.168.52.255 inet6 fe80::3e1d:31ba:f66a:6f80 prefixlen 64 scopeid 0x20 ether 00:0c:29:27:1c:3f txqueuelen 1000 (Ethernet) RX packets 14532 bytes 20210558 (19.2 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 6054 bytes 399142 (389.7 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

(5) reserve the right to copy a regional data configuration file of the named service and modify it.

[root@localhost ~] # cd / var/named/ change the directory [root@localhost named] # ls / / View data dynamic named.ca named.empty named.localhost named.loopback slaves [root@localhost named] # cp-p named.localhost abc.com.zone / / copy [root@localhost named] # vim abc.com.zone $TTL 1D @ IN SOA @ rname.invalid. (0; serial 1D; refresh 1H; retry 1W; expire 3H) Minimum NS @ A 127.0.0.1www IN A 192.168.52.133 / / Domain name resolution by adding IPv4

(6) start the named service.

[root@localhost named] # systemctl start named / / enable the service [root@localhost named] # systemctl stop firewalld.service / / turn off the firewall [root@localhost named] # setenforce 0 / / disable the enhanced security feature [root@localhost named] #

(7) modify the configuration file of httpd service.

[root@localhost named] # cd / etc/httpd/ [root@localhost httpd] # lsconf conf.d conf.modules.d logs modules run [root@localhost httpd] # vim conf/httpd.confListen 192.168.52.133 etc/httpd/ 80 / / enable IPv4 snooping # Listen 80 / / Log out IPv6 snooping # ServerName www.abc.com:80 / / configure domain name

(8) enable the service and view the directory where log files are stored

[root@localhost httpd] # systemctl start httpd.service / / enable the service [root@localhost httpd] # [root@localhost httpd] # ls / var/log/httpd/ View access_log error_log / / access log and error log file [root@localhost httpd] #

(9) use Samba service to mount the tool package from the host to the Linux system.

[root@localhost httpd] # mkdir / mnt/abc / / create a mount directory [root@localhost httpd] # smbclient-L / / 192.168.100.50 / / View shared Enter SAMBA\ root's password: OS= [Windows 10 Enterprise LTSC 2019 17763] Server= [Windows 10 Enterprise LTSC 2019 17763] Sharename Type Comment IPC$ IPC remote IPC share Disk tools Disk Users Disk Connection to 192.168.100.50 failed (Error NT_STATUS_RESOURCE_NAME_NOT_FOUND) NetBIOS over TCP disabled-- no workgroup available [root@localhost httpd] # mount.cifs / / 192.168.100.50/tools / mnt/abc / / Mount Password for root@//192.168. 100.50/tools: [root@localhost httpd] # ls / mnt/abc/ View the mount directory awstats-7.6.tar.gz fiddler.exe john-1.8.0.tar.gzcronolog-1.6.2-14.el7.x86_64.rpm intellijideau2018.rar LAMP-C7extundelete-0.2.4.tar.bz2 jdk-8u191-windows-x64.zip [root@localhost httpd] # Tar zxvf / mnt/abc/awstats-7.6.tar.gz-C / opt/ extract the package to the "/ opt/" directory [root@localhost httpd] # ls / opt/ / View the "/ opt/" directory awstats-7.6 rh [root@localhost httpd] # [root@localhost httpd] # mv / opt/awstats-7.6/ / usr/local/awstats / / move the package to the "/ usr/local/" directory Named "awstats" [root@localhost httpd] # ls / usr/local/ View awstats bin etc games include lib lib64 libexec sbin share src [root@localhost httpd] #

(10) enter the "/ usr/local/awstats/tools" directory

[root@localhost httpd] # [root@localhost httpd] # cd / usr/local/awstats/ [root@localhost awstats] # lsdocs README.md tools wwwroot [root@localhost awstats] # cd tools/ [root@localhost tools] # lsawstats_buildstaticpages.pl dolibarr maillogconvert.pl xsltawstats_configure.pl geoip_generator.pl nginxawstats_exportlib.pl httpd_conf urlaliasbuilder.plawstats_updateall.pl logresolvemerge.pl webmin

(11) configure awstats.

[root@localhost tools] #. / awstats_configure.pl-AWStats awstats_configure 1.0 (build 20140126) (c) Laurent Destailleur-This tool will help you to configure AWStats to analyze statistics forone web server. You can try to use it to let it do all that is possiblein AWStats setup, however following the step by step manual setupdocumentation (docs/index.html) is often a better idea. Above all if:- You are not an administrator user,- You want to analyze downloaded log files without web server,- You want to analyze mail or ftp log files instead of web log files,- You need to analyze load balanced servers log files,- You want to 'understand' all possible ways to use AWStats...Read the AWStats documentation (docs/index.html).-> Running OS detected: Linux BSD or Unix- > Check for web server installEnter full config file path of your Web server.Example: / etc/httpd/httpd.confExample: / usr/local/apache2/conf/httpd.confExample: C:\ Program files\ apache group\ apache\ conf\ httpd.confConfig file path ('none' to skip web server setup): > / etc/httpd/conf/httpd.conf / / specify httpd service profile-> Check and complete Web server config file'/ etc/httpd/conf/httpd.conf' Add 'Alias / awstatsclasses "/ usr/local/awstats/wwwroot/classes/"' Add 'Alias / awstatscss "/ usr/local/awstats/wwwroot/css/"' Add 'Alias / awstatsicons "/ usr/local/awstats/wwwroot/icon/"' Add 'ScriptAlias / awstats/ "/ usr/local/awstats/wwwroot/cgi-bin/"' Add''directive AWStats directives added to Apache config file.- > Update Model config file'/ usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf' File awstats.model.conf updated.- > Need to create a new config file? Do you want me to build a new AWStats config/profilefile (required if first install) [y Unip N]? Y / / OK-> Define config file name to createWhat is the name of your web site or profile analysis? Example: www.mysite.comExample: demoYour web site Virtual server or profile name: > www.abc.com / / specify domain name-> Define config file pathIn which directory do you plan to store your config file (s)? Default: / etc/awstatsDirectory path to store config file (s) (Enter for default): > / / specify awstats tool configuration file Direct enter defaults to "/ etc/awstats"-> Create config file'/ etc/awstats/awstats.www.abc.com.conf' Config file / etc/awstats/awstats.www.abc.com.conf created.- > Restart Web server with'/ sbin/service httpd restart'Redirecting to / bin/systemctl restart httpd.service- > Add update process inside a schedulerSorry Configure.pl does not support automatic add to cron yet.You can do it manually by adding the following command to your cron:/usr/local/awstats/wwwroot/cgi-bin/awstats.pl-update-config=www.abc.comOr if you have several config files and prefer having only one command:/usr/local/awstats/tools/awstats_updateall.pl nowPress ENTER to continue... / / enter to continue A SIMPLE config file has been created: / / etc/awstats/awstats.www.abc.com.confYou should have a look inside to check and change manually main parameters.You can then manually update your statistics for 'www.abc.com' with command: > perl awstats.pl-update-config=www.abc.comYou can also read your statistics for' www.abc.com' with URL: > http://localhost/awstats/awstats.pl?config=www.abc.comPress ENTER to finish... / / enter to complete [root@localhost tools] #

(12) modify the configuration file of the httpd service.

[root@localhost tools] # cd / etc/httpd/ [root@localhost httpd] # lsconf conf.d conf.modules.d logs modules run [root@localhost httpd] # vim conf/httpd.conf## This is to permit URL access to scripts/files in AWStats directory.# Options None AllowOverride None # Order allow,deny / / comment # Allow from all / / comment Require all granted / / add

(13) modify the configuration file of awstats log analysis system

[root@localhost httpd] # [root@localhost httpd] # cd / etc/awstats/ [root@localhost awstats] # lsawstats.www.abc.com.conf [root@localhost awstats] # vim awstats.www.abc.com.confLogFile= "/ var/log/httpd/access_log" / / specify the httpd access log file DirData= "/ var/lib/awstats" / / awstats file, which does not exist by default (need to be created)

(14) create the awstats file and restart the httpd service.

[root@localhost awstats] # [root@localhost awstats] # ls / var/lib/awstats / / View awstats file does not exist ls: unable to access / var/lib/awstats: does not have that file or directory [root@localhost awstats] # mkdir / var/lib/awstats / / create awstats file [root@localhost awstats] # ls / var/lib/awstats [root@localhost awstats] # [root@localhost awstats] # systemctl restart httpd.service / / restart the service [root@localhost awstats] #

(15) modify the dns server IP address of the win10 host, and use the win host to access the httpd service domain name.

(16) use the win10 host to visit the home page of awstats statistics, domain name: http://localhost/awstats/awstats.pl?config=www.abc.com. You can see that there are no statistics because the data is not updated.

(17) We go back to the Linux system, switch to "/ usr/local/awstats/tools/", and update the data.

[root@localhost awstats] # cd / usr/local/awstats/tools/ [root@localhost tools] # lsawstats_buildstaticpages.pl awstats_exportlib.pl dolibarr httpd_conf maillogconvert.pl urlaliasbuilder.pl xsltawstats_configure.pl awstats_updateall.pl geoip_generator.pl logresolvemerge.pl nginx webmin [root@localhost tools] #. / awstats_updateall.pl now / / Update data Running'"/ usr/local/awstats/wwwroot/cgi -bin/awstats.pl "- update-config=www.abc.com-configdir=" / etc/awstats "'to update config www.abc.comCreate/Update database for config" / etc/awstats/awstats.www.abc.com.conf "by AWStats version 7.6 (build 20161204) From data in log file" / var/log/httpd/access_log "... Phase 1: First bypass old records Searching new record...Searching new records from beginning of log file...Phase 2: Now process new records (Flush history on disk after 20000 hosts)... Jumped lines in file: 0Parsed lines in file: 106 Found 0 dropped records, Found 0 comments, Found 0 blank records, Found 1 corrupted records, Found 0 old records, Found 105 new qualified records. [root@localhost tools] #

(18) enter the statistics page in the win10 system again, and you already have the statistics.

(19) to facilitate the use of awstats service statistics, we can set up periodic tasks.

[root@localhost tools] # crontab-e / / Edit periodic tasks * / 5 * / usr/local/awstats/tools/awstats_updateall.pl now / / update data every five minutes [root@localhost tools] # you have an email in / var/spool/mail/root [root@localhost tools] #

(20) the domain name "http://localhost/awstats/awstats.pl?config=www.abc.com"" of our visit to the data statistics page is too long to remember. We can optimize the address of the web page and use "http://www.abc.com/aws.html"" to access it.

[root@localhost tools] # [root@localhost tools] # cd / var/www/html/ [root@localhost html] # vim aws.html [root@localhost html] # systemctl restart httpd.service / / restart the service

(21) enter the address "http://www.abc.com/aws.html"" in the win10 host browser and it will automatically go to the data statistics page.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.