In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-28 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Development >
Share
Shulou(Shulou.com)06/02 Report--
This article mainly explains "how to use Python tools to analyze Web server log files", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "how to use Python tools to analyze Web server log files" bar!
Lars is a Web server log toolkit written by Python. This means that you can use Python to backtrack (or parse the log in real time) with simple code and do whatever you want with the data: store it in a database, save it as a CSV file, or immediately use Python for more analysis.
Lars is another hidden gem written by Dave Jones. I first saw Dave demo lars in the local Python user group. A few years later, we started using it in the piwheels project to read Apache logs and insert rows into our Postgres database. When Raspberry sends users to download the Python package from piwheels.org, we record the file name, timestamp, system architecture (Arm version), release name / version, Python version, etc. Because it is a relational database, we can add these results to other tables to get more contextual information about the file.
You can install lars using the following methods:
$pip install lars
On some systems, the right way is sudo pip3 install lars.
First, find an Web access log and make a copy. You need to download the log file to your computer for operation. I used the Apache log in the example, but with some small (and intuitive) changes, you can use Nginx or IIS. On a typical Web server, you will find Apache logs in / var/log/apache2/, usually access.log, ssl_access.log (for HTTPS), or gzip compressed rotation log files, such as access-20200101.gz or ssl_access-20200101.gz.
First of all, what does the log look like?
81.174.152.222-[30/Jun/2020:23:38:03 + 0000] "GET / HTTP/1.1" 6763 "-" Mozilla/5.0 (X11; Ubuntu; Linux x8634; rv:77.0) Gecko/20100101 Firefox/77.0 "
This is a request that shows the source IP address, timestamp, request file path (in this case, home page /), HTTP status code, user agent (Firefox on Ubuntu), and so on.
Your log file will be filled with entries, not only for every open page, but also for every file and resource returned: every CSS stylesheet, JavaScript file and image, every 404 request, every redirect, every crawler. To get meaningful data from the log, you need to parse, filter, and sort entries. That's what Lars is for. This example opens a log file and prints the contents of each line:
With open ('ssl_access.log') as f: with ApacheSource (f) as source: for row in source: print (row)
It displays the following results for each log:
Row (remote_host=IPv4Address ('81.174.152.222'), ident=None, remote_user=None, time=DateTime (2020, 6, 30, 23, 38, 3), request=Request (method='GET', url=Url (scheme='', netloc='', path_str='/', params='', query_str='', fragment=''), protocol='HTTP/1.1'), status=200, size=6763)
It parses the log entries and puts the data into a structured format. The entry has become a named tuple namedtuple with attributes related to the entry data, so, for example, you can use row.status to access the status code and row.request.url.path_str to access the path:
With open ('ssl_access.log') as f: with ApacheSource (f) as source: for row in source: print (f'hit {row.request.url.path_str} with status code {row.status}')
If you only want to display 404 requests, you can do the following:
With open ('ssl_access.log') as f: with ApacheSource (f) as source: for row in source: if row.status = 404: print (row.request.url.path_str)
You may want to de-duplicate this data and print a separate number of 404 pages:
S = set () with open ('ssl_access.log') as f: with ApacheSource (f) as source: for row in source: if row.status = 404: s.add (row.request.url.path_str) print (len (s))
Dave and I have been working hard to expand piwheel's logger to include web page clicks, package searches, and so on, thanks to lars, which is not difficult. It doesn't tell us any answers about users. We still need to do data analysis, but it removes the complex and inconvenient file format and puts it in our database in a way that we can use.
Thank you for your reading, the above is the content of "how to use Python tools to analyze Web server log files". After the study of this article, I believe you have a deeper understanding of how to use Python tools to analyze Web server log files, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 257
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.