Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to submit the website 404 dead chain through shell script under Apache. The article is very detailed and has certain reference value. Interested friends must finish reading it!

Website operators must be familiar with the concept of dead chain, some data deletion or page revision of the site are easy to create a dead chain, affecting the user experience, too much dead chain will also affect the overall weight or ranking of the site.

Baidu webmaster platform provides a dead chain submission tool, which can submit the dead chain (protocol dead chain, 404 pages) of the website, which can quickly delete the dead chain and help the site SEO optimization. It is too troublesome to manually fill in the dead chain one by one in the documents that submit the dead chain, and we advocate complex automation in our work, so in this article we communicate and share the Apache service to organize the dead chain of the website through shell script, so it is easy for us to submit.

、

1. Configure Apache record search engine

Apache is the most mainstream web service in website construction, but the log file of apache does not record the crawler of Baidu, Google and other major search engines by default, so we need to set up the configuration file of Apache first.

Locate the configuration file httpd.conf for Apache, and find the following two lines in the configuration file:

CustomLog "logs/access_log" common # CustomLog "logs/access_log" combined

Common is used by default, so here we just need to comment out the # before the line common, and then remove the # before the line combined. Then save and restart the Apache service.

Note: if you add multiple sites to your server, each site has a separate profile, then we only need to set the CustomLog entry in the configuration file of the corresponding site, for example:

Vim / usr/local/apache/conf/vhost/www.chanzhi.org.conf ServerAdmin [email protected] DocumentRoot "/ data/wwwroot/www.chanzhi.org" ServerName www.chanzhi.org ServerAlias chanzhi.org ErrorLog "/ data/wwwlogs/www.chanzhi.org_error_apache.log" CustomLog "/ data/wwwlogs/www.chanzhi.org_apache.log" combined SetOutputFilter DEFLATE Options FollowSymLinks ExecCGI Require all granted AllowOverride All Order allow,deny Allow from all DirectoryIndex index.html index.php

The following is the site logging format before and after configuration:

Before configuration:

After configuration:

two。 Write shell scripts

We use the shell script to obtain the crawl records of the specified crawlers in the website log, and then summarize them into a file for later use. The code is as follows, such as saving as deathlink.sh

#! / bin/bash# initialization variable # defines spider UA information (default is Baidu spider) date of the day before UA='+ http://www.baidu.com/search/spider.html'# (apache log) DATE= `date +% Y%m%d-d "1 day ago" `# defines log path logfile=/data/wwwlogs/www.chanzhi.org_apache.log-$ {DATE} .log # defines the storage path of dead-chain files deathfile=/data/wwwroot / www.chanzhi.org/deathlink.txt# defines the website access address website= http://www.chanzhi.org# Analysis Log and saves the dead-link data for url in `awk-v str= "${UA}"'$9pm = "404" & $15~str {print $7}'${logfile}'do grep-Q "$url" ${deathfile} | | echo ${website} ${url} > > ${deathfile} done

When using the script, you can adjust the path and fields according to the situation of your server, and then execute the script:

Bash deathlink.sh

3. Submit dead chain

When the above script is executed, a file containing all acquired 404 page links is generated in the specified directory, one line for each connection. For example:

Finally, on the webmaster platform submission dead-link page, enter your own dead-link file address, for example:

After the examination and approval, Baidu will delete the invalid links that have been included, in order to avoid the adverse impact of invalid page links on the site.

The above is all the contents of the article "how to submit the website 404 Dead chain through shell script under Apache". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.