Shulou(Shulou.com)06/01 Report--

This article is about why spiders in seo's search engine crawl abnormalities. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Some web pages are of high quality and users can visit them normally, but search engine spiders cannot access and crawl normally, resulting in a lack of coverage of search results, which is a loss to both search engines and sites. Baidu calls this situation "crawling exception". For sites where a large number of content cannot be crawled normally, Baidu search engine will think that the site has defects in user experience, and reduce the evaluation of the site, which will be negatively affected in crawling, indexing and sorting to a certain extent. finally, it affects the traffic the site gets from Baidu.

Server connection exception

There are two situations when the server connection is abnormal: one is that the site is unstable and the search engine spider is temporarily unable to connect to the server of your site when it tries to connect to your website; the other is that the search engine spider has been unable to connect to the server of your website.

The reason for abnormal server connection is usually that your website server is running too large and overloaded. It is also possible that your website is not working properly, please check that the web servers of the website (such as apache, iis) are installed and running properly, and use your browser to check that the main pages can be accessed properly. Your website and host may also block the access of search engine spiders, you need to check the firewall of the site and host.

Network operator exception

Network operators are divided into telecom and Unicom, and search engine spiders cannot access your website through Telecom or Netcom. If this happens, you need to contact the network service provider, or buy space with a two-line service or purchase a cdn service.

DNS exception

A DNS exception occurs when the search engine spider cannot parse the IP of your site. It may be the wrong IP address of your website, or the domain name service provider has blocked the search engine spiders. Please use WHOIS or host to check whether the IP address of your website is correct and resolvable. If it is incorrect or unresolvable, please contact the domain name registrar to update your IP address.

IP blocking

The IP ban is to restrict the egress IP address of the network and prohibit users of the IP segment from accessing the content. Here, the search engine spider IP is specifically blocked. This setting is only required when your site does not want search engine spiders to visit. If you want search engine spiders to visit your site, please check to see if search engine spider IP has been mistakenly added to the relevant settings. It is also possible that the space service provider where your website is located has blocked Baidu IP, then you need to contact the service provider to change the settings.

UA blocking

UA is the user agent (User-Agent), and the server identifies the visitor through UA. When a website returns an abnormal page (such as 403500) or jumps to another page for the visit of the specified UA, it is blocked by UA. This setting is required only when your site does not want search engine spiders to visit, and if you want search engine spiders to visit your site, whether there is search engine spider UA in the useragent-related settings, and modify it in a timely manner.

Dead chain

The page is invalid, and the page that cannot provide any valuable information to the user is a dead link, including two forms: agreement dead chain and content dead chain.

Dead chain of agreement

The TCP protocol status / HTTP protocol status of the page clearly indicates the dead chain, such as 404,403,503 states, and so on.

Content dead chain: the server returns a normal status, but the content has been changed to an information page that does not exist, has been deleted, or requires permissions that have nothing to do with the original content.

For the dead chain, we suggest that the site use the protocol dead chain and submit it to Baidu through the Baidu webmaster platform-the dead chain tool, so that Baidu can find the dead chain more quickly and reduce the negative impact on users and search engines.

Abnormal jump

Redirecting a network request to another location is a jump. Abnormal jumps refer to the following situations:

1) currently, the page is invalid (content has been deleted, dead link, etc.). Jump to the previous directory or home page directly. Baidu recommends that the webmaster delete the entry hyperlink of the invalid page.

2) Jump to error or invalid page

Note: for those who jump to other domain names for a long time, such as changing the domain name of a website, Baidu recommends using the 301 jump protocol to set it.

Other exceptions:

1) for the exception of Baidu refer: the web page returns behavior different from normal content for the refer from Baidu.

2) for the exception of Baidu ua: the behavior that the web page returns to Baidu UA is different from the original content of the page.

3) JS jump exception: the JS jump code that Baidu cannot recognize is loaded on the web page, which causes the user to jump after entering the page through the search results.

4) accidental ban caused by excessive pressure: Baidu will automatically set a reasonable crawling pressure according to the size of the site, traffic and other information. However, in abnormal circumstances, such as abnormal pressure control, the server will be protected and occasionally blocked according to its own load. In this case, please return 503 in the return code (which means "Service Unavailable"), so that the search engine spider will try to grab the link after a while, and if the site is free, it will be crawled successfully.

Thank you for reading! This is the end of the article on "Why seo search engine spiders have crawling anomalies". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.