Shulou(Shulou.com)06/03 Report--

This article will explain in detail how to solve the problem of baidu simulation crawling failure of the old domain name. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Recently, some students have reported that his domain name has been shown to fail when he uses Baidu simulation crawling, and he has tried N methods. In recent days, I also encountered the same problem when I went to the station. Baidu background verification can be passed, and under normal circumstances, simulated crawling can also be passed, but simulated crawling failed.

Because I used Laomi, when I clicked on the crawl details, I found that Baidu grabbed the IP address of the original domain name.

Because I know that Baidu has its own DNS server in order to improve the access efficiency of the crawler, so I clicked to update IP, but I tried it N times. Previous experience is that the basic click update is OK.

Then I took the initiative to submit the URL and sitemap map of the whole site, which turned out to be normal when I simulated the crawl the next day.

Conclusion, as long as the active submission can be submitted successfully and the sitemap data can be normally extracted by Baidu, then the subsequent Baidu crawl is no problem. So when you encounter the same problem, you can try to actively submit and submit sitemap to update your IP.

So at which stages will such problems occur?

It appears when the domain name changes the server IP.

You will meet when the old corn is new online.

There is also a point to pay attention to, that is, for Baidu, because the submission of sitemap is limited, it seems that each site has 50000 entries, so it is only 5 million per site, so it is best to submit 50000 entries each. Of course, small sites don't have to pay attention to this.

Because Shenma's sitemap submission limit is 10000, of course, the higher the rating, the more the number, so some egg pain, this case can only be submitted separately by each search engine.

On how to solve the old domain name baidu simulation crawl failure to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.