Shulou(Shulou.com)06/01 Report--

Editor to share with you how to identify Baidu spiders in the construction of the website, I believe most people do not know much about it, so share this article for your reference. I hope you will gain a lot after reading this article. Let's learn about it!

1. Check UA. If the UA is wrong, you can directly determine the spiders that are not searched by Baidu. Currently, the published UA is:

Mobile UA:

Mozilla/5.0 (Linux;u;Android 4.2.2 witzhmurcn;) AppleWebKit/534.46 (KHTML,likeGecko) Version/5.1 Mobile Safari/10600.6.3 (compatible;Baiduspider/2.0;+ http://www.baidu.com/search/spider.html))

PC UA:

Mozilla/5.0 (compatible; Baiduspider/2.0;+ http://www.baidu.com/search/spider.html)

Added rendering UA:

Mobile UA:

Mozilla/5.0 (iPhone; CPU iPhone OS 91st likeMac OS X) AppleWebKit/601.1.46 (KHTML,like Gecko) Version/9.0 Mobile/13B143Safari/601.1 (compatible;Baiduspider-render/2.0; + http://www.baidu.com/search/spider.html))

PC UA:

Mozilla/5.0 (compatible;Baiduspider-render/2.0; + http://www.baidu.com/search/spider.html)

2. Reverse check IP

Webmaster can check IP through DNS to determine whether a spider comes from Baidu search engine. Verification methods vary according to different platforms. For example, the verification methods under the three platforms of linux/windows/os are as follows:

1) on the linux platform, you can use the host ip command to decrypt the ip to determine whether it comes from the Baiduspider crawl. The hostname of Baiduspider is named in the format of * .baidu.com or * .baidu.jp, and a person other than * .baidu.com or * .baidu.jp is impersonated.

2) on windows platform or IBM OS/2 platform, you can use the nslookup ip command to decrypt ip to determine whether it comes from Baiduspider crawling. Open the command processor and enter nslookup xxx.xxx.xxx.xxx (IP address) to parse the ip to determine whether the crawl comes from Baiduspider. The hostname of Baiduspider is named in the format of * .baidu.com or *. Baidu.jp, and non-* .baidu.com or *. Baidu.jp is impersonation.

3) on the mac os platform, you can use the dig command to decrypt the ip to determine whether it comes from the Baiduspider crawl. Open the command processor and enter dig xxx.xxx.xxx.xxx (IP address) to parse the ip to determine whether the crawl comes from Baiduspider. The hostname of Baiduspider is named in the format of * .baidu.com or *. Baidu.jp, and non-* .baidu.com or *. Baidu.jp is impersonation.

The above is all the contents of this article "how to identify Baidu spiders in website construction". Thank you for your reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.