Shulou(Shulou.com)06/01 Report--

How to analyze Mediapartners-Google rover, I believe that many inexperienced people do not know what to do about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

After looking at the log, I found several items like this:

2010-07-27 16:03:33 W3SVC1129513256 221.192.135.26 GET / a/view/18691.html-80-203.208.60.251 Mediapartners-Google 20000

What is Mediapartners-Google? search for it. It turns out to be Google Adsense's rover.

The following content is from the webmaster support Center

I want some pages of the site not to be indexed, but to display AdSense ads on those pages. Can you meet my request?

Sure. You can disable (Disallow) all roaming except Mediapartners-Google on these pages. This ensures that web pages are not indexed, but allows the Googlebot-MediaPartners rover to analyze them to determine whether advertisements are displayed. Googlebot-MediaPartners roaming does not share web pages with other Google user agents. For example, you can use the following entries:

User-Agent:*

Disallow:/folder1/

User-Agent:MediaPartners-Google

Allow:/folder1/

How to intercept Googlebot?

Intercept Googlebot

Google uses multiple user-agent. You can block access to a web page by including the rover name in the User-Agent line of the entry. Blocking Googlebot will intercept all rovers that start with "Googlebot".

Googlebot: crawling pages from our website index and news index

Googlebot-Mobile: crawling web pages for our mobile index

Googlebot-Image: crawl web pages for our image index

Mediapartners-Google: crawl the web page to determine the content of the AdSense. We will use this rover to crawl your site only if AdSense ads are displayed on your site.

Adsbot-Google: crawl web pages to measure the quality of AdWords target pages. We will use this rover only if you use AdWords to advertise your website. See more information about this rover and how to perform partial blocking on your website.

For example, to completely intercept Googlebot, you can use the following syntax:

User-agent:Googlebot Disallow:/

Allow Googlebot

If you want to block all rovers except Googlebot from accessing your web page, you can use the following syntax:

User-agent:Disallow:/

User-agent:Googlebot

Disallow:

Googlebot follows a line that points to itself, not to all rovers.

"Allow" extension

Googlebot recognizes the robots.txt standard extension called "Allow". The rovers of other search engines may not recognize this extension, so use other search engines that interest you to find it. The "Allow" line works exactly the same as the "Disallow" line. Just list the directories or pages you want to allow.

You can also use both "Disallow" and "Allow". For example, to block all pages except a page in a subdirectory, you can use the following entries:

User-Agent:Googlebot

Disallow:/folder1/

Allow:/folder1/myfile.html

These entries block all pages in the folder1 directory except myfile.html.

If you want to intercept Googlebot and allow another rover of Google, such as Googlebot-Mobile, you can use the "Allow" rule to allow access to that rover. For example:

User-agent:Googlebot

Disallow:/

User-agent:Googlebot-Mobile

Allow:

After reading the above, have you mastered the method of how to analyze Mediapartners-Google rover? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.