Shulou(Shulou.com)06/01 Report--

The matching picture originally planned to choose a kind of crawler, but it was disgusting to see the picture found by Baidu, so give up and restore the real crawler in the online world.

Crawlers should have emerged with search engine technology. Crawler technology is mainly used to capture the content of web pages on the World wide Web, and the pages that are crawled will be included in the search engine. In general, enterprises as portals very much want crawlers to climb to, and then included, so that users will be shown when searching. On the one hand, with the rapid development of search technology and the rapid development of www business, all kinds of crawlers, * to the Internet, crawlers are a waste of traffic for some websites. For example, a certain treasure is determined not to crawl to a certain degree, on the one hand, to ensure the source of traffic, on the other hand, to ensure normal traffic, because malicious crawlers in www sites will affect normal traffic access.

So robots agreement arises at the historic moment.

The full name of "citing" Robots protocol (also known as crawler protocol, robot protocol, etc.) is "Web crawler exclusion criteria" (Robots Exclusion Protocol). The website uses the Robots protocol to tell search engines which pages can be crawled and which pages cannot be crawled.

Prevent spider from crawling arbitrarily: (prohibit any crawler from crawling any directory of the website)

Spider user-agent:*

Disallow:/

These rules support regularity, so they can be very flexible. For business-oriented pages can be set not to be crawled.

There is also the most important point: for the robots protocol to take effect, you must have access to robots.txt when web is accessed, and robots is located in the root directory of the website. For example:

Www.taobao.com/robots.txt can see Taobao's robots protocol.

Finally, if you don't want to leave some evidence of huai on a real-name site, don't hope that the site will ban crawlers, so be careful to maintain good online behavior.

Alipay-reward:

Wechat-reward:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.