Shulou(Shulou.com)06/02 Report--

How to prevent spiders from accessing virtual hosts? Many novices are not very clear about this. In order to help you solve this problem, the following editor will explain it in detail. People with this need can come and learn. I hope you can gain something.

Virtual hosts organize spider access, which can be implemented using the robots protocol. Robots is an agreement between a website and a crawler, which tells the corresponding crawler the allowed permissions in a simple and direct txt format, that is, robots.txt is the first file to view when visiting a website in a search engine.

When we manage virtual hosts, in some cases, we don't want search engine spiders to access our content, so we need to write a crawler protocol that forbids search engines from grabbing data. Or prohibit some search engine crawlers from visiting, and allow other search engines. Can be achieved through the robots file.

When a search spider visits a site, it will first check whether robots.txt exists in the root directory of the site. If so, the search robot will determine the scope of access according to the contents of the file; if the file does not exist, all search spiders will be able to access all pages on the site that are not password protected.

When virtual hosts block spider access, such as banning all search engines, you can write as follows:

User-agent: *

Disallow: /

When a virtual host blocks access by a spider, it can be written as follows:

User-agent: xxxspider

Disallow:/

When a virtual host blocks a directory from being accessed by spiders, it can be written as follows:

User-agent: *

Disallow: / admin/

It means / admin/ directory, and all search engines are not allowed to access it.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.