Shulou(Shulou.com)06/02 Report--

This article is about how to use robots.txt files in Imperial CMS. The editor thought it was very practical, so I shared it with you as a reference. Let's follow the editor and have a look.

Before talking about the use of robots.txt files in Imperial CMS, let's explain what robots.tx does.

Robots protocol (also known as crawler protocol, crawler rules, robot protocol, etc.), that is, robots.txt, website uses robots protocol to tell search engines which pages can be crawled and which pages can not be crawled. Robots protocol is a common code of ethics in the Internet community of websites. Its purpose is to protect website data and sensitive information, and to ensure that users' personal information and privacy are not violated. Because it is not an order, it needs to be followed by the search engine consciously. Some viruses such as malware (Marvell virus) often obtain the background data and personal information of the website by ignoring the robots protocol.

The robots.txt file is a text file that can be created and edited using any common text editor, such as the Notepad that comes with the Windows system. Robots.txt is a protocol, not a command. Robots.txt is the first file in a search engine to view when you visit a website. The robots.txt file tells the spider what files can be viewed on the server.

Recommended to learn the Imperial cms course

When a search spider visits a site, it will first check whether robots.txt exists in the root directory of the site. If so, the search robot will determine the scope of access according to the contents of the file; if the file does not exist, all search spiders will be able to access all pages on the site that are not password protected. Baidu officially recommends that you use robots.txt files only if your site contains content that you don't want to be included by search engines. If you want the search engine to include all the content on the site, please do not create robots.txt files.

If you think of the website as a room in a hotel, robots.txt is a "do not disturb" or "welcome to clean" sign hung by the owner at the door of the room. This document tells visiting search engines which rooms can be entered and visited, and which rooms are not open to search engines because they store valuables or may involve the privacy of residents and visitors. But robots.txt is neither a command nor a firewall, just as gatekeepers cannot stop malicious intruders such as thieves.

The default robots.txt for Imperial CMS is:

The code is as follows:

# # robots.txt for EmpireCMS#User-agent: * * allow all search engines to crawl Disallow: / d / * forbid all search engines to crawl D directory Disallow: / e/class/ * forbid all search engines to crawl / e/class/ directory Disallow: / e/data/ * prohibit all search engines to crawl / e/data/ directory Disallow: / e/enews/ * prohibit all search indexes Rock crawl / e/enews/ directory Disallow: / e/update/ * prohibit all search engines from crawling / e/update/ directory thank you for reading! On the Imperial CMS in the use of robots.txt files to share here, I hope the above content can be of some help to you, so that you can learn more knowledge. If you think the article is good, you can share it and let more people see it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.