Shulou(Shulou.com)06/01 Report--

What is the impact of Robots.txt protocol on website optimization? I believe many inexperienced people are at a loss about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

The website Robots.txt file is the general protocol for the website to communicate with the search engine. Through the setting of the Robots protocol, it tells the search engine which pages can be crawled and which pages cannot be crawled: on the one hand, it can be used to protect the security of the website, and more importantly, it is used to optimize, reduce the inclusion of invalid pages, and improve the ranking effect of the site.

But in the actual operation, the vast majority of websites have more or less deficiencies in their writing, even due to technical errors in writing, it will also lead to a series of problems, such as website reduction, non-inclusion, being K and so on. For this point, in the customer's SEO diagnosis process, will often encounter, can be regarded as a common fault of many sites. Today is to share: have you written correctly about the robots.txt agreement?

1: set to Allow full site crawl

The more Baidu is included, the higher the ranking of the site? This is the view of the vast majority of webmasters, in fact, it is also the case. But it is not absolutely true: low-quality pages will reduce the ranking effect of the site, have you taken this into account?

If the structure of your site is not very clear, and there are no extra "functional" pages, it is not recommended to crawl the whole site. In fact, in the SEO diagnosis of A5, only a very small number of sites are encountered, so that the whole site can be crawled without blocking. With the enrichment of features, it is not possible to allow site-wide crawling.

Second: what kind of pages are not recommended to crawl?

For functional useful directories and useful pages of the website, the user experience can be better improved. But in terms of search engines, it will cause a burden on the server, such as a large number of page flipping comments, which is of no value to optimization.

In addition, it also includes, such as: after the site has done pseudo-static processing, then it is necessary to block the dynamic links to avoid search engine crawling. User login directory, registration directory, useless software download directory, if it is a static type of site, but also block the dynamic type link Disallow: / *? * Why? Let's take an example:

The above is the problem found by a customer's website, the reason included by Baidu is: someone maliciously submitted this type of link, but the site itself did not do a good job of protection.

Third: notes on the details of writing.

In terms of method, the vast majority of webmasters understand that there is no more to say here, and webmasters who do not understand can have a look at Baidu encyclopedia. Some unusual things are said here today, which may be the questions of many webmasters.

1, for example: the difference between Disallow; / an and Disallow: / a /, many webmasters have seen such a problem, why some agreements add slashes, some do not add slashes? What the author wants to say today is: if there is no slash, it will block all directories and pages that begin with the letter a, while the latter represents blocking the crawling of all pages and subdirectories in the current directory.

Generally speaking, we tend to choose the latter more, because the larger the definition, it is easy to cause "manslaughter".

2. Do JS files and CSS need to be shielded? Many websites have done this blocking, but what the author wants to say is: the google webmaster tool clearly states: blocking css and js calls may affect the judgment of page quality, thus affecting the ranking. In this regard, we have done some understanding, Baidu will also have a certain impact.

3, has been deleted directory shielding, many webmasters often delete some directories, for fear of 404 problems, and shielded, forbidding search engines to crawl such links. In fact, is it really good to do so? Even if you block it, if there is a problem with the previous directory, it will also affect the site if it is not removed from the library by spiders.

The best way to suggest is to sort out the corresponding major error pages, submit dead links, and customize the handling of 404 pages to solve the problem thoroughly, rather than avoiding it.

After reading the above, have you mastered the impact of Robots.txt protocol on website optimization? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.