Shulou(Shulou.com)05/31 Report--

Most people do not understand the knowledge points of this article "configuring the User-Agent method of filtering crawlers in Nginx", so the editor summarizes the following contents, detailed contents, clear steps, and has a certain reference value. I hope you can get something after reading this article. Let's take a look at this "User-Agent method of configuring filtering crawlers in Nginx".

As soon as I can't see it, I was basically brushed by something called "yisouspider" by a user agent. I don't know where the spider is. It's too unscrupulous.

Find the root directory configuration area, add user agent filter judgment statement, and find that the one called "yisouspider" directly returns 403.

Note 1: if you need to add multiple filters, do this

($http_user_agent ~ * "spider1 | spider2 | spider3 | spider4")

, use it in the middle | just separate it.

Note 2: if you are using a subdirectory blog, like mine, find a section such as "location / blog/" to modify it

Location / {. Other configuration if ($http_user_agent ~ * "yisouspider") {return 403;}}

After saving the configuration wq, reload the nginx, and then use the following command to test yourself and change the address yourself. I can't help it if I don't have a curl. I'll install one on my own apt or yum. It's an artifact.

Curl-I-a "yisouspider" www.slyar.com/blog/

It is OK to see that 403 is returned, indicating that the configuration is successful.

The above is about the content of this article on "configuring User-Agent methods for filtering crawlers in Nginx". I believe we all have some understanding. I hope the content shared by the editor will be helpful to you. If you want to know more about the relevant knowledge, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.