Shulou(Shulou.com)06/03 Report--

It is found that although wget follows the robots.txt rules, that can still be bypassed. Now the 56 cloud editor shares my own shielding method with you:

Block downloading of any files

.htaccess

SetEnvIfNoCase User-Agent "^ wget" bad_bot

Order Allow,Deny

Allow from all

Deny from env=bad_bot

Block some downloaded files

.htaccess

SetEnvIfNoCase User-Agent "^ Wget" bad_bot

SetEnvIfNoCase User-Agent "^ Wget/1.5.3" bad_bot

SetEnvIfNoCase User-Agent "^ Wget/1.6" bad_bot

Order Allow,Deny

Allow from all

Deny from env=bad_bot

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.