Shulou(Shulou.com)06/02 Report--

This article introduces how python selenium forbids the implementation of certain requests. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

Problem description

When the target website is requested through selenium, the real data (I am the CAPTCHA image here) has been loaded, and it will take a long time for the website to make a third-party report, but it is not necessary to report this request.

For example

The verification code has been loaded, but the huatuo.qq.com response time is too long. The mechanism of webdriver.get () is to wait for all the url responses of the request to be completed before the next step. The function of display wait and implicit wait is to check the number of seconds at which the address is loaded, so it does not take effect here.

Then what I need to do is: when requesting the target url, I hope webdriver will not report or block the huatuo.qq.com. This saves a lot of time for the next step.

Solution

After opening the target url through selenium, insert the js plug-in to block the reporting url.

Configure selenium properties and add shielding rules

Chrome_options.add_argument ('--host-resolver-rules=MAP report.huatuo.qq.com 127.0.0.1')

Final effect

In this way, you can focus on the target url and execute the next step faster.

Other attribute configuration

Options.add_argument ('headless') # headless mode options.add_argument (' window-size= {} x {} '.format (width Height)) # Direct configuration size as set_window_size options.add_argument ('disable-gpu') # disable GPU acceleration options.add_argument (' proxy-server= {} '.format (self.proxy_server)) # configure proxy options.add_argument ('-no-sandbox') # sandboxie mode runs options.add_argument ('- disable-setuid-sandbox') # disable sandboxie options.add_argument ( '- disable-dev-shm-usage') # write / tmp instead of / dev/shmoptions.add_argument ('-user-data-dir= {profile_path} '.format (profile_path)) # user data is stored in the specified file options.add_argument (' no-default-browser-check) # No browser default check options.add_argument ("- disable-popup-blocking") # allows pop-up options. Add_argument ("- disable-extensions") # disable extended options.add_argument ("- ignore-certificate-errors") # ignore untrusted certificate options.add_argument ("- no-first-run") # initialize a blank page options.add_argument ('- start-maximized') # maximize launch options.add_argument ('- disable-notifications') # disable notification warning options.add_argument (' -enable-automation') # Notification (notify users that their browser is being controlled by automated testing) options.add_argument ('- disable-xss-auditor') # disable xss protection options.add_argument ('- disable-web-security') # turn off security policy options.add_argument ('- allow-running-insecure-content') # allow unsafe content options.add_argument ('- disable-webgl') to run # disable webgloptions.add_argument ('- homedir= {}') # specify home directory location options.add_argument ('- disk-cache-dir= {temporary file directory}') # specify temporary file directory options.add_argument ('disable-cache') # disable cache options.add_argument (' excludeSwitches' ['enable-automation']) # this is how the developer model implements that python selenium forbids loading certain requests. I hope the above content can be of some help to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.