Shulou(Shulou.com)06/02 Report--

This article mainly introduces "the detailed usage of Python crawler Selenium library". In daily operation, I believe many people have doubts about the detailed usage of Python crawler Selenium library. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "detailed usage of Python crawler Selenium library". Next, please follow the editor to study!

In the process of crawling the web page, we often find that the data we want can not be obtained simply by parsing the HTML code. The data is displayed on the page after AJAX asynchronous loading or JS rendering.

Selenuim is an automated testing tool that supports multiple browsers. In crawlers, we can use it to simulate browsing pages in a browser, thus solving the problem of JavaScript rendering.

1. Use the example

two。 Detailed introduction

2.1 declare browser objects

That is, tell the program which browser to use to operate.

2.3 find elements

After a successful visit to the web page, we may need to do some actions, such as finding the search box, typing the keyword and then hitting enter.

Therefore, you need to look for elements in selenium.

2.3.1 single element

There are two ways for selenium to find elements.

The first is to specify which method to use to find elements, such as according to CSS selection or xpath.

The following is a detailed method of finding elements

Find_element_by_namefind_element_by_xpathfind_element_by_link_textfind_element_by_partial_link_textfind_element_by_tag_namefind_element_by_class_namefind_element_by_css_selector

The second is to use find_element () directly, with the first parameter passed in as the element lookup method to be used

2.3.2 multiple elements

Finding multiple elements is basically the same as finding a single element (you only need to add an s to the func of finding a single element).

Looking for multiple elements returns a list.

2.5 Interactive action

An interactive action is a serial execution of an action attached to the interaction chain, which requires the use of ActionChains.

2.6Execute JavaScript

Like drag and drop.

2.8 Frame

If you locate the parent frame, you cannot find the information about the child frame, so you need to switch to the child frame to find it. Similarly, the information of the parent frame cannot be found in the child frame.

2.9.2 explicit wait

Shows that the wait includes the wait condition and the wait time.

First of all, determine whether the waiting condition is established, and if it is established, return directly; if the condition is not established, the longest waiting time is the waiting time, and if the waiting condition is not met after exceeding the waiting time, an exception is thrown.

An explicit wait is a wait for a specified element.

2.11 operate on Cookies

At this point, the study on the "detailed usage of the Python crawler Selenium library" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.