Shulou(Shulou.com)06/02 Report--

This article introduces why Scrapy cannot use Cookies to log in. The content is very detailed. Interested friends can refer to it for reference. I hope it can help everyone.

We understand that the website uses Cookies to record the user's login status. If we copy Cookies from our browser and put them in our crawler, in some cases, we can get the crawler to visit the login page directly.

Take the exercise page http://exercise.kingname.info/exercise_login_success as an example. In the case of no login, visiting this address will automatically jump to the login page, as shown in the following figure:

Enter the user name kingname, password genius, click automatic login, and click the login button to jump to the login success page, as shown in the following figure:

Once the login is successful, even if you close the browser and reopen the URL, you will still go directly to the login page.

We can see in Chrome's Developer Tools the request sent when we refresh the page after logging in:

You can see the request header in the Request Headers, which contains the Cookie item.

When we use Requests to access this URL, as long as the request header contains this Cookie, we can directly access the successful request page, as shown in the following figure:

The whole process is very simple and intuitive.

But if you try to do the same thing with Scrapy, you'll find that the login fails. As shown below:

The key to this is that Scrapy does not use cookies in Headers when it makes a request, but has a separate cookie parameter.

When Request is used, cookies exist as strings in Headers, with semicolons between items and colons between keys within items.

But in Scrapy, the value passed to the cookie parameter is a dictionary, the dictionary key is the key of each item, and the value is the value of each item.

So we changed Scrapy's code slightly, removing cookies from Headers, changing it to dictionary form, passing cookies parameters, and requesting discovery again:

About why Scrapy can not use Cookies to log in to share here, I hope the above content can be of some help to everyone, you can learn more knowledge. If you think the article is good, you can share it so that more people can see it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.