Shulou(Shulou.com)06/01 Report--

Scrapy framework docking selenium how to simulate Zhihu login, in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Summary: I have written the foundation of scrapy before, and I am ready to try to simulate Zhihu login and crawl questions and answers.

But I found that Zhihu had been updated and encrypted the parameters, so as a rookie, I had to try selenium. In this article, we will learn how the scrapy framework interfaces with selenium.

1. New project

New projects and basic operations should be understood in the previous article, if you don't already know

Here is a link: installation and basic use of the crawler framework Scrapy

2. Log in to simulate Zhihu

First of all, you need to change ROBOTSTXT_OBEY = True to ROBOTSTXT_OBEY = False,ROBOTSTXT_OBEY in seetings.py, which can be said to be a gentleman's agreement. Let's change it to False, otherwise we can't crawl.

Then write the following code on spider:

What you don't know about selenium can check out: the use of selenium (interesting)

3. Run

Here is a little trick to teach you.

We create a main file under the project and write the following code:

That's it.

So we don't have to run under the black window every time, it's troublesome.

This time I simply wrote down the use of selenium in the scrapy framework without using any new knowledge.

This is the answer to the question of how to simulate Zhihu login with Scrapy framework and selenium. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.