Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to log in to Renren with scrapy framework." Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let Xiaobian take you to learn "how to log in to Renren. com with scrapy framework"!

Necessary analysis before writing crawler

The above picture is our Renren website. Usually we want to log in to a website. There are two ways of thinking: ① grab the post data by grabbing the package tool, see if there is a form inside, and then submit the form to log in ② find the form tag by observing and analyzing the html source code of the webpage and find the relevant submission parameters.

After my analysis, I used F12 to look at the capture data, is not found post data (you can look at it yourself), unexpectedly this does not work, we will directly view the source code!

As you can see from where I put the arrow, the URL we need to send the account and password information is www.renren.com/PLogin.do, and then we need to send two parameters email and password.

When we submit these two parameters, if the login is successful, it will return the source code of a successful login interface. We will judge whether the login is successful through this source code!

After analyzing, start writing code

The previous creation of crawler projects and files and other operations I will not mention more, will not be friends can go to see the article I wrote earlier.

I first directly throw the code to everyone, the old rules, explain written in the code comments!

loginrr.py:

# -*- coding: utf-8 -*-import scrapyfrom scrapy.http import Request,FormRequest

class LoginrrSpider(scrapy.Spider): name = 'loginrr' allowed_domains = ['renren.com'] start_urls = ['http://renren.com/'] headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36' }

#Here we write the start_requests method first (it will be executed first) def start_requests(self): #First visit the login page once (here we can monitor for Captcha, if yes we need to add Captcha field, if no we don't) #Then call back parse method #Set meta parameter, set cookiejar to 1, which means cookjar is open return [Request("http://www.renren.com/PLogin.do", meta={"cookiejar": 1}, callback=self.parse)]

def parse(self, response): #Form to send, no Captcha this time data = { #Enter your own account password here 'email': 'xxxxxx', 'password': 'xxxxxxxx' } print("Landing... ") #Login via FormRequest.from_response method return [FormRequest.from_response(response, #Set cookie information meta={"cookiejar": response.meta["cookiejar"]}, headers=self.headers, formdata=data, #Call back to next method callback=self.next, )]

def next(self, response): #Print the interface source code after successful login print(response.body)

Note that you need to set the setting.py file between runs, and set ROBOTSTXT_OBEY to False

Run Results:

At this point, I believe that everyone has a deeper understanding of "how to log in to Renren with scrapy framework." You may wish to actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.