Shulou(Shulou.com)06/03 Report--

This article is about how python rewrites the start_requests method. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Description

1. In scrapy, start_url is handled by start_requests. By overriding this method, start_url can carry request header information.

2. Cookie cannot be placed in the headers in scrapy. There are special cookies parameters when building the request.

You can receive cookie in dictionary form. You may need to set up the ROBOTS protocol and USER_AGENT in settings.

Example

Import scrapy class Git1Spider (scrapy.Spider): name = 'git1' allowed_domains = [' github.com'] start_urls = ['https://github.com/GitLqr'] def start_requests (self): "overrides start_requests and sends Request with cookies. The default start_requests is just an ordinary get request and does not carry the custom header information "" url = self.start_urls [0] temp ='_ octo=GH1.1.1045146750.1615451260; _ device_id=cd8d64981fcb3fd4ba7f587873e97804' # convert the cookies string into the dictionary cookies = {data.split ('=') [0]: data.split ('=') [- 1] for data in temp.split (' ')} yield scrapy.Request (url=url, callback=self.parse, cookies=cookies) def parse (self, response): print (response.xpath (' / html/head/title/text ()'). Extract_first ()) Thank you for reading! This is the end of the article on "how to rewrite the start_requests method in python". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.