Shulou(Shulou.com)06/01 Report--

1 Scrapy set User-Agent1.1 to set the USER_AGENT parameter in the settings.py file can achieve 1.1.1 using the fixed User-Agent method

Add the following code to the settings.py file (USER_AGENT is fixed), and you can change the blue font content below by yourself with what kind of agent you use.

USER_AGENT = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)

1.1.2 using the random User-Agent method

In the settings.py file, you can create an agent list and then randomly take a value from it and assign it to the USER_AGENT variable.

The code is as follows:

Import random

UserAgentList = [

"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)"

"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"

]

USER_AGENT = random.choice (UserAgentList) 1.1.3 use a third-party user-agent library (fake-useragent) to implement the agent setting method

Add the following code to the settings.py file:

From fake_useragent import UserAgent

Ua = UserAgent ()

USER_AGENT = ua.random1.2 1.2.1 by writing the middleware method by writing the user-agent method in the setting file

Write a UserAgentList list in the setting.py file

UserAgentList = [

'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1500.55 Safari/537.36'

'Mozilla/5.0 (Macintosh; Intel Mac OS X 10, 8) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17'

]

Enable DOWNLOADER_MIDDLEWARES in the setting.py file and comment out the default

DOWNLOADER_MIDDLEWARES = {

# 'amazon.middlewares.MyCustomDownloaderMiddleware': 543

'amazon.middlewares.MyUserAgentMiddleware': 543

}

Add the following code to the middlewares.py file:

Rom scrapy.downloadermiddlewares.useragent import UserAgentMiddleware

From amazon.settings import UserAgentList

Import randomclass MyUserAgentMiddleware (UserAgentMiddleware):

Def _ _ init__ (self, user_agent=''):

Self.user_agent = user_agent

Def process_request (self, request, spider):

Agent = random.choice (list (UserAgentList))

Request.headers ['User-Agent'] = agent1.2.2 sets the User-Agent method using the fake-useragent library

Install the fake-useragent library (useragent's library)

Pip install fake-useragent

Import fake-useragent libraries in a middlewares.py file

From fake_useragent import UserAgent

Write a middleware middleware in middlewares.py file

From fake_useragent import UserAgent

From scrapy.downloadermiddlewares.useragent import UserAgentMiddlewareclass MyUserAgentMiddleware (UserAgentMiddleware):

Def _ _ init__ (self, user_agent=''):

Self.user_agent = user_agent

Def process_request (self, request, spider):

Ua = UserAgent ()

Request.headers ['User-Agent'] = ua.random

Enable DOWNLOADER_MIDDLEWARES in the setting.py file and comment out the default

DOWNLOADER_MIDDLEWARES = {

# 'amazon.middlewares.MyCustomDownloaderMiddleware': 543

'amazon.middlewares.MyUserAgentMiddleware': 543

}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.