Shulou(Shulou.com)06/01 Report--

This article will explain in detail all the goods and their information about how python crawls a keyword on Taobao. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

Source code: https://github.com/Ckend/taobao_scrapy

The source code contains two Python files, get_items.py is used to obtain basic information about goods through keywords, including product name, store, sales volume, number of comments, pictures, offers, place of shipment, original price and so on. The output format is a .csv file.

Another Python file: get_review_cloud.py is used to get the comment cloud of a product through userID and itemID (which are obtained from get_items.py). Export the .csv file.

The source code is very simple, more than ten lines, the logic is as follows:

1. Access interface (f=urllib.request.urlopen (url))

two。 Read the source code and decode it to utf-8 and then convert it to json

3. Write to the .csv file.

The following is the content of get_items.py

Get_items.py:

Import urllib.request

Import urllib.parse

Import json

Import csv

Def write_csv (data, filename):

Count = 0

With open (filename,'a') as outf:

Dw = csv.DictWriter (outf, fieldnames= ["title", "sold", "commentCount", "item_id", "shipping", "fastPostFee", "userId", "nick", "userType", "isB2c", "location", "sellerLoc", "pic_path", "type", "tItemType", "zkType", "zkGroup", "priceColor", "priceWithRate", "auctionURL", "isP4p", "itemNumId", "originalPrice", "freight", "act", "coinLimit", "priceWap", "price", "category" "auctionType", "url", "img2", "wwimUrl", "previewUrl", "favoriteUrl", "isMobileEcard", "iswebp", "name", "iconList", "icons", "area"])

If count = = 0:

# write the header only on the first line

Dw.writeheader ()

Count + = 1

For row in data:

Dw.writerow (row)

Def get_items (searchWords, page):

Url = "https://s.m.taobao.com/search?event_submit_do_new_search_auction=1&_input_charset=utf-8&topSearch=1&atype=b&searchfrom=1&action=home%3Aredirect_app_action&from=1&q="+ str (urllib.parse.quote (searchWords)) +" & sst=1&n=20&buying=buyitnow&m=api4h6&token4h6=&abtest=20&wlsort=20&page= "+ str (page)

F = urllib.request.urlopen (url)

Result = json.loads (f.read () .decode ('utf-8'))

Return result ['listItem']

Def get_items_by_keywords (searchWords, page):

For i in range (1):

Result = get_items (searchWords, I)

Try:

Write_csv (result, ". / result/" + str (searchWords) + "_ result.csv")

Except INdexError:

# keywords the product has been crawled

Print ("all the keyword products have been crawled")

Get_items_by_keywords (Air freshener, 100)

This is the end of this article on "how python crawls all the goods and information of a keyword on Taobao". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it out for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.