How to build a crawler program in Python
In this issue, the editor will bring you about how to build the crawler program in Python. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.
Development tools
Python version: 3.6.4
Related modules:
Scrapy module
Pyecharts==1.5.1 module
Wordcloud module
Jieba module
And some modules that come with python.
Environment building
Install Python and add it to the environment variable, and pip installs the relevant modules you need.
Data crawling
Let's start with a wave of self-open source libraries that use requests to simulate login:
Https://github.com/CharlesPikachu/DecryptLogin
Currently, the websites that support simulated login in the library include:
1. micro-blog
The functions of the library and some small applications related to the library will be continuously added and improved in the future. Of course, I can't use it today, because I found that his Miao Zhihu fan data has always been a naked API, and even after the revision, he doesn't need to verify the login cookies or anything like that.
To get to the point, let's briefly talk about how to grab this data. in fact, it is very simple. F12 opens the developer tool and refreshes the follower page to find:
Request this API to return the fan data of the target user directly. The composition of the API is as follows:
Https://www.zhihu.com/api/v4/members/{ user domain name} / followers?
There is nothing special to pay attention to, and there is no doubt that it is as simple as that. Scrapy will finish climbing a new project:
Scrapy startproject zhihuFansSpider
Define items:
Class ZhihufansspiderItem (scrapy.Item):
Then create a new crawler and write a main program called OK:
'' Zhihu fan crawler''
Run the following command to start crawling fan data for the target user:
Scrapy crawl zhihuFansSpider-o followers_info.json-t json
Data visualization
According to the old rule, you can visually climb to the data (here, take the follower data of my own Zhihu account as an example, Tweet).
First draw a fan home page title of the word cloud shock?
! [https://upload-images.jianshu.io/upload_images/2539976-ada286149ecb2285?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)] the above is how to build a crawler program in Python shared by the editor. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.
