Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how to obtain dynamic charts by Python". Many people will encounter such a dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Development tools

Python version: 3.6.4

Related modules:

Re module

Requests module

Urllib module

Pandas module

And some modules that come with Python.

Environment building

Install Python and add it to the environment variable, and pip installs the relevant modules you need.

Take a look at bilibili's "data Visualization" section in 2019. The first video has more than 2 million views and 40, 000 + on-screen comments.

Baidu index

To get the Baidu Index, you first need to log in to your Baidu account.

Take the keyword "Arena of Valor" as an example, the time is customized to 2020-10-01-2020-10-10.

Through the developer tools, we can see the data interface of the graph.

However, looking at the result of the request, it is found that there is no data, because JS encryption is used here.

Find a solution, successfully implement crawling, code implementation

Import timeimport jsonimport execjsimport datetimeimport requestsfrom urllib.parse import urlencode def get_data (keywords, startDate, endDate, area): "get encrypted parameter data"# data_url =" http://index.baidu.com/api/SearchApi/index?area=0&word=[[%7B%22name%22:%22%E7%8E%8B%E8%80%85%E8%8D%A3%E8%80%80%22, % 22wordType%22:1%7D]] & startDate=2020-10-01&endDate=2020-10-10 "params = {'word': json.dumps ([[{' name': keyword, 'wordType': 1}] for keyword in keywords]),' startDate': startDate, 'endDate': endDate 'area': area} data_url =' http://index.baidu.com/api/SearchApi/index?' + urlencode (params) # print (data_url) headers = {# copy login cookie "Cookie": 'your cookie', "Referer": "http://index.baidu.com/v2/main/index.html", "User-Agent": "Mozilla/5.0 (Windows NT 10.0 Win64 X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36 "} # get data and uniqid res = requests.get (url=data_url Headers=headers) .json () data = res ["data"] ["userIndexes"] [0] ["all"] ["data"] uniqid = res ["data"] ["uniqid"] # get the parameter t = "ev-fxk9T8V1lwAL6,51348+.9270-%" t_url = "http://index.baidu.com/Interface/ptbk?uniqid={}".format(uniqid) rep = requests.get (url=t_url) in the js function Headers=headers) .json () t = rep ["data"] return {"data": data, "t": t} def get_search_index (word, startDate, endDate, area): "" get the final data "" word = word startDate = startDate endDate = endDate # call get_data to get data and uniqid res = get_data (word, startDate, endDate) Area) e = res ["data"] t = res ["t"] # read the js file with open ('parsing_data_function.js', encoding='utf-8') as f: js = f.read () # convert it to a js object docjs = execjs.compile (js) # call the function method and get the exponential value res = docjs.call (' decrypt', t) E) # print (res) return res def get_date_list (begin_date, end_date): "" get the time list "" dates = [] dt = datetime.datetime.strptime (begin_date, "% Y-%m-%d") date = begin_date [:] while date

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.