Shulou(Shulou.com)06/01 Report--

Editor to share with you how python crawled Weibo hot search and achieve data visualization, I believe most people do not understand, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to understand it!

Preface

First, go to the video to see the effect:

First, climb the hot search content on Weibo.

The most popular search site for Weibo is:

Https://s.weibo.com/top/summary

After analysis, the hot search data of Weibo is on the web page, which can be directly requested by requests, then parsed by BeautifulSoup to obtain the content, and finally stored in the form. The code is as follows (the complete code is at the end of the article):

For I, item in enumerate (items [1:11]): result = [] rank = 'the {0} name' .format (iTun1) # Weibo ranks num = str (item.find ('span')). Replace ('','). Replace ('' '') # Weibo fever title = item.find ('a'). Text # Weibo content result.append (time_stamp) result.append (rank) result.append (num) result.append (title) with open ('1.csvents,' axiom dagger') as f: f_csv = csv.writer (f) f_csv.writerow (result)

Hot searches on Weibo are usually updated every minute, so add a timer to the code:

Schedule.every (1) .minutes.do (run) # run is a custom hotsearch crawler function, which sets while True: schedule.run_pending () to crawl once a minute.

Let the program run for a while, and our data will be ready.

Start drawing dynamic diagrams

1.pandas reads data

Import pandas as pddata=pd.read_csv ('Weibo hot search. Csv',encoding='gbk')

two。 Basic dynamic drawing method

From pyecharts import options as optsfrom pyecharts.charts import Bar, Timelinefrom pyecharts.faker import Fakerx = Faker.choose () tl = Timeline () for i in range (2015, 2020): bar = (Bar () .add _ xaxis (x) .add _ yaxis ("", Faker.values ()) .set _ global_opts (title_opts=opts.TitleOpts ("annual turnover of a store {}" .format (I) tl.add (bar "{} year" .format (I)) tl.render_notebook ()

two。 Reverse the graph and import Abscissa (ranking), ordinate (heat)

Tl = Timeline () for i in range (20): bar = (Bar () .add _ xaxis (list (data ['content']) [i*10:i*10+10] [::-1]) .add _ yaxis ("Weibo Hot search list" List (data ['hot']) [i*10:i*10+10] [::-1]) .reversal _ axis () .set _ global_opts (title_opts=opts.TitleOpts ("annual turnover of a store {}" .format (I) tl.add (bar, "{} year" .format (I)) tl.render_notebook () `

3. Place the label on the right side of the drawing and move the drawing to the right as a whole

From pyecharts.charts import Bar, Timeline,Gridtl = Timeline () for i in range (20): bar = (Bar () .add _ xaxis (list (data ['content']) [i*10:i*10+10] [::-1]) .add _ yaxis ("Weibo Hot search list" List (data ['heat']) [i*10:i*10+10] [::-1]) .reversal _ axis () .set _ global_opts (title_opts=opts.TitleOpts ("annual turnover of a store {}" .format (I)) .set _ series_opts (label_opts=opts.LabelOpts (position= "right")) # place the label on the right side of the graph) tl.add (bar '') grid = (Grid () .add (bar, grid_opts=opts.GridOpts (pos_left= "25%", pos_right= "0%")) # move the whole graph to the right) tl.add (grid,'') tl.render_notebook ()

4. Set playback speed, hide timeline components, and set auto playback

Tl = Timeline () for i in range (20): bar = (Bar () .add _ xaxis (list (data ['content']) [i*10:i*10+10] [::-1]) .add _ yaxis ("Weibo Hot search list" List (data ['heat']) [i*10:i*10+10] [::-1]) .reversal _ axis () .set _ global_opts (title_opts=opts.TitleOpts ("annual turnover of a store {}" .format (I)) .set _ series_opts (label_opts=opts.LabelOpts (position= "right")) # place the label on the right side of the graph) tl.add (bar "") grid = (Grid () .add (bar, grid_opts=opts.GridOpts (pos_left= "25%", pos_right= "0%") # move the graph to the right as a whole) tl.add (grid, ") tl.add_schema (play_interval=100, # playback speed is_timeline_show=False, # whether the timeline component is_auto_play=False is displayed ) # whether to automatically play tl.render_notebook ()

5. Set the theme and add a time label

Tl = Timeline ({"theme": ThemeType.MACARONS}) for i in range (20): bar = (Bar ({"theme": ThemeType.MACARONS}) .add _ xaxis (list (data ['content']) [i*10:i*10+10] [::-1]) .add _ yaxis ("Weibo Hot search list" List (data ['Heat']) [i*10:i*10+10] [::-1]) .reversal _ axis () .set _ global_opts (title_opts=opts.TitleOpts ("{}" .format (list (data ['time']) [iTunes 10]), pos_right='0%',pos_bottom='15%'), xaxis_opts=opts.AxisOpts (splitline_opts=opts.SplitLineOpts (is_show=True)) Yaxis_opts=opts.AxisOpts (splitline_opts=opts.SplitLineOpts (is_show=True), axislabel_opts=opts.LabelOpts (color='#FF7F50')),) .set _ series_opts (label_opts=opts.LabelOpts (position= "right", color='#9400D3')) grid = (Grid () .add (bar, grid_opts=opts.GridOpts (pos_left= "25") Pos_right= "0%")) tl.add (grid, "{} year" .format (I)) # set label tl.add_schema (play_interval=100, # playback speed is_timeline_show=False, # whether to display timeline component is_auto_play=True,) tl.render_notebook ()

These are all the contents of this article entitled "how python crawls Weibo hot search and data visualization". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.