Shulou(Shulou.com)05/31 Report--

This article mainly introduces how to use python crawler to crawl the Tencent Cloud technology community. It is of certain reference value. Interested friends can refer to it. I hope you will learn a lot after reading this article. Let the editor take you to know it.

Text

Programming idea

Get the addresses of all articles

Content extraction from a single article page

Extract the content of all articles and store the results in the MongoDB database

Using word Segmentation system and wordcloud to construct word Cloud

Note: I added a random number before storing all the article addresses, and then randomly selected the articles to extract them later.

Prevent the result from being localized due to different dates

Get the article list page, all the article information

Save in the following format:

Index random number index

Title article name

Address article address

Content article content

Def getonepageall (self, url): try: html = self.getpageindex (self.baseURL) # parse soup = BeautifulSoup (html, 'lxml') title = soup.select (' .article-item > .title') address = soup.select ('.article-item > .title > a [href]') for i in range (len (title)): # generate random index randomnum = random.randrange (0) 6500) content = self.parsecontent ('https://www.qcloud.com' + address [I]. Get (' href'). Strip ()) yield {'index': randomnum,' title': title [I]. Get _ text (). Strip (), 'address':' https://www.qcloud.com' + address [I]. Get ('href'). Strip (),' content': content} # skip except IndexError: pass when you encounter an index error

Parse the content of the article def parse_content (self, url): html = self.get_page_index (url) soup = BeautifulSoup (html, 'lxml') # here we directly use the content content = soup.select (' .J-article-detail') return content [0] .get _ text () in div where class is J-article-detail.

Here, I will directly release the final result.

Because the word segmentation system is not very good, the result is not very ideal.

Here I use regular expressions to remove all non-Chinese characters from the content.

Due to the poor configuration of the personal computer, I divided the results into 20 parts, each consisting of 100 randomly selected articles.

Thank you for reading this article carefully. I hope the article "how to use python crawler to crawl Tencent Cloud technology community" shared by the editor will be helpful to you. At the same time, I hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.