In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-18 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Development >
Share
Shulou(Shulou.com)06/03 Report--
This article focuses on "how to use Python to make web crawlers", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to use Python to make web crawlers.
A simple way to make a crawler: get the html code of the site in the "website_links" list and search the first tag to get its title. In this way we get the title (title) of the main article on a website.
Web crawler is an epic small piece of software that you can build in a short time. It is very simple to use the above methods to detect breaking news on the Internet in real time. The following is a code example.
Import requests from bs4 import BeautifulSoup website_links = ["https://www.aljazeera.com/"," https://www.thehindu.com/", "https://www.ndtv.com/"] consolidatedTitleString =" for I, website in enumerate (website_links): page = requests.get (website) soup = BeautifulSoup (page.text) 'html.parser') # to get the headings and display title = soup.find (' h2'). Get_text () consolidatedTitleString + = "\ n\ n" + str (I) + ")" + title.strip ("\ n")
You can get the title of the site in this way, using five main Python packages: scrapy, BeautifulSoup, requests, urllib, and re. The last one is' re' or the regular expression library is very useful. In html code, we have the following uses
Indonesia quake, tsunami toll tops 800
To make anything useful, we need to delete the html tag and just get the text, which is usually done using the ".get _ text ()" function when using the soup library. However, it is useful to understand how to do this using regular expressions.
The following code helps to extract a list of links from tweets, rather than the links to newspaper websites given above. We use the pattern "htttps" to detect links using the re library.
For iJournal status in enumerate (tweepy.Cursor (api.home_timeline) .items (7)): try: listOfLinks.append ("(? Phttps?:// [^\ s] +)", status.text) .group ("url") except AttributeError: print ("A link does not exist")
You can also make a "ImageCrawler" to download all the images on the web.
R = requests.get ("http://pythonforengineers.com/pythonforengineersbook/") data = r.text soup = BeautifulSoup (data," lxml ") for link in soup.find_all ('img'): image = link.get (" src ") image =" http: "+ image question_mark = image.find (?) Image = image [: question_mark] image_name = os.path.split (image) [1] print (image_name) R2 = requests.get (image) with open (image_name, "wb") as f: f.write (r2.content) so far, I believe you have a deeper understanding of "how to use Python to make a web crawler". You might as well do it in practice! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.