Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

How to use Python to make Web Crawler

2025-01-18 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Development >

Share

Shulou(Shulou.com)06/03 Report--

This article focuses on "how to use Python to make web crawlers", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to use Python to make web crawlers.

A simple way to make a crawler: get the html code of the site in the "website_links" list and search the first tag to get its title. In this way we get the title (title) of the main article on a website.

Web crawler is an epic small piece of software that you can build in a short time. It is very simple to use the above methods to detect breaking news on the Internet in real time. The following is a code example.

Import requests from bs4 import BeautifulSoup website_links = ["https://www.aljazeera.com/"," https://www.thehindu.com/", "https://www.ndtv.com/"] consolidatedTitleString =" for I, website in enumerate (website_links): page = requests.get (website) soup = BeautifulSoup (page.text) 'html.parser') # to get the headings and display title = soup.find (' h2'). Get_text () consolidatedTitleString + = "\ n\ n" + str (I) + ")" + title.strip ("\ n")

You can get the title of the site in this way, using five main Python packages: scrapy, BeautifulSoup, requests, urllib, and re. The last one is' re' or the regular expression library is very useful. In html code, we have the following uses

Indonesia quake, tsunami toll tops 800

To make anything useful, we need to delete the html tag and just get the text, which is usually done using the ".get _ text ()" function when using the soup library. However, it is useful to understand how to do this using regular expressions.

The following code helps to extract a list of links from tweets, rather than the links to newspaper websites given above. We use the pattern "htttps" to detect links using the re library.

For iJournal status in enumerate (tweepy.Cursor (api.home_timeline) .items (7)): try: listOfLinks.append ("(? Phttps?:// [^\ s] +)", status.text) .group ("url") except AttributeError: print ("A link does not exist")

You can also make a "ImageCrawler" to download all the images on the web.

R = requests.get ("http://pythonforengineers.com/pythonforengineersbook/") data = r.text soup = BeautifulSoup (data," lxml ") for link in soup.find_all ('img'): image = link.get (" src ") image =" http: "+ image question_mark = image.find (?) Image = image [: question_mark] image_name = os.path.split (image) [1] print (image_name) R2 = requests.get (image) with open (image_name, "wb") as f: f.write (r2.content) so far, I believe you have a deeper understanding of "how to use Python to make a web crawler". You might as well do it in practice! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Development

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report