Shulou(Shulou.com)06/02 Report--

This article is about what the Python crawler framework has. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

1. Scrapy

Scrapy is an application framework written to crawl website data and extract structural data. It can be used in a series of programs, including data mining, information processing, or storing historical data. With this framework, you can easily climb down data such as Amazon product information.

II. PySpider

Pyspider is a powerful web crawler system implemented by python, which can write scripts, schedule functions and view crawl results in real time on the browser interface. The back end uses common databases to store crawl results, and can set tasks and task priorities regularly.

III. Crawley

Crawley can crawl the content of the corresponding website at high speed, support relational and non-relational databases, and the data can be exported to JSON, XML and so on.

IV. Portia

Portia is an open source visual crawler tool that allows you to crawl websites without any programming knowledge! Simply annotate the page you are interested in, and Portia will create a spider to extract data from similar pages.

5. Newspaper

Newspaper can be used to extract news, articles, and content analysis. Use multithreading, support more than 10 languages, and so on.

VI. Beautiful Soup

Beautiful Soup is a Python library that can extract data from HTML or XML files. It can use your favorite converter to achieve the usual way to navigate, find, and modify documents. Beautiful Soup will save you hours or even days of work.

7. Grab

Grab is a Python framework for building Web scrapers. With Grab, you can build a variety of complex page crawling tools, from simple five-line scripts to complex asynchronous site crawlers that handle millions of pages. Grab provides an API for executing network requests and processing received content, such as interacting with the DOM tree of an HTML document.

VIII. Cola

Cola is a distributed crawler framework, for users, only need to write a few specific functions, without paying attention to the details of distributed operation. Tasks are automatically assigned to multiple machines, and the whole process is transparent to users.

IX. Selenium

Selenium is an automated testing tool. It supports all kinds of browsers, including mainstream interface browsers such as Chrome,Safari,Firefox. If a Selenium plug-in is installed in these browsers, the test of Web interface can be easily realized. Selenium supports browser drivers. Selenium supports multi-language development, such as Java,C,Ruby, etc. PhantomJS is used to render and parse JS,Selenium to drive and interface with Python, and Python for post-processing.

10. Python-goose framework

The information that can be extracted by the Python-goose framework includes:

The main content of the article

The main pictures of the article

Any Youtube/Vimeo video embedded in the article

Meta description

Meta label

Thank you for reading! This is the end of this article on "what is the framework of Python crawler?". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.