Shulou(Shulou.com)06/01 Report--

This article "Pytho crawler beginners need to master what knowledge" most people do not understand, so the editor summed up the following content, detailed, clear steps, with a certain reference value, I hope you can get something after reading this article, let's take a look at this "Pytho crawler beginners need to master what knowledge" article.

1) JAVA

The syntax of Java is relatively regular, using strict object-oriented programming methods, and there are many large-scale development frameworks, which are more suitable for enterprise applications. Java has a long learning curve, not only to learn language-related features, but also to learn object-oriented software construction methods, after which we need to learn some framework usage.

(1) uses: Android & IOS application development, video game development, desktop GUIs (graphical user pages), software development, architecture, etc.

(2) advantages: object-oriented open source, cross-platform, strong market demand; the cornerstone of Android development is the mainstream language of Web development.

(3) disadvantages: it takes up a lot of memory, takes a long time to start, and does not directly support hardware-level processing.

2) Python

Python is a dynamic and flexible interpretive language, and Python is used from software development to Web development.

Because it is an interpretive scripting language, it is suitable for lightweight development. Python is a relatively easy language to learn.

(1) uses: crawlers, Web development, video game development, desktop GUIs (graphical user pages), software development, architecture, etc.

(2) advantages: dynamic interpretation, strong open source class library, high development efficiency, open source, flexibility, low entry and easy to use.

(3) disadvantages: the running speed is lower than that of compiled languages, and it is weak in the field of mobile computing.

3) C++

C++ is closer to the bottom layer, so it is convenient to manipulate memory directly. C++ not only has the practical characteristics of efficient computer operation, but also strives to improve the programming quality of large-scale programs and the problem description ability of programming languages. C++ is efficient, and it is easy to build large-scale software, so it is suitable for software with high efficiency. The content of C++ is very complex, and the language has evolved for decades, so it is difficult to learn and the development efficiency is low.

1) uses: neural networks in machine learning, large game programming, background services, desktop software, etc.

2) advantages: high efficiency, security, object-oriented, simplicity, good reusability, etc.

3) disadvantages: there is no garbage collection mechanism, which may cause memory leakage; learning is difficult and the development efficiency is relatively low.

By searching for "crawler" on GitHub, we can find that Python accounts for 80% of the open source crawler projects on the market.

And engineers with more than two years of experience in Python crawlers are generally paid more than Java and C++ crawler engineers on the market.

Therefore, Python is the best choice for developing crawlers. So what knowledge does a beginner know that he can use Python to develop crawlers?

1. The network

Urllib-Network Library (stdlib).

Requests-Network library.

Grab-Network library (based on pycurl).

Pycurl-Network library (bind libcurl).

Urllib3-Python HTTP library, secure connection pool, support file post, high availability.

Httplib2-Network library.

RoboBrowser-A simple, Python-styled Python library that browses the web without a separate browser.

MechanicalSoup-A Python library that automatically interacts with websites.

Mechanize-stateful, programmable Web browsing library.

Socket-underlying network interface (stdlib).

2. HTML/XML parser

Lxml-C language to write efficient HTML/ XML processing library. XPath is supported.

Cssselect-parses the DOM tree and CSS selector.

Pyquery-parses the DOM tree and jQuery selector.

BeautifulSoup-inefficient HTML/ XML processing library, pure Python implementation.

Html5lib-the DOM that generates the HTML/ XML document according to the WHATWG specification. The specification is used in all browsers today.

Feedparser-parses the RSS/ATOM feeds.

MarkupSafe-A string that is securely escaped for XML/HTML/XHTML.

3. Browser automation and simulation

Selenium-automates real browsers (Chrome, Firefox, Opera, IE).

Ghost.py-Encapsulation of PyQt's webkit (requires PyQT).

Spynner-Encapsulation of PyQt's webkit (requires PyQT).

Splinter-General API browser emulator (selenium web driver, Django client, Zope).

4. Multiple processing

The thread running of the threading-Python standard library. It is very effective for Istroke O-intensive tasks. It is not useful for CPU binding tasks because of python GIL.

Multiprocessing-the standard Python library runs multiple processes.

Celery-Asynchronous task queue / job queue based on distributed messaging.

The concurrent-futures-concurrent-futures module provides a high-level interface for invoking asynchronous execution.

5. Asynchronous network programming library

Asyncio-(Python standard library in Python version 3.4 + and above) Asynchronous Icano, time loops, collaborative programs and tasks.

Gevent-A protocol-based Python network library that uses greenlet.

6. Web page content extraction

Text and metadata for HTML pages

Newspaper-use Python for news extraction, article extraction, and content curating.

Html2text-converts HTML to Markdown format text.

Python-goose-HTML content / article extractor.

Lassie-A humanized web content retrieval tool

After understanding the above knowledge points, what are the ways in which we can obtain network data?

Method 1: the browser submits the request-> download the web page code-> parse into the page.

Method 2: simulate the browser to send a request (get the web page code)-> extract useful data-> store it in a database or file

What a crawler has to do is way 2.

1. Initiate a request

Use the http library to initiate a request to the target site, that is, send a Request

Request includes: request header, request body, etc.

Request module bug: unable to execute JS and CSS code

2. Get the response content

If the server responds normally, you will get a Response

Response includes: html,json, pictures, videos, etc.

3. Analyze the content

Parsing html data: regular expressions (RE modules), third-party parsing libraries such as Beautifulsoup,pyquery, etc.

Parsing json data: json module

Parsing binary data: writing to a file in wb

4. Save data

Database (MySQL,Mongdb, Redis), file

At present, there are mainly eight open source crawler projects on the market:

1.Scrapy

Scrapy is an application framework written to crawl website data and extract structural data. It can be used in a series of programs, including data mining, information processing, or storing historical data. With this framework, you can easily climb down data such as Amazon product information.

2.PySpider

Pyspider is a powerful web crawler system implemented by python, which can write scripts, schedule functions and view crawl results in real time on the browser interface. The back end uses common databases to store crawl results, and can set tasks and task priorities regularly.

3.Crawley

Crawley can crawl the content of the corresponding website at high speed, support relational and non-relational databases, and the data can be exported to JSON, XML and so on.

4.Portia

Portia is an open source visual crawler tool that allows you to crawl websites without any programming knowledge! Simply annotate the page you are interested in, and Portia will create a spider to extract data from similar pages.

5.Newspaper

Newspaper can be used to extract news, articles, and content analysis. Use multithreading, support more than 10 languages, and so on.

6.Beautiful Soup

Beautiful Soup is a Python library that can extract data from HTML or XML files. It can use your favorite converter to achieve the usual way to navigate, find, and modify documents. Beautiful Soup will save you hours or even days of work.

7.Grab

Grab is a Python framework for building Web scrapers. With Grab, you can build a variety of complex page crawling tools, from simple five-line scripts to complex asynchronous site crawlers that handle millions of pages. Grab provides an API for executing network requests and processing received content, such as interacting with the DOM tree of an HTML document.

8.Cola

Cola is a distributed crawler framework, for users, only need to write a few specific functions, without paying attention to the details of distributed operation. Tasks are automatically assigned to multiple machines, and the whole process is transparent to users.

The above is about the content of this article on "what knowledge do Pytho crawler beginners need to master". I believe we all have a certain understanding. I hope the content shared by the editor will be helpful to you. If you want to know more related knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.