Shulou(Shulou.com)06/03 Report--

This article mainly explains "what does the web crawler refer to". Interested friends might as well take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what does a web crawler mean?"

1. What is a reptile?

Web crawlers (also known as web spiders and web robots) are programs and scripts that automatically capture information on the World wide Web according to certain rules. Other less commonly used names are ants, automatic indexing, simulators, or worms.

Generally speaking, we compare the Internet to a big spider web, and the resources of each website are compared to the nodes on the spider web. Reptiles are like spiders, finding target nodes and obtaining resources on this spider web according to the designed routes and rules. You can use Sun http, a professional crawler aid.

2. Why do we need to use reptiles?

You can imagine a scene: you admire a Weibo celebrity very much and are fascinated by his Weibo. You want to extract every word he has said on Weibo over the past ten years and make celebrity quotes. What are you going to do at this time? Go to Ctrl+C and Ctrl+V manually? This method is true, we can do this when the amount of data is very small, but when the data is thousands, do you still need to do this?

Let's imagine another scenario: if you want to be a news aggregator, you need to visit several news sites regularly every day to get the latest news. We call it a RSS subscription. Do you regularly go to various subscription sites to copy news? I'm afraid it's hard for a person to do that.

In the above two scenarios, the problem can be easily solved by using crawler technology. Therefore, we can see that crawler technology can mainly help us to do two things: one is the data acquisition requirement, which is mainly aimed at obtaining a large amount of data under specific rules; the other is the automation requirement, which is mainly used in similar information aggregation and search.

3. Classification of reptiles: reptiles can be divided into general reptiles and focused reptiles.

General web crawler, also known as full-web crawler (ScalableWebCrawler), the crawler object extends from some seed URL to the whole network, mainly collecting data from search engines and large Internet service providers. This kind of web crawler has a large crawling range and number, high requirements for crawling speed and storage space, and low order for crawling pages. For example, our common Baidu and Google search. When we enter keywords, they will find the web pages related to keywords from the whole network and present them to us in a certain order.

Focused web crawler (FocusedCrawler) refers to selectively crawling web crawlers related to predefined topics. Compared with general web crawlers, focused crawlers only need to crawl specific web pages, and the breadth of crawling will be much smaller. For example, we need to grab the fund data of the Oriental Wealth Network, and we only need to make crawling rules for the website of the Oriental Fortune Network.

In general, general-purpose crawlers are similar to spiders and need to look for specific food, but because they don't know which node of the spider's web, they can only start from one node. If you encounter a node, just take a look. Where there is food, there is food. If this node indicates that one node has food, follow the instructions to find the next node. And the focus web crawler is that the spider knows which node has food, and it only needs one node to get the food.

4. The process of browsing the web.

In the process of browsing the web, we may see a lot of beautiful pictures

This process is actually that after the user enters the site, the user finds the server host through the DNS server and sends a request to the server. After the server analysis, the browser HTML, JS, CSS and other files sent to the user are analyzed by the browser, and the user can see a variety of images.

Therefore, the web pages seen by users are essentially composed of HTML code, and reptiles crawl these contents. By analyzing and filtering these HTML codes, the acquisition of images, text and other resources is realized.

5. The meaning of URL.

URL, that is, uniform resource locator, which is what we call a website, simply indicates the location and access method of resources that can be obtained from the Internet, and is the address of standard resources on the Internet. Every file on the Internet has a unique URL that contains information indicating the location of the file and what the browser should do with it.

At this point, I believe you have a deeper understanding of "what the web crawler refers to". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.