Shulou(Shulou.com)06/03 Report--

This article mainly introduces "what does web crawler mean?". In daily operation, I believe that many people have doubts about what web crawler means. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the question of "what exactly is the meaning of web crawler?" Next, please follow the editor to study!

At the beginning of the Internet, it is to make it easier for people to share data and communicate. The Internet is a bridge that connects people all over the world. The click and browse of the website are artificial, and the people chatting with you are also living people. However, with the development of technology and people's desire for data, all kinds of network robots have appeared. at this time, you don't know whether you are chatting with a person or a dog on the other end of the screen, and you don't know whether the number of views of your website is clicked by people or crawled out by machines.

On the surface, there are all kinds of people on the Internet; secretly, there are all kinds of web crawlers.

I. Web crawlers in the era of search engines

With regard to the concept of web crawlers, let's take a look at the definition above in Wikipedia:

Web crawler (English: web crawler), also known as web spider (spider), is a web robot used to browse the World wide Web automatically. Its purpose is generally to compile a network index.

The compilation of web indexes mentioned here is what search engines do. We are no stranger to search engines. Search engines such as Google and Baidu may help us get information quickly every day. Little apes may ask, what is the working process of search engines?

First of all, there are web crawlers constantly crawling the web pages of each website and storing them in the database of the search engine.

Then, the indexing program reads the web pages of the database to clean up and establish an inverted index.

Finally, the search program receives the user's query keywords, goes to the index to find the relevant content, and presents the most relevant and best results to the user through a certain sorting algorithm (Pagerank, etc.).

It seems that the three simple parts constitute a powerful and complex search engine system. Web crawler is the most basic and important part of it, which determines the integrity and richness of search engine data. We also see that the main role of web crawlers is to obtain data.

To put it simply, a web crawler is an automated tool for obtaining public data on the Internet.

It should be emphasized here that web crawlers crawl public data on the Internet, rather than non-public data obtained by illegally hacking into the website server through special technology.

You may want to ask, what is "open data"? In short, it is the data that users can browse and obtain publicly on the website.

Although the data is public, legal disputes arise when a person or organization (such as a search engine) collects a large amount of data and profits from it, which will also upset the data producer, the website. For example, in the early years, Google was involved in a lawsuit.

Websites look at search engines because search engines grab their own content and make a profit, but they are also happy because of the traffic brought by search engines, so websites take the initiative to conduct search engine optimization (SEO, Search Engine Optimization), that is, to tell search engines, my content here is good, come and grab it!

The game between search engines and websites has spawned a gentleman's agreement: robots.txt. The website puts this file on its own website to tell the crawler which content can be caught and which can not be caught; the search engine reads the robots.txt of the site to know its crawling scope, and also uses User-Agent to identify itself to the site when visiting the site (this is also a gentleman's agreement, which is technically easy to pretend to be others). For example, the crawler of Google is called Googlebot, and the crawler of Baidu is called Baiduspider. In this way, the two coexist peacefully and benefit each other.

Second, the web crawlers in the era of big data

With the development of the times, data is becoming more and more important. "big data" has become a topic of discussion in various industries, people's desire for data has become greedy, data has become "oil", and reptiles have become "drilling rigs".

To get oil, people use drilling rigs; to get data, people use crawlers. In order to obtain data, people drill the Internet "riddled with holes". Haha, it's a bit of an exaggeration. But people's access to data, the gentleman's agreement that has been broken, and the websites played a game of cat and mouse, launched a contest between as virtue rises one foot and vice rises ten.

Why is it a contest? Because the behavior of a large number of crawlers will bring network bandwidth, server computing power and other aspects of great pressure, but hardly bring any benefits. In order to reduce this unprofitable pressure and prevent their own data from being collected centrally by others, the website must limit the crawler through technical means; on the other hand, the crawler tries to break through this restriction in order to obtain oil-like data.

The understanding of this contest is more thorough by looking at living examples.

Did you spend dozens of yuan to ask some software to help you grab a train ticket?

Attack: ticket crawlers will constantly visit 12306 to get train ticket seat data, and then buy train tickets

Prevention: 12306 website has an abnormal authentication code, people often identify errors.

All kinds of second killing make you very hurt!

Attack: study the second kill mechanism of the website, write the crawler in advance, the second kill time, people are no faster than machines

Defense: some second-kill propaganda is very effective and lazy to guard against; some second-kill mechanisms are so complex that it is difficult for you to write the corresponding crawler; some second-kill success is found to be cheating will also be cancelled.

Crawlers are becoming more and more unscrupulous, and websites have to use a variety of technical means to prohibit or restrict crawlers. These measures roughly include:

Use accounts to protect data, which is visible only to logged-in users

Data is loaded asynchronously multiple times

Limit the frequency of IP access, or even block IP

Enter a CAPTCHA code to gain access

The data is encrypted on the server side and decrypted on the browser side

……

And these means are also the problems that crawlers have to solve and break through in the implementation of technology.

Third, the self-restraint of web crawlers.

After reading the description of the cat-and-mouse game above, the little apes can't help but ask, will the confrontation between websites and crawlers cause legal problems?

This is a good question, and it is worth thinking about by every crawler developer.

Reptiles as a technology itself may not be good or evil, but the people who use it can be divided into good and evil. How to use crawlers and how to use crawled data may cause potential legal problems. As small apes in technological development, all should think about this problem. No matter what the purpose, web crawlers can not break through the bottom line of the law, but also abide by certain guidelines:

Follow the robots.txt protocol

Avoid visiting the target website with high concurrency in a short time and avoid interfering with the normal operation of the target website.

Do not grab personal information, such as mobile phone address book, etc.

Use the captured data to pay attention to privacy protection and legal compliance.

At this point, the study of "what exactly does a web crawler mean?" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.