Shulou(Shulou.com)06/03 Report--

This article mainly introduces the novice crawler collection easy to encounter what problems, with a certain reference value, interested friends can refer to, I hope you read this article after a great harvest, the following let Xiaobian take you to understand.

1. The coding problem.

At present, the two most common encodings of websites are utf-8 or gbk. When the source website code we collected is inconsistent with the code stored in our database, such as http://163.com code using gbk, and we need to store utf-8 encoded data, then we can use the encode() and decode() methods provided in Python to convert, such as: content = content.decode ('gbk ', 'ignore') #to convert gbk encoding to unicode encoding.

content = content.encode ('utf-8','ignore') #Convert unicode encoding to utf-8 encoding

unicode code appears in the middle, we need to switch to unicode code in order to convert to gbk or utf-8.

2. Incremental crawling.

Incremental crawling is a crawler that does not repeatedly download downloaded content. In order to achieve incremental crawling, we need to use a new concept-URL pool. URL pool is used to manage all URLs in a unified way. We keep track of what our python crawlers access through URL pools to avoid duplication. The purpose of the URL pool can also be achieved breakpoint climbing, etc. Breakpoint continued crawling is to continue crawling websites that have not been crawled before.

3. Crawlers are prohibited.

Crawlers will bring a lot of load to the server, so many servers will limit crawlers or even disable crawlers. It is well known to construct reasonable http access headers, such as user-agent field values. However, there are many other problems to avoid being banned, such as slowing down the crawler's access speed, making the crawler's access path consistent with the user's access path, using dynamic ip addresses, and so on.

Thank you for reading this article carefully. I hope that Xiaobian will share "What are the problems that are easy to encounter when collecting novice crawlers". This article is helpful to everyone. At the same time, I hope that everyone will support you a lot. Pay attention to the industry information channel. More relevant knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.