Shulou(Shulou.com)06/02 Report--

Python crawler position to master the relevant skills and knowledge, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

Python crawler position requires more skills, for example, first of all, you have to know Python language, secondly, you need to understand web markup language, that is, HTML, and furthermore, you also need to know some knowledge of operation and maintenance. Anyway, it is a lot. Let me tell you in detail what related skills crawlers need.

1. Basic coding basis (at least one programming language), which is necessary for any programming work. You have to know the basic data structure. Data names and worthwhile correspondence (dictionaries), processing of some url (lists), and so on. In fact, the more firmly you master, the better. Crawlers are not a simple job, nor are they more demanding on programming languages than other jobs. It is always good to be familiar with the programming language you use and the relevant frameworks and libraries. I mainly use Python, and there are also people who use Java to write crawlers. In theory, any language can write crawlers, but it is best to choose a related library and develop a language quickly. Writing in C must be asking for trouble.

2. Task queue when the crawler task is very large, it is not appropriate to write a program to run down:

If you make a mistake in the middle, stop and start all over again?

How do I know where the program failed?

How can I divide the work if I have two machines?

So we need a kind of task queue, and its function is to put all the web pages that we plan to crawl into the task queue. Then worker takes one execution from the queue, if one fails, record it, and then execute the next one. In this way, worker can be executed one by one. Also increased scalability, hundreds of millions of tasks in the queue is no problem, if necessary, you can add worker, just like eating without a pair of chopsticks. Commonly used task queues are kafka,beanstalkd,celery and so on.

3, the database needless to say, data preservation must be the database. But sometimes some small data can be saved as json or csv and so on. Sometimes I want to grab some pictures and save the file directly according to the folder. It is recommended to use NoSQL databases, such as mongodb, because the data captured by crawlers are usually fields-worthy of correspondence. Some fields have or do not exist on some websites. Mongo is more flexible in this respect. Moreover, the data relationship crawled by crawlers is very weak, and table-to-table relationships are rarely used.

4. HTTP knowledge HTTP knowledge is a necessary skill. Because you want to climb the web page, you must understand the web page. First of all, you should understand the parsing methods of html documents, such as child nodes, parent nodes, attributes, and so on. The web page we see is colorful, but it is handled by the browser, and the original web page is made up of a lot of tags. It is best to use html's parser, because there will be a lot of holes if you use regular matching yourself. I personally like xpath very much, cross-language, the expression is better than the price, but also has shortcomings, regular, logical judgment is a bit awkward. The HTTP protocol should be understood. The HTTP protocol itself is stateless, so how is login implemented? This requires a look at session and cookies. The difference between the GET method and the POST method (in fact, there is no difference except literally).

Be proficient in browsers. The process of crawlers is actually a simulation of the human process of browsing data. So how do browsers visit a website, you have to learn to observe, how to observe? Developer Tools! Chrome's Developer Tools provides all the information about visiting the website. You can see all outgoing requests from traffic. The copy as curl function can generate a curl request that is exactly the same as the browser request! My general process of writing a crawler is like this: first visit it with a browser, then copy as curl to see what header,cookies there are, then simulate the request in code, and finally save the result of processing the request.

5. The topic of operation and maintenance has a lot to say, and in actual work, the time spent on operation and development is almost or even more. Maintaining reptiles that are already working is a heavy task. As the working time increases, we usually learn to make the written crawlers easier to maintain. For example, the log system of the crawler, the statistics of the amount of data and so on. It is not reasonable to separate the crawler engineer from the operation and maintenance staff, because if a crawler does not work, the reason may be that the structure of the page to be caught has been updated, or it may appear on the system, or it may be that the anti-scraping strategy was not found when the crawler was developed. there may be a problem after launching, or it may be that the other website found that you are a crawler and blocked you, so generally speaking, the development of a crawler should take care of the operation and maintenance staff.

After reading the above, do you have any ways to master the relevant skills and knowledge to master the Python crawler position? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.