Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to use node.js to climb Zhihu picture related knowledge, the content is detailed and easy to understand, the operation is simple and fast, has a certain reference value, I believe that everyone after reading this article on how to use node.js to climb Zhihu picture article will have a harvest, let's take a look.

Principle

When I first entered the crawler's pit, I didn't have much esoteric theoretical knowledge. To get a picture in Zhihu's post, I summed it up as the following steps.

Prepare a url

Get the html content of this url, analyze the dom structure in it, and traverse to find these beautiful girl pictures url

Get the content of the picture

Write the contents of the picture to the local file

Start to do it.

After we probably know the principle, we can do it.

Prepare a url

This is the simplest. A random search of Zhihu is a lot. We use

Send a picture of a beautiful woman that you think is beautiful?

As an example, let's first analyze the dom structure of this page. In fact, it is very simple. Zhihu's page will contain many types of pictures, including avatars, users' comments on uploaded pictures and so on. Basically, the corresponding image address can be found in noscript.

Get the html content of this url and get the img link in the noscript of the current page

This step we need to know some simple knowledge of nodejs, as well as to use a library called cheerio, this library is used for what, please move to cheerio for details.

Simply put, you can use jQuery on the command line to search and traverse to get the corresponding elements.

So how can I get the html of this post?

Https module using nodejs

Var https = require ('https') getAllHtml (url, callback) {let sHtml ='', _ this = this; https.get (url, (res) = > {res.on ('data', (data) = > {sHtml + = data;}); res.on (' end', () = > {callback.bind (_ this, sHtml) ();}). On ('error', (err) = > {console.log (err);});}

After getting the html of the website through the above operations, we will traverse the address of the image we need.

FilterHtml (sHtml, filePath) {let $= cheerio.load (sHtml), / / pass the website html obtained in the previous step to cheerio.load, and you will get a wrapped jQuery object. You can select elements $Imgs = $('noscript img'), imgData = [], _ this = this; $Imgs.each ((I, e) = > {let imgUrl = $(e) .attr (' src')) like the selector of jQuey; / / take out the corresponding url imgData.push (imgUrl) / / pass url to start download _ this.downloadImg (imgUrl, _ this.filePath, function (err) {console.log (imgUrl + 'has be down');}); console.log (imgData);}

With the url of the picture, how can I download it locally?

We need to use the request library and simply call api combined with node's native api to write the file.

DownloadImg (imgUrl, filePath, callback) {let fileName = this.parseFileName (imgUrl); request (imgUrl) .pipe (fs.createWriteStream ('. /'+ filePath +'/'+ fileName)). On ('close', callback & & callback);} this is the end of the article on "how to use node.js to climb Zhihu pictures". Thank you for reading! I believe you all have a certain understanding of "how to use node.js to climb Zhihu pictures". If you want to learn more, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.