Shulou(Shulou.com)06/01 Report--

This article shows you how to use JAVA to write a crawler, the content is concise and easy to understand, can definitely brighten your eyes, through the detailed introduction of this article, I hope you can learn something.

I actually wrote this article a long time ago. I'll reorganize it this time. Java writing crawler may not be tried by many friends, it may be due to the lack of information in this area, or it may be that Python is too convenient to write crawlers.

Basic concept

Jsoup is a Java library for working with real-world HTML. It provides a very convenient API for extracting and manipulating data, using the best of DOM, CSS, and jquery-like methods.

The above is the official explanation given by jsoup, which means jsoup is a HTML parser for Java, which can directly parse a URL address and HTML text content. It provides a very labor-saving API that can fetch and manipulate data through DOM,CSS and jQuery-like operations.

Generally speaking, it can help us parse the HTML page and grab the contents of the html.

Start writing code.

Our goal is to grab the information on the rookie's notes (article title and link).

Public static void main (String [] args) {try {/ / the following line of code connects to our target site, and get goes to his static HTML code Document document=Jsoup.connect ("http://www.runoob.com/w3cnote").get(); / / Let's print the acquired document and see what's inside? System.out.println (document);} catch (IOException e) {e.printStackTrace ();}}

Look at the result of our code running:

You will find that we have obtained the HTML source code of the website "Rookie Notes" through this sentence.

Let's analyze this string of html source code

We found that these two were exactly the data we wanted, and we continued to grab them.

Public static void main (String [] args) {try {Document document=Jsoup.connect ("http://www.runoob.com/w3cnote").get();") / / the next line of code is that we further crawl to the specific HTML module, div represents the tag, / / the following post-intro represents the class of div / / since the div.post-intro tag has multiple tags (one for each title), we first get all its Elements elements=document.select ("div.post-intro"). / / Let's go through it, because div.post-intro has a lot of for (int iTunes).

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.