Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to use Nutch to crawl websites that need to log in". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The background management program that comes with Tomcat itself requires users to log in. How can such a website be crawled with Nutch? Nutch can handle the slightly simpler authentication of Http authentication (BASIC, DIGEST), but Nutch is powerless for the popular user-defined Form forms to submit data authentication in the form of Post or Get, let alone complex CAPTCHA.

Here's a simple example of how to configure Nutch to crawl sites that need Http authentication (BASIC, DIGEST).

1. Modify the Tomcat configuration file conf/tomcat-users.xml, add the following configuration, and then restart, and one user can access all resources:

2. Modify the Nutch configuration file conf/httpclient-auth.xml by adding the following configuration to specify the user name and password you need to show when visiting a specific website:

3. Enable the httpclient plug-in, re-specify the value of the configuration item plugin.includes in nutch-site.xml, and change protocol-http to protocol-httpclient:

Plugin.includes protocol-httpclient | urlfilter-regex | parse- (html | tika) | index- (basic | anchor) | indexer-solr | scoring-opic | urlnormalizer- (pass | regex | basic)

4. URL file to be injected:

Mkdir urls echo 'http://localhost:8080/' > urls/url

5. Modify URL filtered file conf/regex-urlfilter.txt to limit the crawling range:

#-[? * @ =] + ^ http://localhost:8080/-.

6. Run the crawler. The parameters are:

Bin/nutch crawl urls-dir data-solr http://localhost:8983/solr/collection1-depth 30 &

7. Check the crawled URL and its status and find it successful!

This is the end of the content of "how to use Nutch to crawl the website you need to log in". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.