Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what is the specific content of the directory of the Nutch dataset". In the daily operation, I believe that many people have doubts about the specific content of the directory of the Nutch dataset. The editor consulted all kinds of materials and sorted out a simple and easy-to-use method of operation. I hope it will be helpful for everyone to answer the question of "what is the specific content of the directory of the Nutch dataset?" Next, please follow the editor to study!

The contents of the directory of the Nutch dataset

# Export crawldb

Bin/nutch readdb data/crawldb/-dump data/crawldb_dump

# View crawldb

The Crawldb content mainly includes the details of the url address, such as the crawling status, the time it took to crawl the page, and the importance score of the page.

# View linkdb

# View segments

A crawl will produce many segment, which stores the pages captured by the crawler during a crawl and the index of these pages. When a crawler crawls, it generates a prefetching list (fetch list) needed for each crawling cycle according to a certain crawling strategy according to the link relationship in crawldb, and then the Fetcher class grabs these web pages and indexes them through the URL in the prefetching list, and then stores them into segments.

# View the segments directory

Content: save the source content crawled by fetcher, html script

Crawl_fetch: contains the status of each crawled page

Crawl_generate: contains a list of URLs crawled

Crawl_parse: the external link address containing the URL, used to update the crawldb database

Parse_data: contains external links and metadata for each page

Parse_text: contains parsed text for each crawled page

At this point, the study on "what is the specific content of the directory of the Nutch dataset" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.