Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to use hanlp for distributed word segmentation in spark cluster". In daily operation, I believe many people have doubts about how to use hanlp for distributed word segmentation in spark cluster. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to use hanlp for distributed word segmentation in spark cluster". Next, please follow the editor to study!

There are two steps:

Step 1: implement hankcs.hanlp/corpus.io.IIOAdapter

1. Public class HadoopFileIoAdapter implements IIOAdapter {

two。

3. @ Override

4. Public InputStream open (String path) throws IOException {

5. Configuration conf = new Configuration ()

FileSystem fs = FileSystem.get (URI.create (path), conf)

7. Return fs.open (new Path (path))

8.}

9.

10. @ Override

11. Public OutputStream create (String path) throws IOException {

Configuration conf = new Configuration ()

13. FileSystem fs = FileSystem.get (URI.create (path), conf)

14. OutputStream out = fs.create (new Path (path))

15. Return out

16.}

17.}

Step 2: modify the configuration file. Root is the packet on hdfs. Change IOAdapter to the class we implemented above.

Ok, so you can use hanlp for word segmentation on distributed clusters.

At this point, the study on "how to use hanlp for distributed word segmentation in spark clusters" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.