Shulou(Shulou.com)06/03 Report--

Spark Chinese word segmentation

First, import the required word segmentation package

Import org.ansj.domain.Termimport org.ansj.recognition.impl.StopRecognitionimport org.ansj.splitWord.analysis.ToAnalysis

2. Stop word filtering

Def filter (stopWords: Array [String]): StopRecognition = {/ / add stop words val filter = new StopRecognition filter.insertStopNatures ("w") / / filter punctuation filter.insertStopNatures ("m") / / filter m pattern filter.insertStopNatures ("null") / / filter null filter.insertStopNatures (") / / filter filter.insertStopRegexes (" ^ [a-zA-Z] {1 ") } ") / / filter English alphabet filter.insertStopRegexes (" ^ [0-9] + ") / / filter number filter.insertStopRegexes (" [^ a-zA-Z0-9\\ u4e00 -\ u9fa5] + ") filter.insertStopRegexes ("\ t ") for (x

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.