How to remove duplicate lines from large text by JAVA
In this issue, the editor will bring you about how to remove duplicate lines from large text in JAVA. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.
To repeat the line, it's easy to write in SQL, just a SELECT DISTINCT. FROM . But the file can not directly use SQL, want to use SQL also have to find a database to build tables, it is also very troublesome. If it's a small file, it's not too hard to deal with it with java, something like this:
String file = "d:/urls.txt"
ArrayList result = new ArrayList ()
BufferedReader br=null
Try {
Br = new BufferedReader (new FileReader (file))
String row = br.readLine ()
While (rowdy flowers null) {
If (! result.contains (row)) {
Result.add (row)
}
Row = br.readLine ()
}
} finally {
If (brinkmanship null)
Try {
Br.close ()
} catch (IOException e) {
}
}
But if it's a large file, the code must be dead. At this time, you have to use the file as a cache, or sort first, which is very troublesome.
In this case, it will be much easier if you have an aggregator, using only one sentence in SPL:
File ("d:/urls.txt"). Cursor (). Groupx (# 1). Fetch ()
You can even write SQL directly to the file:
$select distinct # 1 from d:/urls.txt
The above is the editor for you to share the JAVA how to achieve large text to remove repetition lines, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.
