Shulou(Shulou.com)06/01 Report--

This article analyzes "how to remove duplicate values in big data's Chinese text by line". The content is detailed and easy to understand. Friends who are interested in "how to remove duplicate values by line" can follow the editor's train of thought to read it slowly and deeply. I hope it will be helpful to you after reading. Let's follow the editor to learn more about "how to remove duplicate values in big data's Chinese text by line".

To repeat the line, it's easy to write in SQL, just a SELECT DISTINCT. FROM . But the file can not directly use SQL, want to use SQL also have to find a database to build tables, it is also very troublesome. If you write a program directly, the simple idea is to open the file first and then read the text line by line. Then the text is compared with the unique value in the cache, the duplicate text is discarded, otherwise it is appended to the cache, and after the file is read, the deduplicated content in the cache is written out to the output file.

Although the above ideas are simple, they can only deal with small files, not large ones. When the file is very large (there is no room for memory), it can only be cached with the file, or the source file can be sorted first and then repeated. However, it is still a bit difficult and troublesome to write by yourself to achieve out-of-memory caching or sorting of large files.

In this case, it will be much easier if you have an aggregator, using only one sentence in SPL:

File ("d:/urls.txt"). Cursor (). Groupx (# 1). Fetch ()

You can even write SQL directly to the file:

$select distinct # 1 from d:/urls.txt

This is the end of the sharing of big data's Chinese text on how to remove duplicate values by line. I hope the above content can improve everyone. If you want to learn more knowledge, please pay more attention to the editor's updates. Thank you for following the website!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.