Shulou(Shulou.com)06/03 Report--

Dealing with massive data is a necessary skill for big data engineers. It is a very necessary work to excavate valuable information through mining and analyzing PB-level data to provide a basis for enterprises or governments to make correct decisions. The following are common massive data processing methods!

1. Bloom filter

Bloom filter is a binary vector data structure, which has good space efficiency and time efficiency, and can be used to detect whether an element belongs to a set. The advantage of this method is that its insertion and query time are constant, and it does not save the element itself, so it has good security, but its accuracy is slightly lower because of its algorithm. It can be determined that there is no data that does not exist, and that the existing data does not necessarily exist, so it is suitable for situations where low error rate can be tolerated.

2. Hash

Hash is a hash function, which is a function that compresses a message of arbitrary length to a message digest of a fixed length. According to the different processing requirements, there are different Hash functions, and there are corresponding Hash methods for strings, integers and permutations. The commonly used Hash construction methods include direct addressing method, digital analysis method, square center method, folding method, random number method and residual method.

3. BitMap

BitMap is a method that uses an array to indicate the existence of some data. It can quickly find, judge and delete the data. Generally speaking, the data range is less than 10 times that of int, and Bloom can be regarded as an extension of BitMap.

4. Heap

Heap is a general term for a special data structure in computer science, which is an array object that can be regarded as a tree. Its principle is to find out the first k of the n numbers to be found to build a small top heap. Then read the following elements in turn and compare them with the top of the small top heap. If the current elements are small or equal, continue to read the following elements. If the current element is large, replace the heap top element with the current element, and then adjust the small top heap. The maximum heap requires the front k to be small, the minimum heap for the front k, and the double heap for the median.

5. Double bucket

Double bucket is not a data structure, but an algorithm idea, similar to the divide-and-conquer idea. Because the range of elements is so large that the direct addressing table cannot be used, the scope is determined step by step through multiple partitions, and finally within an acceptable range. The double-bucket method is generally suitable for finding the k-th large number, looking for the median, and looking for numbers that do not repeat or repeat.

6. Database optimization method

Massive data are stored in the database, how to extract useful information from the database needs to use the database optimization method. Common database optimization methods include data partitioning, indexing, caching mechanism, batch processing, optimizing query statements, using sampled data for data mining and so on.

7. Inverted index

Inverted index is the most commonly used storage method for search engines by search engine companies at present, which is used to store the mapping of the storage location of a word in a document or a group of documents under full-text search. When dealing with complex multi-keywords, the logical operations such as union and intersection of queries can be completed in the inverted table, and then the results can be accessed after getting the results. in this way, the query of records can be converted into the operation of address set, and there is no need to randomly access each record, so as to improve the speed of search.

8. External sorting

External sorting is the sorting of large files. Due to memory limitations, all the contents to be sorted cannot be read into memory at once for sorting. It is necessary to exchange data between memory and external memory for many times to sort the whole file. The commonly used external sorting method is merging and sorting, that is, several sub-files are generated first, and these sub-files are sorted respectively. Then these sub-files are merged many times, so that the primary key of the orderly merge segment is expanded, and finally a single merge segment of the whole file is formed on the external memory.

External sorting is suitable for big data's sorting and weight removal, but the defect of external sorting is that it consumes a lot of IO and is not efficient.

9. Trie tree

Trie tree is a kind of multi-tree result for fast string retrieval, which uses the common prefix of strings to reduce space overhead. It is often used by search engine systems for file word frequency statistics. The advantage is that it minimizes unnecessary string comparisons and is more efficient than hash tables. It is suitable for situations where the amount of data is large and repetitive, but the type of data is small and can be put into memory.

10. MapReduce

MapReduce is one of the core technologies of cloud computing. It is a distributed programming model that simplifies parallel computing. The main purpose is for large-scale cluster systems to work in parallel on the big data set and for parallel computing of large-scale data.

The above are common methods to deal with massive data, which can be selected and used according to the characteristics of the data to be processed.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.