Shulou(Shulou.com)06/01 Report--

This article mainly explains how to initialize and use victoriaMetrics Kublon filter. Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to initialize and use the victoriaMetrics Kublon filter.

Overview of victoriaMetrics Kublon filter

The vmstorage component of victoriaMetrics will receive metrics passed upstream. In real situations, the order of magnitude of metrics or instantaneous metrics may be very scary. If you do not limit the size of the cache, excessive slow insert may occur due to cache miss.

To do this, vmstorage provides two parameters, maxHourlySeries and maxDailySeries, to limit the unique sequence added to the cache per hour / day.

A unique sequence refers to a unique time series, such as metrics {label1= "value1", label2= "value2"} belongs to one time series, but several metrics {label1= "value1", label2= "value2"} with different values belong to the same time series. VictoriaMetrics uses the following ways to obtain the unique identity of the timing:

Func getLabelsHash (labels [] prompbmarshal.Label) uint64 {bb: = labelsHashBufPool.Get () b: = bb.B [: 0] for _, label: = range labels {b = append (b, label.Name...) B = append (b, label.Value...)} h: = xxhash.Sum64 (b) bb.B = b labelsHashBufPool.Put (bb) return h} initialization of speed limiter

VictoriaMetrics uses a concept similar to a speed limiter to limit the unique sequence added per hour / day, but unlike an ordinary speed limiter, it needs to be limited at the sequence level, that is, to determine whether a sequence is a new unique sequence, and if so, it needs to further determine whether the number of new sequences in the cache exceeds the limit over a period of time, rather than simply limiting it at the request level.

HourlySeriesLimiter = bloomfilter.NewLimiter (* maxHourlySeries, time.Hour) dailySeriesLimiter = bloomfilter.NewLimiter (* maxDailySeries, 24*time.Hour)

The following is the function for the new speed limiter, passing in a maximum (sequence) value and a refresh time. In this function:

Initialize a speed limiter whose maximum number of elements is maxItems

A goroutine is enabled, and the speed limiter is reset when the time reaches refreshInterval.

Func NewLimiter (maxItems int, refreshInterval time.Duration) * Limiter {l: = & Limiter {maxItems: maxItems, stopCh: make (chan struct {}) } l.v.Store (newLimiter (maxItems)) / / 1 l.wg.Add (1) go func () {defer l.wg.Done () t: = time.NewTicker (refreshInterval) defer t.Stop () for {select { Case 0.003 {t.Fatalf ("too big false hits share for maxItems=%d:% .5f FalseHits:% d ", maxItems, p, falseHits)}

The following is the Add operation of the filter to add an element to the filter. Instead of using multiple hash functions to calculate the hash value of an element, the Add function changes the value of the same element, and then applies the same hash function to the corresponding value. The number of element changes is limited by hashesCount.

Get the full storage of the filter and convert it to bit units

Convert element h to byte array to facilitate xxhash.Sum64 calculation

HashesCount hashes will be performed later to reduce the leakage rate.

Calculate the hash of element h

Increment the element h to prepare for the next hash

Obtaining the bit range of an element by residue method

Get the slot where the element resides (that is, the bit range of uint64 size)

Gets the bit bit in the slot where the element is located. A bit of 1 indicates that the element exists, and a value of 0 indicates that the element does not exist.

Gets the mask of the bit where the element is located

The numeric value of the slot where the element is loaded

If the w & mask result is 0, the element does not exist

Setting the bit bit (mask) of the element in the slot (w) where the element is located to 1 indicates that the element has been added

Because the Add function can be accessed concurrently, bits [I] may be modified by other operations, so the existence of this element in bits [I] needs to be set by reloading (14) and looping.

Func (f * filter) Add (h uint64) bool {bits: = f.bits maxBits: = uint64 (len (bits)) * 64 / / 1 bp: = (* [8] byte) (unsafe.Pointer (& h)) / / 2b: = bp [:] isNew: = false for i: = 0; I < hashesCount ITunes + {/ / 3 hi: = xxhash.Sum64 (b) / / 4 hackers + / / 5 idx: = hi% maxBits / / 6 I: = idx / 64 / / 7 j: = idx% 64 / / 8 mask: = uint64 (1)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.