Shulou(Shulou.com)05/31 Report--

This article mainly explains how to insert data into Prometheus time series database. Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let Xiaobian take you to learn "How to insert data into Prometheus time series database"!

preface

In the previous article, I explained in detail the storage structure of Prometheus temporal database in memory and disk. With the previous foreshadowing, the author can explain the insertion process of data in this article.

Insertion of monitoring data

I won't discuss Promtheus grabbing data from endpoints here. It's just about how data gets inserted into Prometheus. Corresponding method:

func (a *headAppender) Add(lset labels.Labels, t int64, v float64) (uint64, error) { ...... //If there is no series corresponding to lset, create one. Also put the new series into the inverted Posting map s, created := a.head.getOrCreate(lset.Hash(), lset) if created { //if a new one is created, put the new one into a.series a.series = append(a.series, record.RefSeries{ Ref: s.ref, Labels: lset, }) } return s.ref, a.AddFast(s.ref, t, v) }

Let's take the following add function call as an example:

app.Add(labels.FromStrings("foo", "bar"), 0, 0)

The first is getOrCreate, as the name implies, if it does not exist, create one. The creation process includes maintenance of seriesHashMap/Postings/LabelIndex. As shown below:

Then there's the AddFast method.

func (a *headAppender) AddFast(ref uint64, t int64, v float64) error{ //remove the corresponding memSeries s := a.head.series.getByID(ref) ...... //Set to wait for submission s.pendingCommit=true ...... //For the transaction concept, put it into temp storage, wait for the real commit time and then write memSeries a.samples = append(a.samples, record.RefSample{Ref: ref,T: t,V: v,}) // }

Prometheus does not add data points directly to memSeries(i.e., the structure used by the query), but instead adds them to a temporary slice of samples. At the same time, the memSeries corresponding to this data point is synchronously added to another sampleSeries.

transaction visibility

Why would you do that? In order to implement commit semantics, the data is visible (can be queried) only after committing. Otherwise, these data cannot be seen. The commit action is mainly WAL(Write Ahead Log) and writing headerAppender.samples data to its corresponding memSeries. In this way, the query can see these data, as shown in the following figure:

WAL

Because Prometheus 'most recent data is stored in memory, it does not prevent server downtime from losing data. He wrote the journal WAL before committing. When the service restarts, retrieve the information from the WAL log and replay it.

For performance, Prometheus adds another goroutine to sync files, so there is no guarantee that WAL will not be lost. There is no guarantee that the monitoring data will not be lost. This is also determined by the characteristics of the monitoring service.

Write code as:

commit() |=> func (a *headAppender) log() error { ...... //Write the corresponding series information to WAL if len(a.series) > 0 { rec = enc.Series(a.series, buf) buf = rec[:0] if err := a.head.wal.Log(rec); err != nil { return errors.Wrap(err, "log series") } } ...... //Write real samples to WAL if len(a.samples) > 0 { rec = enc.Samples(a.samples, buf) buf = rec[:0] if err := a.head.wal.Log(rec); err != nil { return errors.Wrap(err, "log samples") } } }

The corresponding WAL log format is:

Series records

┌────────────────────────────────────────────┐ │ type = 1 │ ├────────────────────────────────────────────┤ │ ┌─────────┬──────────────────────────────┐ │ │ │ id │ n = len(labels) │ │ │ ├─────────┴────────────┬─────────────────┤ │ │ │ len(str_1) │ str_1 │ │ │ ├──────────────────────┴─────────────────┤ │ │ │ ... │ │ │ ├───────────────────────┬────────────────┤ │ │ │ len(str_2n) │ str_2n │ │ │ └───────────────────────┴────────────────┘ │ │ . . . │ └────────────────────────────────────────────┘

Sample records

┌──────────────────────────────────────────────────────────────────┐ │ type = 2 │ ├──────────────────────────────────────────────────────────────────┤ │ ┌────────────────────┬───────────────────────────┐ │ │ │ id │ timestamp │ │ │ └────────────────────┴───────────────────────────┘ │ │ ┌────────────────────┬───────────────────────────┬─────────────┐ │ │ │ id_delta │ timestamp_delta │ value │ │ │ └────────────────────┴───────────────────────────┴─────────────┘ │ │ . . . │ └──────────────────────────────────────────────────────────────────┘

See Prometheus WAL.md

off-tray storage

All the data described above is written to memory. The final landing is to pack every two hours of data into a block through the compator routine.

At this point, I believe that everyone has a deeper understanding of "how to insert data into Prometheus time series database," so let's actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.