Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to use Grafana Tempo for distributed tracking". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let Xiaobian take you to learn "How to use Grafana Tempo for distributed tracking"!

Grafana Tempo is a new open source, high-volume distributed tracing backend.

Grafana's Tempo is an easy-to-use, large-scale, distributed tracking backend from Grafana Labs. Tempo integrates Grafana, Prometheus, and Loki, and it requires only object storage to operate, so it is inexpensive and easy to operate.

I've been involved with this open source project since the beginning, so I'll cover some basics about Tempo and explain why the cloud native community noticed it.

distributed tracing

It is common to want to collect telemetry data for application requests. But in today's servers, a single application is often split into multiple microservices, possibly running on several different nodes.

Distributed tracing is a way to obtain fine-grained information about the performance of an application, which may consist of discrete services. When a request arrives at an application, it provides a unified view of the lifecycle of the request. Tempo's distributed trace, which can be used for monolithic or microservice applications, provides information on the scope of requests, making it the third pillar of observability (the other two being metrics and logging).

Next is an example of a distributed tracing system generating an application Gantt chart. It uses Jaeger HotROD's demo app to generate traces and save them to Grafana Cloud hosted Tempo. This graph shows request processing times by service and function.

Gantt chart from Grafana Tempo

Reduce index size

In a rich and well-defined data model, traces contain a lot of information. In general, there are two types of interactions with the trace backend: filtering traces using metadata selectors (such as service name or duration), and filtering visual traces.

To enhance search, most open source distributed tracing frameworks index many fields in the trace, including service name, operation name, tag, and duration. This leads to large indexes and forces you to use databases like Elasticsearch or Cassandra. But these are hard to manage and costly to operate at scale, so my team at Grafana Labs started coming up with a better solution.

In Grafana, our on-call debugging workflow starts with using metrics reports (we use Cortex to store metrics in our app, which is a cloud-native foundation incubated project to extend Prometheus), delves deeper into the problem, filters logs for problematic services (we store logs in Loki, which is like Prometheus, except Loki keeps logs), and then looks at tracking a given request. We realized that all the indexing information we needed for filtering could be found in Cortex and Loki. However, we need a strong integration for trace discoverability with these tools and a nice store for key lookups based on trace IDs.

This was the beginning of Grafana Tempo. By focusing on tracing for a given retrieval trace ID, we designed Tempo as a minimally dependent, high-capacity, low-cost distributed tracing backend.

Easy to operate, cost-effective

Tempo uses an object storage backend, which is its only dependency. It can be used either in a single binary or in microservices patterns (see the examples in the repository for an easy introduction). Using object storage also means that you can store a large number of traces of the application without any sampling. This ensures that you never drop traces of the one in a million requests that are faulty or have higher latencies.

Powerful integration with open source tools

Grafana 7.3 includes the Tempo data source, which means you can visualize traces from Tempo in the Grafana UI. Also, Loki 2.0's new query feature makes tracking in Tempo easier. To integrate with Prometheus, the team is adding support for exemplar, which is high-cardinality metadata information that can be added to time-series data. The metrics store backend doesn't index them, but you can retrieve and display metrics in the Grafana UI. Although the example can store various metadata, in this case the trace ID is stored for tight integration with Tempo.

This example shows an example using a histogram with request delays, where each example data point is linked to a trace in Tempo.

Using exemplars in Tempo

metadata consistency

Telemetry data emitted by applications running as containerized applications typically has some associated metadata. This can include cluster IDs, namespaces, pod IPs, etc. This is good for providing information based on requirements, but even better if you can use the information contained in metadata for productive things. For example, you can import trace information into Tempo using the Grafana cloud broker, which polls the Kubernetes API for metadata information using Prometheus service discovery mechanisms and adds these tags to the cross-domain data emitted by the application. Since this metadata is also indexed in Loki, it is easy to jump from tracing to viewing logs for a given service by converting metadata to Loki tag selectors.

The following is an example of consistent metadata that can be used to view a given range of logs in a Tempo trace.

cloud-native

Grafana Tempo works as a containerized app, and you can run it on orchestration engines like Kubernetes, Mesos, etc. Depending on the workload on the fetch/query path, various services can scale horizontally. You can also use cloud-native object storage, such as Google Cloud Storage, Amazon S3, or Tempo Azure Blog Storage. For more information, read the Architecture section of the Tempo documentation.

At this point, I believe everyone has a deeper understanding of "how to use Grafana Tempo for distributed tracking". Let's do it in practice! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.