Shulou(Shulou.com)06/01 Report--

Editor to share with you how OpenTsdb query or read data, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to understand it!

   OpenTSDB provides many ways to extract, process, and analyze data. The data can be queried through the CLI tool, HTTP API, and displayed as a GnuPlot graphic output. Open source tools such as Grafana and Bosun can access TSDB's data. Querying using OpenTSDB's tag-based system can be a bit tricky, so read this document carefully and check the following pages for more in-depth information. The sample query on this page follows the HTTP API format.

Query component

   OpenTSDB provides several tools and interfaces that allow for a variety of query definitions over time. The original syntax supports simple filtering, aggregation, and downsampling. Later versions have added support for functions and expressions. In general, each query has the following components:

The parameter data type Required describes the start time of the sample Start TimeString or IntegerRequired query. Can be absolute time or relative time 24h-agoEnd TimeString or the end time of an IntegerOptional query. If no end time is provided, the current time is the full name of the metric in the end time 1h-agoMetricStringRequired system. Must be a full name and case-sensitive mathematical function used by sys.cpu.userAggregation FunctionStringRequired to combine multiple time series (that is, how to merge time series values in a group) sumFilterStringOptional filters tag values to reduce the number of time series selected in the query or group, and aggregates the optional intervals and functions of each label host=*,dc=laxDownsamplerStringOptional Used to reduce the number of data points returned over time 1h-avgRateStringOptional is used to calculate the rate of change per second rateFunctionsStringOptional data processing functions, such as additional filtering, time switching, etc. HighestMax (…) ExpressionsStringOptional data processing functions, such as dividing one sequence into another (M2 / (M1 + m2)) * 100 time

   supports human-readable absolute timestamps or Unix-style integer formats. Relative time is usually used to refresh the dashboard. Currently, all queries can cover a single time period. In the future, we hope to provide an offset query parameter that can aggregate or chart metrics over different periods of time, such as a comparison from last week to a year ago.

   although OpenTSDB can store data at millisecond resolution (precision), most queries return data at second resolution to provide backward compatibility with existing tools. Unless you use a query that specifies a downsampling algorithm, the data is automatically downsampled to 1 second using the same aggregate function specified in the query. In this way, if multiple data points are stored in a given number of seconds, they will be aggregated and correctly return a normal query.

For    to extract data at millisecond resolution, use the / api/query interface and specify the msResolution (ms is also possible, but not recommended) JSON parameter or query string identification, which bypasses sampling (unless specified) and returns all timestamps at Unix epoch millisecond resolution. In addition, the scan command line tool returns the timestamp written to the store.

Filter

   each time series consists of an indicator and one or more label name / value pairs. In OpenTSDB, filters are applied to tag values (the current TSDB does not provide filtering for metrics or tag names). Because the filter is optional in the query, if you only request the metric name, any numeric or tag value will be returned in the aggregate result. A filter is similar to the Where clause in a SQL statement. For example, we store the following dataset:

Sys.cpu.user host=webserver01,cpu=0 1356998400 1

Sys.cpu.user host=webserver01,cpu=1 1356998400 4

Sys.cpu.user host=webserver02,cpu=0 1356998400 2

Sys.cpu.user host=webserver02,cpu=1 1356998400 1

   makes a simple query with at least a start time, aggregator, and metrics, such as:

Start=1356998400&m=sum:sys.cpu.user

   we get one that aggregates four time series into a group at 1356998400 points in time, with a value of 8.

   if we want to scale a specific sequence or a series of sequences, we can use a filter. For example, we can filter on the host tag:

Start=1356998400&m=sum:sys.cpu.user {host=webserver01}

   the query will return a value of 5 that contains only the time series host=webserver01. To drill down to a specific time series, you must include all the tags of the series, such as queries:

Start=1356998400&m=sum:sys.cpu.user {host=webserver01,cpu=0}

   will return 1.

Polymerization

One of the powerful features of    OpenTSDB is the ability to integrate the instant aggregation of multiple time series into a set of data points. Raw data is always available for storage, but we can extract it quickly in a meaningful way. An aggregate function is a method that combines two or more data points of a single timestamp into a single value.

Note:

   OpenTSDB aggregates data by default and requires an aggregation operator for each query. Each aggregator must handle the loss of multiple sequences or data points in different timestamps. This is done by interpolation, and if the user does not know what TSDB is doing, it may result in unexpected results when querying.

Downsampling

   OpenTSDB can ingest large amounts of data, even if only one data point per second is extracted from a given time series. Therefore, the query may return a large number of data points, and accessing the query results of a large number of points from API may consume a lot of bandwidth. High-frequency data can easily overwhelm the Javascript graphics library, so you can choose to use GnuPlot. The graphics created by GUI are difficult to read, resulting in dense broken lines, as shown in the following figure:

Downsampling can be used when    queries to reduce the number of data points returned so that you can extract better information from the chart or pass less data over connections. Downsampling requires an aggregate function and a time interval. Aggregate functions are used to calculate new data points on all data points in a specified interval through appropriate mathematical functions. For example, if you use sum aggregation, all data points in the interval will accumulate together into a single value. If avg is selected, the average of all data points in the interval is returned.

Using downsampling, we can clean up the previous figure to get something more useful:

Rate

   many data sources return values in the form of incremented counters. One example is a website click counter. When you start the Web server, its counter may be 0. After five minutes, the value may be 1024. It could be 2048 in five minutes. The graph of the counter is a straight line and tilts to the right, which is not always very useful. OpenTSDB provides a rate conversion function that calculates the rate of change of values over time. This converts the counter to a broken line (curve) with spikes so that it can be shown to you and more useful when the activity occurs.

The ratio is the first derivative of these values. It is defined as (v2-v1) / (T2-T1), and the time is in seconds. So you get the rate of change per second. Currently, the rate of change between millisecond values is calculated per second by default.

   OpenTSDB 2.0 supports special monotonous increment counter data processing, including the ability to set "flip" (out of bounds) values and suppress abnormal fluctuations. When you specify a counterMax value in a query, if the data point is close to the value and the subsequent point is less than the previous value, the maximum value is used to calculate the accuracy of the given two points. For example, if we record an integer counter with 2 bytes, the maximum value will be 65535. If the value is 64000 at t0 and 1000 at T1, the rate obtained per second will be calculated as-63000. However, we know that the counter may flip, so we can set the maximum value to 65535, and now calculating 65535-t0 + T1 will return us 2535.

The system on which    tracks the data in the counter usually returns to 0 on reboot. When this happens, if we use the maximum counter function, we may get a false result. For example, if the counter reaches 2000 at t0 and someone restarts the server, the next value at T1 may be 500. If we set the maximum value of 65535, then the result is 65535-2000 + 500returns 64035 to us. If the normal speed is a few points per second, this particular peak, between the data points in 30s, will create a rate spike of 2134.5! To avoid this, we can set resetValue to return a data point with a value of 0 when the rate exceeds this value to avoid spikes in either direction. For the above example, if we know that the speed almost never exceeds 100, we can configure resetValue to 100, and when the above data point is calculated, it will return 0 instead of 2134.5. The default value of 0 means that the reset value will be ignored and rates will not be suppressed. (does not affect rates)

Operation sequence

It is important for    to understand the order of operations. When the query results are returned, the following is the order in which they are processed:

1. Filter

1. Grouping

1. Downsampling

1. Interpolation.

1. Polymerization

   1. Rate conversion

1. Function

1. Expression.

The above is all the contents of the article "how to query or read data from OpenTsdb". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.