Shulou(Shulou.com)06/01 Report--

File version:

V1.0

Document number:

R&D0031

Release date:

Make-up:

Examination and approval:

Influxdb relay + grafana server

Catalogue

1. Introduction 3

1.1 Features 4

1.2 influxdb comparison 4

1.3 use 4

two。 System deployment Foundation 4

2.1 installation 4

2.2 Startup 5

2.3 web interface display 5

2.3.1 No authentication web login 5

2.3.2 verify web login 6

2.4 Login terminal with password 7

2.5 Database general operations 8

2.5.1 increase the database by 8

2.5.2 Database deletion and modification 8

2.5.3 Database query 8

2.5.4 Database addition and table operations 9

2.5.5 Database Save Policy (Retention Policies) 10

2.5.6 continuous database queries 10

2.5.7 Database user Management 10

3. Data display tool 11

3.1 grafana introduction 11

3.1 grafana install, run, start 11

3.2 grafana access 11

3.3.Create Dashbord13

4. IfluxDB Relay13

4.1 introduction 13

4.2 Topology introduction 13

4.3 Source package installation Test 14

4.4 influxdb-relay start 15

4.5 influxdb-relay data write synchronization 16

4.6 influxdb-relay node fault data write synchronization 17

5. FAQ18

Revision record

Version number

Release date

Fictitious person

Revision description

V1.0

2017-01-17

Release for the first time

V1.1

2017-02-14

Add account and password

Influxdb server

1. Brief introduction

InfluxDB is an open source distributed time series, event and indicator database written in GE language, which is quite different from the traditional database. Similar databases include Elasticsearch, Graphite, and so on.

1.1 Features

1. API of Http interface is provided to manipulate data.

2. Database statements similar to sql are provided.

3. Based on time series, support time-related functions (such as maximum, minimum, summation, etc.)

4. Measurability: you can calculate large amounts of data in real time

5. Event-based: it supports arbitrary event data

6. Unstructured (modeless): can be any number of columns

7. Expandable support for min, max, sum, count, mean, median and other functions to facilitate statistics

8. Native HTTP support, built-in HTTP API

9. Powerful SQL-like syntax

10. Built-in management interface, easy to use

1.2 influxdb comparison

Comparison of libraries, tables, etc.:

InfluxDB

Concepts in traditional databases

Database

Database

Measurement

Tables in the database

Points

A row of data in a table

The composition of influxdb data:

Point consists of a time stamp (time), data (field), and a label (tags).

Point attribute

Concepts in traditional databases

Time

Each data recording time is the primary index in the database (will be automatically generated)

Fields

Various record values (attributes without indexes) that is, recorded values: temperature, humidity

Tags

Various indexed attributes: region, altitude

Here we have to mention another noun: series:

All the data in the database needs to be displayed through a chart, and this series represents the data in this table, which can be plotted into several lines on the chart: calculated by tags permutation and combination. You can query it through SHOW SERIES FROM "Table name".

1.3 uses

Generally used to store real-time data, with a set of UI interface to display information.

two。 System deployment Foundation 2.1 installation

Basic environment, ready.

Root@ubuntu:~# ls

Grafana_4.1.1-1484211277_amd64.deb influxdb_1.1.1_amd64.deb

Root@ubuntu:~# cat / etc/issue

Ubuntu 14.04.5 LTS\ n\ l

Root@ubuntu:~# uname-a

Linux ubuntu 3.16.0-30-generic # 40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86 "64 GNU/Linux

Start installation

Root@ubuntu:~# dpkg-I influxdb_1.1.1_amd64.deb

Selecting previously unselected package influxdb.

(Reading database... 90134 files and directories currently installed.)

Preparing to unpack influxdb_1.1.1_amd64.deb...

Unpacking influxdb (1.1.1-1).

Setting up influxdb (1.1.1-1).

Adding system startup for / etc/init.d/influxdb...

/ etc/rc0.d/K20influxdb->.. / init.d/influxdb

/ etc/rc1.d/K20influxdb->.. / init.d/influxdb

/ etc/rc6.d/K20influxdb->.. / init.d/influxdb

/ etc/rc2.d/S20influxdb->.. / init.d/influxdb

/ etc/rc3.d/S20influxdb->.. / init.d/influxdb

/ etc/rc4.d/S20influxdb->.. / init.d/influxdb

/ etc/rc5.d/S20influxdb->.. / init.d/influxdb

Processing triggers for man-db (2.6.7.1-1ubuntu1).

2.2 start

Root@ubuntu:~# / etc/init.d/influxdb start

Starting the process influxdb [OK]

Influxdb process was started [OK]

Root@ubuntu:~# netstat-lnp | grep influ

Tcp6 0 0: 8083: * LISTEN 5057/influxd

Tcp6 0 0: 8086: * LISTEN 5057/influxd

Tcp6 0 0: 8088: * LISTEN 5057/influxd

8083 the user name and password of http://ip:8083 on the management side of 8083 web are admin, and the port of Web management interface of InfluxDB is 8083.

8086 HTTP API listening port is 8086

8088 cluster port (it is not clear yet. If the bind-address is configured globally, it will be enabled if it is not configured by default)

After the startup is successful, we can start using influxdb!

2.3 web interface shows 2.3.1 unauthenticated web login

Root@ubuntu:~# which influx

/ usr/bin/influx

Root@ubuntu:~# influx

Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

Connected to http://localhost:8086 version 0.13.0

InfluxDB shell version: 0.13.0

>

> show databases

Name: databases

Name

-

_ internal

> CREATE DATABASE "testDB" # create a database

> show databases # create a database

Name: databases

Name

-

_ internal

TestDB

> use testDB # using the testDB database class mysql

Using database testDB

Access to http://ip:8083 through Web interface

Note: the web display is disabled by default in the new version. For more information, please see the FAQ solution below:

Operate commands through the interface.

2.3.2 verify web login

1. Add users first

Set authorized for TS to improve security. For data that specified users have permissions to access the database, TS default users are divided into ordinary users and administrator users, and permissions are divided into read,write,all privileges three permissions.

Root@ubuntu:/home/leco# influx

Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

Connected to http://localhost:8086 version 1.1.1

InfluxDB shell version: 1.1.1

> CREATE USER "influxdb" WITH PASSWORD 'influxdb' WITH ALL PRIVILEGES

Add the account number and password first, both of which are influxdb

The switch of database in Web interface. At this point, you need to enter your account number and password when you log in to web. After verification, you can enter normally. The configuration is as follows:

two。 Restart influxdb

Root@ubuntu:~# / etc/init.d/influxdb restart

Stopping influxdb...

Influxdb process was stopped [OK]

Starting influxdb...

Influxdb process was started [OK]

Enter the correct IP, port, user name and password on the interface and click save to enter.

2.4 Log in to the terminal with password

If password authentication is set, you must enter the account number and password in order to log in normally, otherwise the error is as follows

1. Error login

Root@ubuntu:/home/leco# influx

Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

Connected to http://localhost:8086 version 1.1.1

InfluxDB shell version: 1.1.1

> show databases

ERR: unable to parse authentication credentials # indicates that authentication method has been added

Warning: It is possible this error is due to not setting a database.

Please set a database with the command "use".

> quit

two。 Log in correctly

Root@ubuntu:/home/leco# influx-host' localhost'-port '8086'-username 'influxdb'-password' influxdb'

Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

Connected to http://localhost:8086 version 1.1.1

InfluxDB shell version: 1.1.1

> show databases

Name: databases

Name

-

_ internal

Cmz

# at this time, localhost is connected to the local machine, and if it is connected remotely, it will be replaced with the corresponding IP. Port is the influxdb default port 8086 and password is the authorized account and password.

2.5 Database General Operations 2.5.1 Database increase

> use testDB

Using database testDB

Insert weather,altitude=1000,area= north temperature=11,humidity=-4

Or add it through the HTTP interface

Curl-I-XPOST 'http://localhost:8086/write?db=testDB'-- data-binary' weather,altitude=1000,area= North temperature=11,humidity=-4'

Line Protocol format

The format of the inserted data seems strange because influxDB stores the data in Line Protocol format.

In the above two methods of inserting data, there is the same part.

Weather,altitude=1000,area= North temperature=11,humidity=-4

Where:

1. Weather: table name

2. Altitude=1000,area= North: tag

3. Temperature=11,humidity=-4: field

Refer to the official documentation:

Https://docs.influxdata.com/influxdb/v0.10/write_protocols/line/

2.5.2 Database deletion and modification 2.5.3 Database query

> use testDB

Using database testDB

> SELECT * FROM weather ORDER BY time

Name: weather

Timealtitudeareahumiditytemperature

-

14850746132673340411000 North-411

Through the Http interface

Curl-G 'http://localhost:8086/query?pretty=true'-data-urlencode "db=testDB"-data-urlencode "q=SELECT * FROM weather ORDER BY time DESC LIMIT 3"

InfluxDB supports SQL-like statements, and the specific query syntax is similar, so it will not be described in detail. See the official documentation for details.

2.5.4 Database addition and table operation

# create a database named db_name

> CREATE DATABASE "db_name"

# display all databases

> SHOW DATABASES

Name: databases

Name

-

_ internal

TestDB

Db_name

# Delete the library whose name is db_name

> DROP DATABASE "db_name"

# switch to testDB database

> USE testDB

Using database testDB

# display the tables in the database

> SHOW MEASUREMENTS

Name: measurements

Name

-

Weather

# create a table

# specify the table name directly when inserting data (weather is the table name)

Insert weather,altitude=1000,area= north temperature=11,humidity=-4

# Delete weather table

> DROP MEASUREMENT "weather"

2.5.5 Database Save Policy (Retention Policies)

InfluxDB does not provide a way to delete Points directly, but it does provide Retention Policies. Mainly used to specify the retention time of data: when the data exceeds the specified time, it will be deleted.

# 1. View the Retention Policies of the current database

> SHOW RETENTION POLICIES ON "testDB"

NamedurationshardGroupDurationreplicaNdefault

-

Autogen0s168h0m0s1true

# 2. Create a new Retention Policies

> CREATE RETENTION POLICY "rp_name" ON "testDB" DURATION 30d REPLICATION 1 DEFAULT

Description:

1. Rp_name: policy name

2. TestDB: specific database name

3. 30d: save for 30 days, and the data before 30 days will be deleted

It has various time parameters, such as h (hour), w (week)

4. REPLICATION 1: the number of copies. Just enter 1 here.

5. DEFAULT is set as the default policy

# 2. Modify Retention Policies

> ALTER RETENTION POLICY "rp_name" ON "testDB" DURATION 3w DEFAULT

# 3. Delete Retention Policies

> DROP RETENTION POLICY "rp_name" ON "testDB"

2.5.6 continuous database query

When the data exceeds the time specified in the save policy, it will be deleted. If we don't want to delete it completely, for example, do a statistical sample of data: save the original data per second as data per hour, so that the space occupied by the data is greatly reduced (at the expense of reduced accuracy). This requires the continuous query (Continuous Queries) provided by InfluxDB.

# 1. Continus Queries of the current database

> SHOW CONTINUOUS QUERIES

Name: _ internal

Namequery

-

Name: testDB

Namequery

-

# 2. Create a new continus Queries

2.5.7 Database user Management

# display users

> SHOW USERS

Useradmin

-

# create a user

> CREATE USER "realcloud" WITH PASSWORD '123456'

# create a user with administrator privileges

> CREATE USER "cmz" WITH PASSWORD 'cmz' WITH ALL PRIVILEGES

# deleting a user

> DROP USER "cmz"

3. Introduction to data display tool 3.1 grafana

In the end, the data needs a set of UI to display, and this kind of real-time data display, there are many projects. For example:

1. Official Chronograf

2. Grafana

3. Others.

3.1 grafana install, run, start

At this point, I chose to use Grafana

Root@ubuntu:~# ls

Grafana_4.1.1-1484211277_amd64.deb influxdb_1.1.1_amd64.deb

# installation

Root@ubuntu:~# dpkg-I grafana_4.1.1-1484211277_amd64.deb

Selecting previously unselected package grafana.

(Reading database... 61968 files and directories currently installed.)

Preparing to unpack grafana_4.1.1-1484211277_amd64.deb...

Unpacking grafana (4.1.1-1484211277).

Setting up grafana (4.1.1-1484211277).

Adding system user `grafana' (UID 106)...

Adding new user `grafana' (UID 106) with group `grafana'...

Not creating home directory `/ usr/share/grafana'.

# NOT starting grafana-server by default on bootup, please execute

Sudo update-rc.d grafana-server defaults 95 10

# In order to start grafana-server, execute

Sudo service grafana-server start

Processing triggers for ureadahead (0.100.0-16).

Ureadahead will be reprofiled on next reboot

# start

Root@ubuntu:~# / etc/init.d/grafana-server start

* Starting Grafana Server [OK]

For installation steps, please see: http://grafana.org/download/

3.2 grafana access

1) Log in to Web and visit http://your_ip:3000 directly.

Default account: admin

Default password: admin

2) add database testDB to Data Sources

If user and password are not set, you can fill in them as you like.

After saving, you can test it through Test Connection to see if it is correct.

3.3.Create Dashbord

Just click the New button.

Introduction to 4.IfluxDB Relay4.1

This project adds a basic high availability layer to influxdb. Highly available settings are achieved through the correct architecture and disaster recovery process.

Hint: influxdb-relay must be built with Go 1.5 +

Reference link:

Https://github.com/influxdata/influxdb-relay/blob/master/README.md

4.2 Topology introduction

The architecture is fairly simple, including a load balancer, two or more influxdb relay processes, and two or more influxdb processes. The load balancer should use path / write to direct UDP traffic and http post requests to two relays and path / query to both influxdb servers. The installer should be as follows:

It gives me the impression that this influxdb-relay is the role of database middleware and acts as an agent.

4.3 Source package installation test

1. Experimental environment

IP

System environment

Install softwar

192.168.5.200

Ubuntu14.04 x64

Influxdb

Influxdb-relay

192.168.5.202

Ubuntu14.04 x64

Influxdb

At this time, only influxdb is installed on the 200 machines, and influxdb should also be installed on the 202s at this time. Can also be installed on only one machine, I installed separately, more in line with the actual production situation.

two。 Install the golang environment

Influxdb release's deb package itself is compiled and does not require a golang environment. Influxdb-relay is currently only stored on github, so you need to install the golang environment first. It is necessary to download a later version of golang manually.

3. Download and extract the 1.6.2 golang package

Root@ubuntu:~# apt-get install git

Root@ubuntu:~# wget http://www.golangtc.com/static/go/1.6.2/go1.6.2.linux-amd64.tar.gz

Root@ubuntu:~# ls go1.6.2.linux-amd64.tar.gz

Go1.6.2.linux-amd64.tar.gz

Root@ubuntu:~# tar-zxf go1.6.2.linux-amd64.tar.gz-C / usr/local/

4. Configure environment variables

Root@ubuntu:~# tail-9 / etc/profile

# add by caimengzhi at 2017-01-23 for go. Start

Export GOROOT=/usr/local/go

Export GOBIN=$GOROOT/bin

Export GOPKG=$GOROOT/pkg/tool/linux_amd64

Export GOARCH=amd64

Export GOOS=linux

Export GOPATH=$HOME

Export PATH=$PATH:$GOBIN:$GOPKG:$GOPATH/bin

# add by caimengzhi at 2017-01-23 for go. End

Root@ubuntu:~# source / etc/profile

Root@ubuntu:~# go version

Go version go1.6.2 linux/amd64

Root@ubuntu:~# GOPATH=/root/go

Root@ubuntu:~# go get github.com/influxdata/influxdb-relay

Root@ubuntu:~# ls

Go go1.6.2.linux-amd64.tar.gz grafana_4.1.1-1484211277_amd64.deb influxdb_1.1.1_amd64.deb src

# the go file will be found under / root

Root@ubuntu:~# cp $GOPATH/src/github.com/influxdata/influxdb-relay/sample.toml. / relay.toml

Root@ubuntu:~# vim relay.toml

-bash: / root/go/bin/influxdb-relay: No such file or directory

Root@ubuntu:~# which influxdb-relay

/ usr/local/go/bin/influxdb-relay

5. Profile interpretation

Root@ubuntu:~# cat relay.toml

`http`

Name = "example-http"

Bind-addr = "192.168.5.200 purl 9096"

Output = [

{name= "local1", location = "http://192.168.5.200:8086/write"}

{name= "local2", location = "http://192.168.5.202:8086/write"}

]

`udp`

Name = "example-udp"

Bind-addr = "192.168.5.200 purl 9096"

Read-buffer = 0 # default

Output = [

{name= "local1", location= "192.168.5.200v 8089", mtu=512}

{name= "local2", location= "192.168.5.202 location= 8089", mtu=1024}

]

4.4 influxdb-relay start

Root@ubuntu:~# / usr/local/go/bin/influxdb-relay-config relay.toml > / tmp/influxdb_relay.log 2 > / dev/null &

[1] 10943

Re-specify port 9096 previously configured by influxdb-relay to successfully enter other influxdb

Root@ubuntu:~# influx-port 9096

Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

Connected to http://localhost:9096 version relay

InfluxDB shell version: 1.1.1

4.5 influxdb-relay data write synchronization

Verify that data can be written through influxdb-relay

Root@ubuntu:~#

Curl-I-XPOST 'http://localhost:9096/write?db=testDB'-- data-binary' cpu_load_short,host=server01,region=us-west value=0.64 14340555620000000'

17:53:43 on 2017-01-23 Problem posting to relay "example-http" backend "local2": Post http://127.0.0.1:7086/write?db=testDB: dial tcp 127.0.0.1 purl 7086: getsockopt: connection refused

HTTP/1.1 204 No Content

Date: Mon, 23 Jan 2017 09:53:43 GMT

1. Enter node 1 to verify data writing

Root@ubuntu:~# influx

Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

Connected to http://localhost:8086 version 1.1.1

InfluxDB shell version: 1.1.1

> use testDB

Using database testDB

> select * from cpu_load_short

Name: cpu_load_short

Timehostregionvalue

-

1434055562000000000server01us-west0.64

two。 Enter node 2 to verify data writing

> use testDB

Using database testDB

> select * from cpu_load_short

Name: cpu_load_short

Timehostregionvalue

-

1434055562000000000server01us-west0.64

From the above, we can see that by calling port 9096 of relay. Write data, both nodes will have relative data to write.

Note:

# defect. I don't know if it's a pit. Anyway, I didn't see this notice on the github of the official website.

# manually create databases on all nodes

The write operation does not involve creating the database, so you need to manually create the database in advance on all nodes

4.6 write synchronization of influxdb-relay node fault data

# Operation of failed nodes

1. Stop the influxdb of Node 1

Root@loocha15:~# / etc/init.d/influxdb stop

Stopping influxdb...

Influxdb process was stopped [OK]

two。 Continue to write data at this time

Curl-I-XPOST 'http://localhost:9096/write?db=cmz'-- data-binary' cpu_load_short,host=beijing,region=east value=2 2222222222

15:19:26 on 2017-02-06 Problem posting to relay "example-http" backend "local1": Post http://192.168.3.35:8086/write?db=cmz: dial tcp 192.168.3.35 purl 8086: getsockopt: connection refused

HTTP/1.1 204 No Content

Date: Mon, 06 Feb 2017 07:19:26 GMT

3. Node 2 validates data

> use cmz

Using database cmz

> select * from cpu_load_short

Name: cpu_load_short

Timehostregionvalue

-

1111111111nanjingeast123

2222222222beijingeast2 # is the data written after Node 1 failure

1434055562000000000server01us-west0.64

Up to this point, the HA of influxdb has been tested.

5. FAQ

1. WEB display 404

Root@ubuntu:~# curl localhost:8086-I

HTTP/1.1 404 Not Found

Content-Type: text/plain; charset=utf-8

X-Content-Type-Options: nosniff

X-Influxdb-Version: 1.1.1

Date: Sun, 22 Jan 2017 09:36:48 GMT

Content-Length: 19

The new version of Influxdb web interface does not show solutions.

1. The newly installed influxdb web interface is not displayed

During the visit, 404.

Solution:

First, the cause of the problem

After InfluxDB version 0.13, the web management page is closed by default, while most of the domestic documents are written with the old version of InfluxDB as the standard, so after downloading and installing the latest version, there will be a problem that the port 8083 web management page cannot be accessed.

II. Solutions

Although the new version of InfluxDB closes the web management page by default, we can open it in a very simple way.

Open the configuration file and vim / etc/influxdb/influxdb.conf finds the following lines:

After modification, it is as follows:

Restart the service

Root@ubuntu:~# / etc/init.d/influxdb restart

Influxdb process already stopped [OK]

Starting influxdb...

Influxdb process was started [OK]

Check the port number 8083 (there was no one before)

Root@ubuntu:~# netstat-lnp | grep influ

Tcp6 0 0: 8083: * LISTEN 5057/influxd

Tcp6 0 0: 8086: * LISTEN 5057/influxd

Tcp6 0 0: 8088: * LISTEN 5057/influxd

Web interface access display:

Http://ip:8083

OK.

2. Command

Root@ubuntu:~# go get-u github.com/influxdata/influxdb-relay

Go: missing Git command. See https://golang.org/s/gogetcmd

Package github.com/influxdata/influxdb-relay: exec: "git": executable file not found in $PATH

Resolve:

Root@ubuntu:~# apt-get install git

The 3.relay command line cannot manage

Root@ubuntu:~#! net

Netstat-lnp | grep 9096

Tcp 0 0127.0.0.1 9096 0.0.0.0 * LISTEN 3502/influxdb-relay

Udp 0 0127.0.0.1 9096 0.0.0.0 * 3502/influxdb-relay

Root@ubuntu:~# influx-port 9096

Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

Connected to http://localhost:9096 version relay

InfluxDB shell version: 1.1.1

> show database

ERR: invalid write endpoint

Warning: It is possible this error is due to not setting a database.

Please set a database with the command "use".

6. Reference article

Http://dbspace.blog.51cto.com/6873717/1881705

Http://grafana.org/features/

Https://www.influxdata.com/

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.