Shulou(Shulou.com)06/01 Report--

Use background

The company recently launched a new application, the underlying database uses the domestic distributed database-SequoiaDB.

Because the SequoiaDB cluster needs to be integrated into the company's entire monitoring system, it is necessary to collect the SequoiaDB status, performance indicators and other information, and then provide monitoring system use.

SequoiaDB database itself provides a graphical monitoring interface-SAC, but the monitoring items in it are very different from the indicators commonly used by our company in the past. So after consulting the relevant personnel of SequoiaDB, I decided to develop a set of monitoring programs by myself.

Monitoring of SequoiaDB Storage engine

In the SequoiaDB database, there are two large systems, one is the computing layer, for example, we use the MySQL instance, and the other is the distributed storage layer of SequoiaDB, which has the greatest impact on the performance of the whole database.

With regard to MySQL monitoring, the company already has a complete monitoring program, so there is no need for additional development. However, for the underlying distribution of SequoiaDB, it is very necessary to collect the relevant indicators.

SequoiaDB in the monitoring system, in fact, it is relatively complete, but in the way of presentation, it still needs to be polished. All the running information of SequoiaDB underlying distribution can be obtained by snapshot or list command.

I learned from SequoiaDB technicians that SequoiaDB official monitoring tools such as SAC or sdbtop are actually based on snapshot and list commands. You can check the information center of the official website for more method instructions, snapshot method introduction and list method introduction.

2.1 Snapshot description of SequoiaDB

In the SequoiaDB storage engine, if you want to see the health status, you can get the information through snapshots.

SequoiaDB's snapshot command is very simple. If you use the sdb client it provides, you can do this, such as looking at the usage of each table in the entire cluster:

> db.snapshot (SDB_SNAP_COLLECTIONS) {"Name": "foo.bar", "UniqueID": 4294967297, "Details": [{"GroupName": "group1", "Group": [{"ID": 0, "LogicalID": 0, "Sequence": 1, "Indexes": 1, "Status": "Normal" "TotalRecords": 1, "TotalDataPages": 1, "TotalIndexPages": 2, "TotalLobPages": 0, "TotalDataFreeSpace": 65432, "TotalIndexFreeSpace": 65486, "TotalDataRead": 1, "TotalIndexRead": 0, "TotalDataWrite": 1, "TotalIndexWrite": 1, "TotalUpdate": 0, "TotalDelete": 0 "TotalInsert": 1, "TotalSelect": 1, "TotalRead": 1, "TotalWrite": 1, "TotalTbScan": 1, "TotalIxScan": 0, "ResetTimestamp": 2020-05-26-13.42.20.163109 "," NodeName ":" datanode:11820 "}]}

You can understand from the returned results, first of all, SequoiaDB's distributed storage engine, when taking snapshots, it returns the result format as JSON, which is very different from the Oracle or MySQL data we used in the past, and some friends may not be comfortable at the beginning. But when you get used to the flexible structure of JSON, you will open up a new continent.

In the example I show you, I query the snapshot information at the table level of the entire cluster. It allows us to clearly understand the distribution of each table on each group, as well as its corresponding data read, index read this kind of key information of the instantaneous absolute value. Of course, if you look at the information like this directly, it is estimated that everyone will be blind, so in the follow-up tools, SequoiaDB database still needs a lot of efforts.

2.2 SequoiaDB SQL Fast processing

If you are already using the snapshot and list functions provided by the SequoiaDB storage engine, have you also found a problem? the computing power of the api command provided by the sdb client is too weak. For example, I want to associate SDB_SNAP_SESSIONS snapshots (http://doc.sequoiadb.com/cn/sequoiadb-cat_id-1479173713-edition_id-304) and SDB_SNAP_TRANSACTIONS snapshots (http://doc.sequoiadb.com/cn/sequoiadb-cat_id-1479173720-edition_id-304)) to see which transactions are waiting for locks in the current SequoiaDB storage engine. At this time, simply using api will be very painful, because you have to write an associated program by hand. I'm sure most DBA friends will miss the days when they simply used the SQL command.

Through their own continuous efforts (through the official website information center), I finally found an elegant way to solve it, which is the monitoring view of sql syntax (http://doc.sequoiadb.com/cn/sequoiadb-cat_id-1559546719-edition_id-304)).

For example, for the question just raised, you can get the information through this sql command:

> db.exec ("select trans.NodeName as node, session.LastOpType as lastOpType, session.LastOpInfo as lastOpInfo from $SNAPSHOT_TRANS as trans inner join $SNAPSHOT_SESSION as session on trans.RelatedID = session.RelatedID where trans.WaitLock.CSID is not null") {"node": "datanode:11820", "lastOpType": "GETMORE", "lastOpInfo": "ContextID:297, NumToRead:-1"} {"node": "datanode:11820", "lastOpType": "UPDATE", "lastOpInfo": ""}

This simple version of SQL syntax parsing in SequoiaDB storage engine has achieved twice the result with half the effort for daily operation and operation and maintenance monitoring.

2.3 Development language selection

SequoiaDB storage engine, support a variety of development languages to obtain engine monitoring information, including common: Java, PHP, Python, C++, C and so on. When you are developing, you can download the corresponding driver package from the official website of SequoiaDB. During development and compilation, you can add the driver package of SequoiaDB to ClassPath.

Personally, although Java smells good, I chose the REST interface as the way my program interacts with the SequoiaDB engine. The REST interface is not as convenient as the Driver driver, but it is better than breaking away from the requirements of the language and environment. I can call it anywhere, and the result is the same.

Demo program for attracting jade and throwing brick to attract jade

In order to show you, based on the REST interface provided by SequoiaDB, I made a Mini Program that can monitor the data reading and writing of a table in SequoiaDB in real time using Python language, which is a small contribution to the SequoiaDB community.

The source code of the program can be obtained from:

Https://github.com/yuki0703/Demo

Get from the Github project.

The logic of the program is very simple, that is, through the REST interface provided by SequoiaDB, through the monitoring view method in the SQL syntax of SequoiaDB, the snapshot information of a table is obtained, and then the data operation of the table is obtained by calculating the numerical gap of less than 1 second.

The help information of the program is as follows:

SequoiaDB Monitoroptional arguments:-h,-- help show this help message and exit-- host HOST coord host-u USERNAME username-p PASSWORD password-t TABLE table nam

The effect of monitoring a table in SequoiaDB is as follows:

Postscript

According to the arrangement, the interface provided by SequoiaDB is still very rich, but in the visual monitoring interface, we still need a lot of efforts to directly provide access to most of the current monitoring systems in the market, which will be even more perfect. But in any case, it is worth admiring and learning to be able to make a distributed database of our own.

Encourage it together!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.