Shulou(Shulou.com)06/01 Report--

The original author of this article, Bao Guangya, the software development engineer of JD.com Mall basic platform Department, published on my blog with the consent of the author, if you need to reprint it, you need my consent.

I. Preface

Our department uses the open source Zabbix system to monitor the database, which currently monitors tens of thousands of hosts. By analyzing the data structure of the server side of the Zabbix system and the implementation method of parallel computing, this paper tries to explore the potential scalability of the server side of the Zabbix system, and hopes to help to further optimize the running efficiency and stability in the practical application process.

Zabbix system adopts server-proxy-agent architecture, and the main function of server is to collect monitoring data and trigger alarm action based on the collected data. In practical applications, zabbix may monitor 10000 hosts (host, uniquely identified by hostid). If each host sets 50 monitoring indicators (item, uniquely identified by itemid) and collects data every minute, there are a total of 500000 item, and 8333 items of data (value) need to be received and processed per second, that is, vps (values per second) is 8333. If 1/3 of the item has an alarm trigger (trigger, uniquely identified by the triggerid), there are 170000 trigger.

In the above scenarios, in order to ensure the effectiveness and timeliness of monitoring, zabbix needs to find the correct value among 500000 item immediately after receiving each item, and obtain the previous value of the item (previous value,last (), in order to calculate the increment), or calculate the average value in the first 5 minutes (avg (5m)), so as to determine whether the alarm event should be triggered according to the trigger expression (trigger expression, uniquely identified by functionid). Uniquely identified by eventid). At the same time, if the item return type is numeric, you also need to calculate the average (value_avg), maximum (value_max), and minimum (value_min) of the item within an hour. According to the above vps data, zabbix needs to search at least 8333 million 500, 000 times per second. In addition, item and trigger are not static data, and users may add, modify, delete, disable (disable) and enable (enable) some item and trigger,zabbix needs to query the latest status of the item and trigger when processing value. How to accomplish such a large number of operations in a second, zabbix's solution is: hash table.

In the aspect of parallel computing software, because the hosts monitored by Zabbix system are relatively independent, it is very easy to parallelize computing both in terms of tasks and data. In terms of server hardware, the server structure we actually use is 2x8 processor + three-level cache + 16G*8 memory + SSD hard disk + 10Gbps network card (shared by database, zabbix server and web services). The parallel computing of Zabbix mainly adopts the shared memory mode.

2. Types of hash tables in Zabbix

The hash tables used by Zabbix are chained hash tables, which can be divided into the following five categories (all allocate space in shared memory):

1. Valuecache

Valuecache contains two hash tables vc_cache- > items (itemid as key) and vc_cache- > strpool (string as key), which are used to store collected values (including numeric and string types). Each item occupies one slot, each slot is a linked list, and the linked list node stores the actual information needed.

The hash table of Valuecache is created when the service starts and is destroyed when the service exits. The initial number of slots is 1009 (the first prime number after 1000). With the increase of the number of elements in the table, the number of slots will increase according to certain rules. The maximum space that can be used by Valuecache is controlled by the ValueCacheSize parameter in the configuration file, and the allowed range is 128K-64G.

2. Dbcache

The cache- > trends hash table in Dbcache, which is used to cache data from the trends table (hourly average, maximum, minimum value of each item). Zabbix server's history_syncer process continuously receives data from agent or proxy, loads it into the cache, and updates the cache- > trends hash table. The elements in the hash table are ZBX_DC_TREND structures.

Data in the Cache- > trends table is flush into the database when the time exceeds the hour, for example, data between 9 and 10:00 is flush into the database after 10:00.

The Cache- > trends hash table is created when the service starts, and the initial number of slots is the same as vc_cache- > items, which is 1009 (the first prime after 1000). The maximum free space of the Cache- > trends hash table is controlled by the TrendCacheSize parameter in the configuration file, and the allowed range is 128K-2G.

3. Dbconfig

The Dbconfig cache stores multiple hash tables of configuration information related to monitoring, including config- > hosts, config- > items, config- > functions, config- > triggers, and so on. The key values of the hash table of configuration information include hostid, itemid, functionid, triggerid, triggerdepid, expressionid, globalmacroid, hostmacroid, hosttemplateid, interfaceid, host_inventory, etc., of which the largest number is often itemid, functionid and triggerid, at the level of hundreds of thousands (in terms of 10000 host).

Taking config- > items as an example, the element of the hash table is the ZBX_DC_ITEM structure. The data in Config- > items is queried from the database, and zabbix server's configuration syncer process periodically synchronizes the data from the database to the cache.

Other hash tables in the Dbconfig cache, similar to the config- > items table, synchronize data from the database, are created at the start of the service, have an initial slot of 1009, and dynamically expand as the amount of data increases. The amount of free space in the entire dbconfig cache is determined by the CacheSize parameter, and the range of values is 128K-8G.

4. Strpool

Strpool and vc_cache- > strpool here are two independent hash tables. This Strpool cache is used to store string values related to configuration information, and it shares CacheSize space with dbconfig (strpool accounts for 15%). The strings stored by Strpool include host name, item key, item delay_flex, snmp community, snmp securityname, snmp passphrase, logitem format, and so on. When Zabbix needs to use a string such as host name, it will first look for it in strpool.

The initial number of slots in the hash table of Strpool is 1009. The key value is the string itself, and the hash value is the return value of calling the hash function on the string.

5. Other

In addition to the above hash table, there are snmpidx, vmware service and other hash tables.

Third, the realization of hash table

Take the config- > items hash table as an example to illustrate the implementation of the hash table in zabbix.

1. Data structure definition

Zabbix uses a linked hash table, and each slot in the hash table is a linked list. The specific data structure is defined as follows:

two。

The value of slot number and load factor

The hashing process of Zabbix is to call the hash function to calculate the hash value corresponding to the key value, and then use the remainder method to determine the slot number. Therefore, the divisor of the remainder calculation is the slot number, which takes the prime number (because the prime number can be hashed uniformly to the maximum extent). In the config- > items hash table, the initial value of the number of slots is 1009. With the increase of the amount of data, when the load factor (number of elements / number of slots) reaches 0.8, the number of slots will be expanded (expanded to more than 1.5 times the current number, and the prime number will be taken). Therefore, the load factor is always kept between 0.8 and 0.533.

According to the above rules, the number of slots in each extended hash table is shown below. When the number of item is 500000, the number of slots should be 670849.

Serial number

Theoretical value

Prime number (slot number)

Number of elements allowed

0

one thousand

1009

eight hundred and six

one

1513

1523

1217

two

2284

2287

1828

three

3430

3433

2745

four

5149

5153

4121

five

7729

7741

6191

six

11611

11617

9292

seven

17425

17431

13943

eight

26146

26153

20921

nine

39229

39229

31382

ten

58843

58889

47110

eleven

88333

88337

70668

twelve

132505

132511

106007

thirteen

198766

198769

159014

fourteen

298153

298153

238521

fifteen

447229

447233

357785

sixteen

670849

670849

536678

seventeen

1006273

1006279

805022

eighteen

1509418

1509427

1207540

nineteen

2264140

2264149

1811318

3. Hash function

The hash function used by Zabbix is a slight improvement over the fnv-1a function (http://www.isthe.com/chongo/tech/comp/fnv/index.html). This function uses product and bit operation to achieve the purpose of fast hashing. The specific implementation is as follows:

According to the above function, the hashing process of 620000 itemid is simulated (the number of slots is 1006279), and the hashing efficiency is as follows:

Total barrel number

1006279

Number of empty buckets

543405

The number of barrels with a depth greater than 1

127940

Load factor

0.616131311

Maximum barrel depth

seven

Proportion of deep barrels to value buckets

0.276403514

Proportion of deep buckets to total buckets

0.127141677

Proportion of empty buckets to the total

0.540014251

IV. Parallelization of tasks and data

1. Parallelism of tasks

The tasks of the Zabbix system are basically based on the monitored host and item, and there is a strong independence between each host and item. For parallelization, Zabbix splits tasks into relatively independent subtasks, each of which is executed by one or more processes. The process partition on the Zabbix server side is shown in the following table:

Start

Sequence

Process title

Allow

Number of processes

Default value

Task

one

Configuration syncer

1-1

one

Synchronize data from the database to the Dbconfig cache

two

Db watchdog

1-1

one

Periodically check whether the server database is available, and send an alarm message if it is not available

three

Poller # n

0-1000

five

According to the data in dbconfig, collect data from passive agent and snmp devices, and flush to shared memory cache- > history

four

Unreachable poller # n

0-1000

one

Periodically polling a device when it is in the unreachable state

five

Trapper # n

0-1000

five

Receive and process data from active agent and active proxy (json format, zabbix communication protocol) from socket, and flush to shared memory cache- > history

six

Icmp pinger # n

0-1000

one

According to the data in dbconfig, collect icmpping-related item data in batches and flush them to shared memory cache- > history

seven

Alerter

1-1

one

Send all kinds of alarm notifications

eight

Housekeeper

1-1

one

Delete expired historical data periodically

nine

Timer # n

1-1000

one

Evaluate time-related trigger expressions, etc.

ten

Node watcher

1-1

one

Dealing with interaction with node

eleven

Http poller # n

0-1000

one

Collect data related to web monitoring and flush to shared memory cache- > history

twelve

Discoverer # n

0-250

one

Scan the network according to the specified rules and automatically discover host, interface, etc.

thirteen

History syncer # n

1-100

four

Batch update the data in shared memory cache- > history to the database, and flush to shared memory vc_cache, cache- > trends, config- > items, etc.

fourteen

Escalator

1-1

one

When the alarm operation needs to be carried out step by step, control the escalations between the steps.

fifteen

Ipmi poller # n

0-1000

0

Similar to the poller process, dealing with ipmi items

sixteen

Java poller # n

0-1000

0

Similar to the poller process, dealing with JMX items

seventeen

Snmp trapper # n

0-1

0

Similar to the trapper process, dealing with snmp items

eighteen

Proxy poller # n

0-250

one

Interact with passive proxy to get the required json format data at a set frequency and flush the data into shared memory cache- > history

nineteen

Self-monitoring

1-1

one

Process item information related to the running status of zabbix itself, and access collector variables in shared memory

twenty

Vmware collector # n

0-250

0

Collect data related to vmware virtual machine and flush it to shared memory

There are two types of critical processes among all processes: poller/trapper processes, which are used to collect data and load them into shared memory, and history syncer processes, which are used to update the database and trigger events and alarm. Logically, these two kinds of tasks are carried out successively, and the data must be collected first before the alarm can be triggered. The processes of each type of task are independent, multiple poller/trapper processes can be executed at the same time, and multiple history syncer processes can also be executed at the same time.

2. Socket multiplexing's support for multiple processes

The final data source of the Zabbix monitoring system is the monitored host, and the data is received through the socket listening port (the maximum number of connections allowed by the listening port is determined by the operating system). Zabbix shares the same socket by fork multiple child processes, and when reading socket, multiple processes read at the same time through multiplexing based on select () function.

Per 10000 host, data is collected every minute (assuming that all item on each host collects data at the same time, which may not be the case), there are an average of 167connection requests per second.

3. Read and write of Mysql database

Zabbix supports a variety of databases, including Mysql, Oracle, IBM DB2, PostgreSQL, SQLite, and we are actually using Mysql. In order to ensure the persistence of the data, zabbix inserts the data into the database before triggering the alarm. The maximum number of History syncer processes is 100. each process can establish an independent connection to the database and update the data.

Shared memory and inter-process communication

1. Creation of shared memory

Shared memory is the simplest and fastest mechanism in inter-process communication. The interprocess communication of Zabbix mainly adopts the way of shared memory. The main process calls shmget to create shared memory before fork all child processes, and attach to the address space.

Zabbix calls shmget to create a total of 8 shared memory segment, namely config_mem, trend_mem, history_mem, history_text_mem, vc_mem, vmware_mem, strpool.mem_info, collector, which are used for dbconfig cache, cache- > trends data, cache- > history (numbers and string), vc_cache, vmware data, strpool, collector structure for monitoring the status of zabbix itself. If vmware is not enabled in the actual application, only seven shared memory will be attach to the address space of each child process. As shown in the following figure, these shared memory segments will remain in the attach state until the service stops.

As you can see from the figure above, each shared memory segment is attach to 553 processes, that is, each process in zabbix server can access all seven shared memory.

two。 Semaphore mechanism

Zabbix uses binary semaphore mechanism to coordinate multiple processes' simultaneous access to shared memory to avoid resource contention. The system calls the semget function before creating shared memory, creates a semaphore set of 13 semaphores, and initializes the value of each semaphore to 1. Each semaphore is used to control access to different shared memory, as shown below:

# define ZBX_MUTEX_LOG 0

# define ZBX_MUTEX_NODE_SYNC 1

# define ZBX_MUTEX_CACHE 2

# define ZBX_MUTEX_TRENDS 3

# define ZBX_MUTEX_CACHE_IDS 4

# define ZBX_MUTEX_CONFIG 5

# define ZBX_MUTEX_SELFMON 6

# define ZBX_MUTEX_CPUSTATS 7

# define ZBX_MUTEX_DISKSTATS 8

# define ZBX_MUTEX_ITSERVICES 9

# define ZBX_MUTEX_VALUECACHE 10

# define ZBX_MUTEX_VMWARE 11

# define ZBX_MUTEX_SQLITE3 12

When a process needs to write to a shared memory, it will first lock (call the semop function to semaphore-1), and then unlock (semaphore + 1) after the write operation is completed. If the semaphore is 0 when lock is executed, wait until the semaphore is non-0. The semaphore of Zabbix is destroyed when the shared memory is freed.

VI. Statements and conclusions

This article is based on the zabbix 2.2.10 version of the source code analysis, welcome criticism and correction.

The hash function used by Zabbix works well. However, in practical application, the load factor, the expansion speed of slot number, the initial value of slot number and the definition of hash function can still be improved according to the needs and resources.

In the aspect of parallel computing in Zabbix, due to the characteristics of the monitoring system, there is a strong independence between data and tasks, which is very convenient for parallelization. Zabbix implements parallelism through multi-processes + shared memory, and resource contention is controlled by semaphores. From the perspective of practical application, the performance of parallelism is very ideal.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.