Shulou(Shulou.com)06/01 Report--

How to analyze the EOS source code and resource management, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

0 preface

1 what resources are included in EOS resource management

The EOS resource management module mainly manages the following resources:

Bandwidth and disk (Bandwidth and Log Storage (Disk))

Computing resources (Computation and Computational Backlog (CPU))

Memory resources State Storage (RAM)

Transaction data needs to consume bandwidth resources when propagating in the blockchain network and disk resources when each node carries out persistent storage; transactions will consume CPU resources during execution, and subsequent nodes will also consume CPU resources when they need replay blocks; excessive growth in the total amount of transactions and excessive speed (frequency) of transactions are a kind of overhead EOS state database (state db) stores historical data frequently accessed by the application layer, such as transactions, account balances, etc., which consume memory resources and disk resources.

2 Why is it necessary to manage resources

The resources in the EOS network are always limited, and the transaction processing is resource-sensitive. In order to avoid the abuse of resources, we must limit and manage the rights to use the resources to ensure the normal operation of the network.

3 the relationship between Token and resource management

The introduction of Token (pass) of EOS has the following functions:

With the implementation of resource management, the more Token mortgaged, the more resources can be used

Reward BP node (additional issuance)

Initial financing means of entrepreneurial team

Business system value flow (for example, an operation in an intelligent contract that needs to be transferred from the originator to the receiver can be realized by transferring its owned Token)

People who need to use EOS network resources need to mortgage their EOS Token, such as buying RAM, mortgaging Token in exchange for CPU resources and bandwidth resources and so on.

Related APIs and implementation in the eosio.system smart contract, if the smart contract is not loaded and the relevant APIs are called, there will be no restrictions on the use of RAM, CPU and NET of the account, but the trading and block restrictions will still take effect.

4 Resource management mechanism

4.1 Overview

Use the figure above to analyze the EOS resource management mechanism (from top to bottom), that is, how to implement it in the source code:

Resources include RAM memory resources, measured in bytes; CPU computing resources, measured in us microseconds (that is, how long it takes); NET bandwidth resources, measured in bytes; and DISK disk resources implied in the aforementioned section and not reprocessed separately

In the specific implementation, it is divided into three parts: single account limit, single transaction limit and single block limit.

Manage through the comparison of usage and limit; the use of RAM is only cumulative, that is, as long as the account (the account that signs and authorizes each Action in the transaction) RAM still has usage surplus, there is no limit on the RAM usage of a single transaction; the use of CPU and NET is divided into two parts: single and cumulative, that is, it cannot exceed the single limit nor the cumulative limit.

Some basic limit data, such as maximum CPU limit 150ms for a single transaction, maximum bandwidth limit 1m for a single block, etc., are determined by writing dead source code constants, or config.ini configuration files, or set interface calls, or passing parameters during each transaction

Some of the ever-changing limit data, such as account mortgage tokens, network-wide token mortgages, and all resource usage, are stored in the state database (the state database is implemented based on memory file mapping mechanism) It mainly includes four tables, the first is the account usage scale usage_index, the second is the account restriction table limit_index, the third is the block usage table state_index, and the fourth is the block configuration table config_index (the name of the table has been abbreviated)

There are many error codes in resource usage exceptions, including the error codes shown in the figure. The grey list account mechanism needs to be explained, and the limit is slightly different.

When the transaction is initialized, the resource usage is calculated and checked early, and if it exceeds the standard, it is directly refused to execute; the account limit, transaction limit and block limit are checked when the transaction is completed; the cumulative resource usage is updated when the block is packaged, and the limit is updated.

4.2 Virtual resource restrictions

Virtual resource limit (virtual resource limits) is related to the cumulative usage of CPU&NET resources, and is introduced to make better use of resources, because resource usage varies from high to low in different periods of time. After the introduction of virtual resource restrictions, when the current temporary resource usage is low, the virtual resource limit will be expanded, and relatively more resources can be used in the next time period, and vice versa.

As shown in the above figure, the block CPU is the same as the NET restriction structure definition:

Target: expected resource usage for a single block

Max: resource usage limits for individual chunks

Periods: cumulative usage calculation time period, for example, block usage calculation period is 1 hour

Max_multiplier: maximum expansion factor of resource usage limit in idle state

Contract: reduction factor, step size of virtual resource limit reduction

Expand: magnification factor, virtual resources limit the step size of magnification.

When each block is packaged, the total amount of all transactions it contains is calculated. If the resource usage exceeds the expected value target, the contract coefficient is used to reduce the virtual_resource_limits, and if it is lower than the expected value target, the expand coefficient is used to expand the virtual_resource_limits.

The account resource usage limit takes effect only if the token is mortgaged, and the limit value is calculated by combining the block limit value and the token mortgage ratio.

4.3 cumulative usage

Cumulative usage does not have this concept for CPU and NET,RAM.

As shown in the figure above, cumulative usage includes three concepts:

Last_ordinal: the time of the transaction is determined by the time in the block header (block number) in the source code

Value_ex: average resource usage in the current computing cycle periods

Consumed: the average of the past calculation cycle plus the total usage of the current cycle.

It should be noted that when implemented in the source code, the period is a smooth transition rather than strict segmentation, that is, assuming that the previous cycle is [0s-100s], every 0.5m block, the next cycle is [0.5s-100.5s], and the next cycle is [1s-101s].

Each action of each transaction is accumulated. If two action belong to the same block, then accumulate the total usage consumed directly and accumulate the average usage value_ex; directly. If the action does not belong to the same block as the previous action, the average usage value_ex of the previous period of the block is calculated and the current block usage is assigned to the consumed, and the average usage has a concept of attenuation.

Delta indicates the time difference between the action time and the previous action. It can be seen that if an account operates frequently, its attenuation coefficient decay will always be relatively large, resulting in slow value_ex attenuation, and the limit value will become smaller and smaller, thus quickly reaching the upper limit; while if the operation is not frequent, the attenuation will become faster, and the limit value will become larger and larger, so it is not easy to reach the upper limit.

5 detailed description of the source code

5.1 description of table data structure

As shown in the figure above, there are four tables. Resource_limits_config_index is the block limit configuration table, resource_limits_state_index is the block usage record table, resource_limits_index is the account limit configuration table, and resource_usage_index is the account usage record table. Usage_accumulator is the cumulative usage structure of CPU&NET, which is used by block usage record table and account usage record table. Elastic_limit_parameters is a block restriction structure and is used by the block restriction configuration table.

5.1.1 account restriction configuration table

5.1.2 account usage record form

5.1.3 Block limit configuration table

5.1.4 Block usage record table

5.2 description of table operation interface

The figure above shows which interfaces the four tables are accessed by.

5.2.1 methods related to account restrictions

Set_account_limits: set the memory, CPU and bandwidth weight of the account. Parts that do not need to be modified can be obtained and re-passed through get_account_limits.

Get_account_limits: get the account's memory, CPU, bandwidth weight, external interfaces such as memory purchase, mortgage CPU, bandwidth, and so on, and finally call these two internal methods.

Get_account_cpu_limit_ex: get the account CPU limit (total limit, used, surplus). This method obtains the usage ratio by querying the virtual resource limit value of the block, then the account weight and the weight of the whole network, and finally obtains the actual limit value.

Get_account_net_limit_ex: get the account bandwidth limit (total limit, used, remaining), the same as CPU

Get_account_cpu_limit: get the remaining CPU of the account, and call get_account_cpu_limit_ex internally

Get_account_net_limit: get the remaining bandwidth of the account. Call get_account_net_limit_ex internally.

5.2.2 account use of related methods

Update_account_usage: the tagged account uses CPU and bandwidth resources; the usage is 0 only, indicating that the account initiated a transaction and applied for resource usage at this time; it is triggered when the transaction is initialized; even if subsequent transactions fail, there will be a record of its usage during this cycle, which will be used to calculate CPU and bandwidth usage.

Add_pending_ram_usage: increase account memory usage: this method will be called by system methods such as creating accounts, deploying contracts, modifying permissions, and other intelligent contract APIs that need to operate the status database.

Get_account_ram_usage: query account memory usage.

Verify_account_ram_usage: check whether the account memory usage exceeds the limit.

Add_transaction_usage: increase account CPU and bandwidth usage.

The API for querying account CPU and bandwidth usage is in the same API as querying account CPU and bandwidth limit. For more information, please see the methods related to account limit.

5.2.3 methods related to block restrictions

Set_block_parameters: set block CPU and bandwidth limits.

Get_virtual_block_cpu_limit: query virtual CPU limits for chunks.

Get_virtual_block_net_limit: query virtual bandwidth limits for chunks.

Process_account_limit: effective account weight and network-wide weight; for example, if the amount of CPU mortgage increases, the amount of CPU mortgage of the whole network increases synchronously. When a specific update takes effect in a set_account_limits transaction, it will not take effect until process_account_limit is called when the block is packaged.

5.2.4 Block usage record Table

Get_block_cpu_limit: query the remaining CPU usage of the block, that is, the limit value minus the amount used.

Get_block_net_limit: query the amount of remaining bandwidth available in the block, that is, the limit minus the amount used.

Process_block_usage: update the block usage record; called when the block is packaged. Count the current block usage pending_net/cpu_usage into average_block_net/cpu_usage, then adjust the virtual resource limit virtual_net/cpu_limit according to the current block usage, and finally reset pending_net/cpu_usage.

5.3 Analysis from a data perspective

5.3.1 account restrictions and use

As shown in the above figure:

When creating the core account and other accounts of the system, CREATE records the limit configuration of the account. The default CPU, bandwidth and memory limit is-1, which means there is no limit.

An interface setalimits is provided in the eosio.bios smart contract to set the account weight, and in the eosio.system smart contract, the interfaces such as buying memory (buyram), selling memory (sellram), mortgaging CPU and bandwidth (delegatebw), redeeming CPU and bandwidth (undelegatebw) are provided to SET the account weight.

The newly modified account weight does not take effect until the current block is packaged, and will not take effect until the block is packaged.

When sending a transaction, call the relevant GET API to query the account limit of the transaction authorizer.

The above is the process related to account restriction. The following is an analysis of the process related to account usage:

EOS accounts are created by other existing accounts. When creating an account, the creator needs to consume memory, CPU and bandwidth. The created account does not consume resources. When the account is created, CREATE has a usage record. The default usage value is 0.

When the account is authorized to conduct transactions, call the relevant API SET account usage.

When the account is authorized to conduct transactions, call the relevant API GET account usage.

5.3.2 Block restrictions on use

As shown above (green is the main operator, and blue is the API query interface):

When the chain is initialized, CREATE has a block limit record. The maximum limit of block CPU is 200ms, which is no more than half of 500ms. Only the transactions processed by the node itself need to record its resource consumption, and the transactions broadcast by other nodes do not record the resource consumption, which can be understood as a block with its own 200ms processing transactions and the rest of the time waiting for other nodes to synchronize transactions (this logic is uncertain and needs further research). The minimum CPU consumption per transaction is 100us. The maximum bandwidth limit of the block is 1m. This design is based on the assumption that a single transaction takes time-consuming 100us and consumes 200 bytes of bandwidth, and the TPS can reach 10,000 (1 second 2 chunks 500ms 1M/200bytes 100us100 = 10,000, 1 second 2 chunks 1M/200bytes > 10,000). The cumulative period of block usage is 1 hour. The cumulative period of account usage is 24 hours.

There is a SET block limit when the block is packaged, but the value of the current source code implementation is the same as the default value.

During transaction initialization and transaction packaging, the block limit information is GET, and the actual account limit is obtained through the block virtual resource limit (account weight / network-wide weight).

When a block is packaged, the block limit information is GET to determine whether the resource usage of the block exceeds the standard.

Some external API APIs will also call relevant methods to query blocks and account restrictions.

The above is the process related to block restriction. The process related to the use of block is analyzed below:

The block usage table mainly consists of four parts: average usage in the cycle, current block usage, network-wide weight and virtual resource restrictions.

During chain initialization, the CREATE and SET virtual resources are limited to the maximum limit value cpu/net_limit_paramters.max in the block configuration table.

The block usage is SET when the deal is packaged, that is, the usage of all transactions in that block is accumulated.

The average usage during the block cycle, the virtual resource limit, and the block usage are reset when the block is packaged.

Some external API APIs will also call relevant methods to query block usage information.

5.4 Analysis from the perspective of process

The main process includes transaction initialization (transaction_context::init), transaction execution (transaction_context::exec), transaction packaging (transaction_context::finalize), and block packaging (controller_impl::finalize_block).

5.4.1 transaction initialization

The transaction initialization method calculates the total amount of CPU and bandwidth consumed by all action in the transaction, calculates CPU and bandwidth limits, and makes a basic check on CPU and bandwidth.

In this transaction, the bandwidth limit net_limit is the current block maximum limit minus the current block used; if the limit is too large and exceeds the single transaction maximum limit, the net_limit is changed to the single transaction maximum limit; if the transaction is configured with additional limit parameters and is less than net_limit, net_limit changes to the limit parameter value

CPU restrictions on this transaction are the same.

Check the CPU consumed by this transaction (the logic is not clear and may be related to deferred transactions)

Mark that the relevant account has sent the transaction and applied for the use of resources to calculate the cumulative usage.

Calculate the bandwidth and CPU usage of this transaction

Increase the bandwidth usage of this transaction to the local variable net_usage, and check

Check whether this transaction is timed out.

5.4.2 transaction execution

As shown in the figure above, the transaction execution method mainly executes all relevant action, that is, the interface in the intelligent contract. If there are commands to operate the status database in the intelligent contract interface, the memory usage record of its account will be changed.

5.4.3 deal Packaging

As shown in the figure above, the transaction packaging method checks account memory usage, CPU usage, bandwidth usage, block bandwidth usage, and CPU usage.

Check account memory usage

Check the bandwidth usage of this block

Check the CPU usage of this block

Check the account bandwidth and CPU usage, and add this usage to the block pending_usage and check.

5.4.4 Block packing

As shown in the figure above, the main methods of block packaging are:

Make the result of the modified account resource use weight in the block effective

Modify the block limit, but it hasn't actually changed.

Deal with average block usage and adjust virtual resource limits. If the usage of the block exceeds the expected target, the limit value is reduced, otherwise the limit value is expanded.

After reading the above, have you mastered how to analyze the methods of resource management in EOS source code? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.