Shulou(Shulou.com)06/01 Report--

This article mainly introduces the Logback and Log4j2 log framework performance comparison and tuning example analysis, the article is very detailed, has a certain reference value, interested friends must read it!

Performance Test logback synchronization Log

Time-consuming

Without any tuning, the above figure is obtained by using the default configuration of Logback. 1 million logs take time to print (ms). As shown in the figure: the performance is the best in single thread, and the time consumption decreases with the increase of the number of threads.

Thread occupancy

Single thread

Non-blocking state

Multithreading

When multithreading prints logs, there will be a large number of thread blocking, and the more threads, the more blocking states.

Four threads

Eight threads

Sixteen threads

Lock occupancy

The thread occupies the lock multiple times. If you look at the Logback source code, you can see that checking the capacity, putting it into the queue, and taking out the queue all need to be carried out after the lock is acquired.

Asynchronous log (queue expansion)

Sample number 1 million, queue length 1.1 million

Time-consuming

Thread occupancy

Single thread

Multithreading

Four threads

Eight threads

Sixteen threads

Lock occupancy

A lock is required for each write to the queue, and so does the Appender for taking it out of the queue.

Asynchronous log (semi-queue expansion)

Sample number is 1 million, queue length is 500000, and discard policy is not enabled.

Time-consuming

Thread occupancy

Single thread

Multithreading

The write time increases obviously, and the write process is still blocked.

Four threads

Eight threads

Sixteen threads

Lock occupancy

Log4j2 synchronization log

Sample number 1 million, Logger to Appender serial execution, output to file

Time-consuming

Thread occupancy

The thread waits for a long time, mainly because it cannot be written after the buffer ring overflows, and the producer enters the waiting state according to the waiting policy.

Single thread

There is no need to scramble for locks in single-thread production, so there is no blocking in the whole process.

Multithreading

On the whole, the blocking time increases with the number of threads, so multithreading has a great impact on the synchronous log and has a serious performance loss.

Four threads

Eight threads

Sixteen threads

Follow-up monitoring skipped due to too long blocking time

Lock occupancy

Blocking on the output stream on Appender, which is executed in a single thread

Asynchronous log (queue expansion)

Sample number 1 million, queue length 1.1 million, using Yield waiting policy

Time-consuming

Single thread occupies the highest amount, and the time consumption decreases as the number of threads increases until the number of threads exceeds the number of CPU cores. The time consuming of single thread is equivalent to that of logback, and that of multithreading is 2 times shorter than that of logback.

Thread occupancy

Both single-threaded and multithreaded use have no blocking state, ensuring sufficient queue capacity, keeping log operations with high throughput and low latency, and avoiding blocking waiting.

Single thread

Multithreading

Four threads

Eight threads

Using the number of threads equal to the number of CPU cores of the host, log writing is non-blocking and wireless switching.

Sixteen threads

Asynchronous logging (log elimination policy)

Sample number 1 million, queue length 500000, discarding policy enabled

Time-consuming

Thread occupancy

The queue length is 500000, which is normally the same as that of half-queue expansion, resulting in blocking, but the log elimination policy is enabled, and those who cannot be written to the queue will directly abandon the wait without blocking.

Single thread

Multithreading

Four threads

Eight threads

Sixteen threads

Asynchronous log (semi-queue expansion)

Sample number 1 million, queue length 500000, using Yield waiting policy

Time-consuming

When the queue is full, the response time is greatly affected, and the throughput depends on the consumption performance of Appender.

Thread occupancy

When logging with a single thread, the production thread is always working when the first half of the queue is not full, and the production thread begins to wait because the consumption capacity of the second half cannot keep up with the production capacity, resulting in the queue being fully loaded.

Single thread

The waiting time is less than that of multithreading, because the log production speed under single thread is slow, and the log consumption is also much higher.

Multithreading

Some time ago, it was possible to maintain high-performance work, but after the queue was full, it began to send and wait, resulting in a longer time-consuming.

Four threads

Eight threads

Sixteen threads

Lock occupancy

Lock occupancy was not found during logging

Asynchronous logging (waiting policy)

Sample number 1 million, queue length 500000, using Timeout waiting policy

Time-consuming

Thread occupancy

No blocking state is generated

Single thread

Multithreading

There is some blocking because the Timeout wait policy uses locks.

Four threads

Eight threads

Sixteen threads

Lock occupancy

When using the Timeout wait policy, the lock is taken before it is put into the queue, and the consumer thread wakes up action.

Performance tuning asynchronous log

Whether it is logback or log4j2, using asynchronous logs can greatly increase the time-consuming of log operations, and indirectly increase the overall time-consuming of the business side.

Log reliability

Asynchronous logs cannot guarantee the reliability of logs. If the system shuts down unexpectedly, the logs in the queue will be lost. Therefore, highly reliable logs are required, and database or MQ should be selected to ensure.

Logback

Pass through

Set up

Log4j2

Pass through

System.setProperty ("Log4jContextSelector", "org.apache.logging.log4j.core.async.AsyncLoggerContextSelector")

Set up

Log abandonment strategy

Discard the log of the overflow queue and maintain a stable response speed. For the business side, it can maintain a good and stable log service, but some log losses need to be tolerated.

Log4j2

Pass through

System.setProperty ("log4j2.AsyncQueueFullPolicy", "Discard")

Set up

Logback

By specifying a threshold for discard logs

Abandon the boundary

When the remaining capacity of the queue is less than the threshold, logs below ERROR will be discarded.

Disable discard policy

If it is set to 0, it will not be discarded, and the business thread will write after waiting for the queue space to be available.

Default threshold

The default threshold is 20% of the queue length, and the queue length 100 threshold is 20

Log waiting policy

The unique feature of Log4j2, which specifies that when the queue is full, the producer needs to wait without enabling the discard policy.

TimeoutWaitStrategy

The default wait policy of Log4j2, which vacates the wait queue through Object.wait. Locks are added when placed in the queue and are not recommended.

YieldWaitStrategy

Pass through

System.setProperty ("log4j2.asyncLoggerWaitStrategy", "Yield")

Set up. The waiting queue is vacated through System.yield (), which is more efficient than the Timeout waiting policy and more energy efficient than the Busy waiting policy.

Queue capacity

According to the performance test, when the log abandonment strategy is not applicable, the production thread will block waiting for the queue to vacate when the queue is fully loaded, which will directly affect the efficiency of the business side.

Logback

By specifying the queue length, Logback always uses ArrayBlockingQueue as the queue

Log4j2

Pass through

System.setProperty ("log4j2.asyncLoggerRingBufferSize", "x")

Designation

Quadratic length

When calculating the position inside RingBuffer, it is calculated in a binary way, and the calculation speed can be improved by using the exponential length of two.

Length calculation formula

A unified calculation formula has not been found yet. I think it can be calculated through (peak log TPS# consumption TPS) * 15: 60.

Load carrying capacity

The meaning of this formula is that all logs that apply 15 minutes to peak production can be accommodated by the queue.

Cost

From a cost point of view, queues should not be estimated indefinitely. Under the condition that the system is not affected by capacity, the length should be as small as possible to save memory expenses.

Response time

In general, applications should not run at the peak for a long time, and if they run at the peak for a long time, they should expand horizontally to disperse the request pressure. Therefore, to accommodate the peak within 15 minutes, there can be enough time for operation and maintenance to respond and spread the pressure horizontally.

Consumption bottleneck

Log consumption TPS is determined by Appender consumption efficiency. When log TPS exceeds consumption TPS, logs will begin to accumulate in the queue.

Consumption TPS

The number of logs consumed by an Appender in a second. Take FileAppender as an example. Each log consumption costs 100 subtleties (a good host can consume up to 60). A second can consume 10, 000 logs, that is, a consumption TPS of 10, 000.

Request TPS

Generally speaking, Web applications will not carry more traffic than consumption TPS. Assuming that five logs are printed per request, more than 5000 TPS is required to generate log accumulation.

Consumer optimization

When the log TPS exceeds the consumption TPS for a long time (15min +), the consumption capacity should be optimized by using lightweight Appedner, simple Layout, log discard policy, filter, business side specification, etc.

The above is all the content of the article "example Analysis of performance comparison and tuning of Logback and Log4j2 logging Framework". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.