Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how to implement a distributed ID generator based on Ignite". Many people will encounter this dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Distributed atomization types of 1.Apache Ignite

After version 1.5, JDK provides the java.util.concurrent package, in which the java.util.concurrent.atomic subpackage contains a thread-safe programming implementation that supports lock-free for a single variable. The classes in this package, such as AtomicLong, provide atomization operations corresponding to the Long type, such as some increment methods, based on these capabilities, you can develop functions such as a single JVM sequence generator, but there is nothing you can do in a distributed environment. In Ignite, in addition to providing standard key-value-based storage similar to Map, it also provides an implementation of distributed data structures, including IgniteAtomicLong,IgniteSet,IgniteQueue,IgniteAtomicReference,IgniteAtomicSequence,IgniteCountDownLatch,IgniteSemaphore, which not only provides the same functions as JDK, but also adds support for distributed environments, that is, support for cluster-wide atomization operations. Since this article focuses on distributed ID generators, the focus of the following article is on IgniteAtomicSequence.

2.IgniteAtomicSequence

The IgniteAtomicSequence interface provides a distributed atomic sequence, similar to the distributed atomic Long type, but its value can only grow. Its unique function is to support the reservation of a certain range of sequence values to avoid the expensive network consumption and cache updates required every time the sequence gets the next value, that is, when incrementAndGet () (or any other atomic operation) is performed on one atomic sequence. The data structure will reserve a certain range of sequence values forward, and it will guarantee the uniqueness across clusters for this sequence instance. The use of this type is very simple, and the related code is as follows:

Ignite ignite = Ignition.start (); IgniteAtomicSequence seq = ignite.atomicSequence ("seqName", / / sequence name 0, / / initial value true// is created if the sequence does not exist); for (int I = 0; I < 20; iTunes +) {long currentValue = seq.get (); / / get the current value long newValue = seq.incrementAndGet (); / / add 1 before taking the value.}

The seq created in this example starts with an initial value of 0 and then increments, which looks perfect, but when the system restarts for whatever reason, it will start from 0 again, which obviously cannot guarantee uniqueness, so this method still cannot be used in a production environment.

3. Distributed ID Generator based on IgniteAtomicSequence

According to the above, it is very risky to create IgniteAtomicSequence directly according to the initial value of 0, which can not be used in the production environment, and the length is not fixed, so we need to further think of ways, and the focus of the research is to solve the problem of initial value. Because the value of IgniteAtomicSequence is long, the maximum value of long type in Java is 9223372036854775807, which is 19 bits long, which is a large value for practical applications, but it is relatively short for common ID generators that do not have environment dependencies. Therefore, we intend to make a fuss in this aspect. Because an important indicator of system reset is time, it may be an ideal choice to use time as a reference and then add an extension. We use the following rule as the initial value: the yyyyMMddHHmmss+00000 of time is exactly 19 digits, and then add 1 at a time, because it is 2016, and this rule will not exceed the maximum value of long in regular application scenarios. However, there is a risk in this rule, that is, assuming that the actual application and performance are not taken into account, if the operation volume is particularly large, the sequence value will be quickly rounded up, and if the node restarts instantly after a certain time downtime, it is possible that the initial value after restart is less than the original maximum value, so the uniqueness cannot be guaranteed. Here is a calculation of the maximum in this theoretical case, and then developers will know how to improve this rule in their applications to meet personalized needs.

4. Theoretical limit and performance

Assuming that the actual performance is not taken into account, let's take the simplest case, that is, when the access reaches the peak within one second after startup, and then restarts immediately after the downtime, it is easy to see that there is no need to calculate. it is the maximum value of 100000 corresponding to five zeros, and so on, taking into account the difference between carry time and decimal carry We can calculate the maximum transaction volume after one minute, one hour, one day, one month, and one year after downtime, as follows:

1 second per minute per hour per day per month 1 second 100000-1 minute 16610 million-1 hour 27716610 billion-- 1 day 11569444466100 billion-- January 165.99205950535714310 trillion-1 year 32151.997 billion 270,000,000 trillion

Take one minute as an example. Assume that the initial value is 2016011815341200000, and restart immediately after a minute of downtime. The corresponding initial value is 2016011815351200000, the difference is 10000000, and the corresponding transaction volume per second is 166000. Judging from the picture above, the trading volume that can be carried by such a rule is still very large, and the busiest trading system in the world today will not exceed the extreme value of this limit, that is to say, for now, this rule is generally applicable. In actual production, instant restart does not exist, as the restart time goes back, the new initial value will open the gap with the original maximum value, and there is no possibility of conflict. With regard to performance, I tested it on an old laptop in 2011, and I can easily reach the sequence generation speed of 50K/s, which is OK, but this is achieved with reservation enabled, and if reservation is not enabled, performance may drop to 13K/s. In a specific cluster environment, Ignite services are not usually used as ID distribution centers, so developers need to test, evaluate and then choose whether the performance in the actual environment can meet the requirements. In addition, it is also important to note that when reservation is turned on, the resulting ID may not grow linearly over time.

5. Comparison of common distributed ID generators

The aforementioned distributed ID generator based on Ignite has the advantage of simple implementation, the ID generation system after embedding a jar package into the application is consistent with the application life cycle, there is no single point of failure after backup, the numerical value can be linearly increased to compare the size, and the rules can be made shorter according to the business customization. if converted to hexadecimal, it will be very short, do not rely on the database, and do not put pressure on the database. The disadvantages may be performance and some specific business requirements. The requirement of generating globally unique ID is rigid, especially in distributed environment, the problem is particularly complex. At present, there are many implementation schemes in this area, and the general ones are not universal. This paper does not discuss it in detail, but simply enumerates it.

UUID has the advantage of good performance, the disadvantage is relatively long, 128bit, no rules, can not compare the size.

ID distribution center, such as twitter's snowflake service, can achieve high performance, high availability, low latency and orderly time, but the disadvantage is that the whole system becomes complex and the maintenance workload is increased.

MongoDB's ObjectID (similar to UUID) MongoDB driver provides the objectId generation function, which has the advantage of being shorter than UUID and orderly in time, and this id contains a lot of useful information, which can be obtained after decoding.

There are many database-based schemes for database generation, such as Oracle-based and PostgreSQL-based sequences, Flickr and Instagram also have more complex database-based schemes.

Others can make a variety of ID generation solutions to meet various requirements according to different business scenarios, and the more requirements, the more complex the implementation.

This is the end of "how to implement a distributed ID generator based on Ignite". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.