Shulou(Shulou.com)06/02 Report--

The main content of this article is "when to use Java message queue", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn when to use Java message queue.

When message queuing is needed

When you need to use message queuing, you first need to consider its necessity. There are many scenarios in which mq can be used. The most commonly used scenarios are business decoupling / final consistency / broadcast / off-peak flow control, etc. On the other hand, if you need strong consistency and focus on the processing results of the business logic, then RPC seems more appropriate.

Decoupling

Decoupling is the most essential problem to be solved in message queue. The so-called decoupling, to put it simply, is a transaction that only cares about the core process. If you need to rely on other systems but are not so important, you can be notified without waiting for the result. In other words, the message-based model is concerned with "notification", not "processing".

For example, in Meituan Tourism, we have a product center, the upstream of the product center is connected to the main station, mobile background, tourism supply chain and other data sources; downstream docking is the screening system, API system and other display systems. When the upstream data changes, if we do not use the message system, we are bound to call our interface to update the data, which especially depends on the stability and processing power of the product center interface. But in fact, as the product center of tourism, perhaps only for the tourism self-built supply chain, the success of product center renewal is what they care about. For external systems such as group buying, it is not their duty to update the product center, whether it is successful or failed. They just need to make sure that we are notified when the information changes.

Downstream of us, there may be a series of requirements, such as updating the index, refreshing the cache and so on. For the product center, this is not our responsibility. To put it bluntly, if they come to pull the data regularly, they can also ensure the update of the data, but the real-time performance is not so strong. But using interfaces to update their data is obviously too "heavyweight" for the product center, so it may be more reasonable to issue a notification of product ID changes to be handled by the downstream system.

To give another example, for our order system, we may need to send SMS points to users after the order is paid successfully, but this is no longer the core process of our system. If the speed of the external system is slow (for example, the speed of the SMS gateway is not good), the time of the main process will be much longer, and users certainly do not want to click and pay for several minutes to see the result. Then we just need to notify the SMS system that "we paid successfully" and don't have to wait for it to be processed.

Final consistency

Ultimate consistency means that the two systems are in the same state, either succeeding or failing. Of course, there is a time limit, in theory, the sooner the better, but in practice, in various abnormal cases, there may be a certain delay to reach the final consistent state, but the final two systems are in the same state.

There are some message queues for "ultimate consistency" in the industry, such as Notify (Ali), QMQ (Qunar), etc., which are designed for highly reliable notifications in the trading system.

To understand the ultimate consistency in terms of the transfer process of a bank, the requirement of the transfer is very simple. If system A deducts money successfully, then system B will succeed in adding money. Otherwise, roll back together as if nothing had happened.

However, there are many possible surprises in the process:

A deducted money successfully, but failed to call the API B to add money.

A deducts money successfully. Although the call to API B to add money is successful, a network exception causes a timeout when the final result is obtained.

A deducts money successfully, B fails to add money, A wants to roll back the money, but A machine downmachine.

It can be seen that it is really not so easy to get this seemingly simple thing done. The common solution to all cross-VM consistency issues from a technical point of view is:

Strong consistency, distributed transactions, but landing is too difficult and the cost is too high, which will be mentioned later.

The ultimate consistency is mainly in the way of "recording" and "compensation". Before doing all uncertain things, write things down, and then do uncertain things, the result may be: success, failure or uncertainty, "uncertainty" (such as timeout, etc.) can be equivalent to failure. Success can clean up the records, and for failures and uncertainties, you can rely on scheduled tasks and other ways to redo all the failed things until you succeed.

Back to the example just now, when A deducts money successfully, the system records the matter of "notification" to B in the library (in order to ensure the highest reliability, the notification B system can be maintained in a local transaction). If the notification is successful, the record will be deleted, and the failure or uncertainty of the notification will rely on the scheduled task to notify us compensatively until we update the status to the correct one.

The whole model can still be based on RPC, but can be abstracted into a unified model, based on the message queue to do an "enterprise bus".

Specifically, local transactions maintain business changes and notification messages, land together (if they fail, roll back together), and then RPC arrives at broker. After successful landing of broker, RPC returns success, and local messages can be deleted. Otherwise, the local message is constantly retransmitted by scheduled task polling, which ensures that the message can reliably land on the broker.

The process of sending a message from broker to consumer is similar, sending the message until the consumer sends a successful confirmation of the consumption.

We first ignore the problem of repeated messages, through two news landing plus compensation, downstream will certainly be able to receive the news. Then rely on the state machine version number and other ways to do heavy judgment, update their own business, and achieve the ultimate consistency.

Ultimate consistency is not a necessary feature of message queues, but you can rely on message queues to do the ultimate consistency. In addition, all message queues that do not guarantee 100% message loss cannot theoretically achieve final consistency. Well, we should say 100% in theory, troubleshooting serious system failures and bug.

Designs such as Kafka have the possibility of losing messages at the design level (such as flushing the disk regularly and losing messages if the power is lost). Even if only 1/1000 of the news is lost, the business must use other means to ensure that the results are correct.

Broadcast

One of the basic functions of message queuing is broadcasting. If there is no message queue, every time a new business party accesses, we have to jointly call a new interface. With the message queue, we only need to care about whether the message is delivered to the queue. As for who wants to subscribe, it is a downstream matter, which undoubtedly greatly reduces the workload of development and co-tuning.

For example, as mentioned at the beginning of this article, the product center publishes the news of product change, and there are many news to be updated in the scene library, there may be many "concerned" parties, but the product center and scenic spot library only need to release the change message, and who cares who can access it.

Off-peak and flow control

Imagine that the upstream and downstream have different abilities to deal with things. For example, it is not a magic thing for the Web frontend to receive tens of millions of requests per second. You just need to add a little more machines and build some LVS load balancer devices and Nginx. However, the processing capacity of the database is very limited, even if the table is added to the SSD database, the processing capacity of the single machine is still at the level of ten thousand. Due to cost considerations, we cannot expect the number of machines in the database to catch up with the front end.

This problem also exists between the system and the system. For example, due to the short board effect, the speed of the SMS system is stuck on the gateway (hundreds of requests per second), which is not the same order of magnitude as the concurrency of the front end. However, when users receive text messages in about half a minute at night, there is generally no big problem. If there is no message queue, it is not impossible to implement complex solutions such as negotiation and sliding windows between the two systems. However, with the exponential growth of system complexity, it is bound to store upstream or downstream, and have to deal with a series of problems such as timing, congestion and so on. And whenever there is a gap in processing capacity, a separate set of logic needs to be developed to maintain it. Therefore, using the intermediate system to dump the communication content of the two systems, and when the downstream system has the ability to process these messages, it is a relatively general way to process these messages.

In short, message queuing is not a panacea. RPC is superior to message queuing for those that require strong transaction assurance and are delay-sensitive.

You can use message queues to do something that is innocuous or important to others but less concerned about yourself.

Message queues that support ultimate consistency can be used to handle "distributed transaction" scenarios that are less latency-sensitive and may be better than bulky distributed transactions.

When there is a gap in the processing capacity of the upstream and downstream systems, use the message queue to make a general "funnel". When the downstream has the ability to handle it, it will be distributed.

If there are many systems downstream that care about notifications from your system, use message queues decisively.

The dispute over the genre of message queue

The title of this article is very difficult to get up, online is full of various MQ performance comparison, it is easy to make people think that I am also such a "vulgar" person (o (╯□╰) o). What I want to express in this article is that they are not the same thing at all, and their hairy performance is easy to compare.

What is MQ?

Message Queue (MQ), message queuing middleware. Many people say that MQ achieves async and decoupling of applications by separating the sending and receiving of messages. The intuition is that MQ is asynchronous and used to decouple, but this is only the effect of MQ, not the purpose. The real purpose of MQ is for communication, shielding the underlying complex communication protocols, and defining a set of application layer, simpler communication protocols. In a distributed system, the communication between two modules is either HTTP or self-developed TCP, but these two protocols are actually original protocols. It is very difficult for HTTP protocol to communicate between two ends-module A can call either BMageB or active A, if you want to do this, you have to carry WebServer on both ends, and it doesn't support persistent connections (the library of HTTP 2.0 can't be found at all). TCP is even more primitive, sticky bags, heartbeats, proprietary protocols, and the thought of the scalp makes the scalp numb. What MQ needs to do is to build a simple "protocol"-the producer / consumer model-on top of these protocols. The "protocol" brought to me by MQ is not a specific communication protocol, but a higher-level communication model. It defines two objects-the one who sends the data is called the producer, and the one who consumes the data is called the consumer, which provides a SDK that allows us to define our own producers and consumers to communicate messages regardless of the underlying communication protocol.

The genre of MQ

Listing menus to compare MQ differences or to have a "MQ performance contest" is crazy, the first thing to do should be classification. My understanding of MQ is divided into two schools.

There is broker.

This genre usually has a server as a Broker through which all messages are forwarded. The producer ends his task by sending the message to it, while the Broker actively pushes the message to the consumer (or the consumer actively polls).

Multiple Topic flow

Kafka and JMS belong to this genre. Producers send key and data to Broker, and Broker compares key and decides to which consumer. This model is our most common model, and it is our most common impression of MQ. In this mode, a topic is often a relatively large concept, and even there may be only one topic,topic in a system, which in a sense is queue. The producer sending key is equivalent to saying, "hi, put the data in the queue of key."

As shown in the figure above, Broker defines three queues, key1,key2,key3. Producers send key1 and data,Broker when sending data. When pushing data, push data (or bring key). Although the architecture is the same, I don't know how many times the performance of kafka is higher than that of jms, so basically this type of MQ has only kafka as an alternative. If you need a violent data stream (performance is more important than flexibility) then kafka is the best choice.

Light Topic flow

The representative of this is RabbitMQ (or AMQP). The producer sends the key and data, and the consumer defines the queue for subscription. After receiving the data, the Broker will calculate the queue corresponding to the key through certain logic, and then give the data to the queue.

Have you noticed? In this mode, key and queue are decoupled. In this architecture, queue is very lightweight (in RabbitMQ, its upper limit depends on your memory). All consumers care about is that their queue; producers do not have to care about who the data is ultimately assigned to. The mapping in the middle layer is called exchange (switch) in AMQP. There are four kinds of exchange--Direct exchange:key in AMQP equals queue;Fanout exchange: ignore key and give a copy to all queue; Topic exchange:key can use "wide character" fuzzy match queue; last powerful Headers exchange: ignore key, by looking at the header metadata of the message to decide which queue to send (AMQP header metadata is very rich and customizable). The structure of this structure brings great flexibility to communication, and all the communication methods we can think of can be expressed in these four exchange. If you need an enterprise data bus (about flexibility), then RabbitMQ is definitely worth using.

No broker

This faction is the "traitor" of AMQP, and a Taoist friend detests that AMQP is too "heavy" (that's because he didn't see how fast it was when he didn't see how easy it was to implement it in Erlang), so he designed zeromq. This Taoist friend is very wise, he is very keenly aware that MQ is a more advanced Socket, it is to solve communication problems. So ZeroMQ is designed as a "library" rather than a middleware, and this implementation can also be achieved-- without broker.

Messages communicated between nodes are sent to each other's queues, and each node is both a producer and a consumer. What ZeroMQ does is encapsulate a set of API similar to scoket that can send data and read data. If you think about it, ZeroMQ is actually like this.

Did you have an epiphany? Actor model, ZeroMQ is actually a cross-language, heavyweight Actor model mailbox library. You can think of your program as an actor,zeromq library that provides mailbox functions; zeromq can achieve IPC communication on the same machine as well as TCP and UDP communication on different machines. If you need a strong, flexible, savage communication capability, don't hesitate to zeromq.

Can MQ only be asynchronous?

The answer is no, first of all, ZeroMQ supports request-> reply mode; secondly, RabbitMQ provides that RPC is an out-and-out synchronous communication, and only JMS and kafka architecture can only do async. Many of us have this illusion when we first come into contact with MQ, such as JMS.

Summary

Kafka,zeromq,rabbitmq represents three completely different styles of MQ architecture; the concerns are completely different:

What kafka cares about is performance, speed

Rabbitmq pursues flexibility.

Zeromq pursues lightweight and distributed.

If you use zeromq to transfer a large amount of data, either the memory of the producer "explodes" or the consumer is "crushed"; if you use kafka as the communication bus, it will definitely not be faster but slower; if you want rabbitmq to be distributed, it is really difficult for it.

An overview of how to design a message queue

Now that we know the usage scenario of the message queue, the next step is how to design and implement a message queue.

The message-based system model does not necessarily require broker (message queuing server). Akka (actor model) and ZeroMQ on the market are actually message-based system design paradigms, but there is no broker.

The reason why we want to design a message queue and equip it with broker is nothing more than two things:

The dump of the message, delivered at a more appropriate time, or through a series of means to assist the message can eventually be delivered to the consumer machine.

Standardize a paradigm and a general model to meet the requirements of decoupling, final consistency, off-peak and so on.

Breaking it apart and crushing it, the simplest message queue can be made into a message forwarder, turning one RPC into two RPC. The sender delivers the message to the server (hereinafter referred to as broker), and the server forwards the message to the receiver. It is as simple as that.

Generally speaking, the overall idea of designing a message queue is to build an overall data flow first, such as sending producer to broker,broker and sending consumer,consumer reply consumption confirmation, broker deletion / backup messages, etc.

Use RPC to string data streams together. Then consider the high availability of RPC and try to be stateless to facilitate horizontal scale.

After that, consider how to carry message accumulation, and then deliver messages at the right time, and the best way to deal with stacking is storage. The selection of storage needs to comprehensively consider many factors such as performance / reliability and development and maintenance costs.

In order to realize the broadcasting function, we must maintain the consumption relationship, which can be preserved by using zk/config server and so on.

After completing the above functions, message queuing is basically implemented. Then we can consider some advanced features, such as reliable delivery, transaction features, performance optimization and so on.

Below we will focus on the design of the message queue module as the main line, interspersed with some message queue characteristics of the implementation methods, to specifically analyze the design and implementation of a message queue in all aspects.

Implement the basic function of queue RPC communication protocol

As mentioned just now, the so-called message queue is nothing more than two RPC and one dump, of course, the situation that requires the consumer to confirm the consumption is three times of RPC. Since it is RPC, it will inevitably involve a series of topics, such as load balancing, service discovery, communication protocols, serialization protocols, and so on. In this area, my strong advice is not to repeat the wheel. Take advantage of the company's existing RPC framework: Thrift, Dubbo, or other custom frameworks. Because there is no essential difference between the RPC of message queue and the ordinary RPC. Of course, it is not impossible to rewrite a RPC framework using the Memchached or Redis protocols (for example, MetaQ uses its own packaged Gecko NIO framework, and Kafka uses a similar protocol). But the cost and difficulty of realization are undoubtedly doubled. Excluding extreme requirements for efficiency, you can use an off-the-shelf RPC framework.

To put it simply, the server provides two RPC services, one for receiving messages and one for confirming receipt of messages. And make sure that no matter which server receives the message and the confirmation message, the result is the same. Of course, this may also involve the issue of cross-IDC services. Here and the principle of RPC is the same, try to give priority to the delivery of this computer room. You might ask, what if producer and consumer are already in two computer rooms? First of all, broker must be aware of the existence of all consumer. Secondly, producer should try his best to choose the nearest computer room.

High availability

In fact, all the high availability depends on the high availability of RPC and storage. First, let's take a look at the high availability of RPC, Meituan's MTThrift-based RPC framework, Ali's Dubbo, etc., which itself has the functions of automatic service discovery, load balancing and so on. As for the high availability of message queues, as long as the interfaces for broker to accept messages and acknowledge messages are idempotent, and several machines in consumer handle messages idempotently, the availability of message queues is transferred to the RPC framework for processing.

So how to guarantee idempotency? The easiest way is to share storage. Broker multi-machine sharing a DB or a distributed file / kv system, then processing messages is naturally idempotent. Even if there is a single point of failure, other nodes can be topped immediately. In addition, failover can rely on compensation for scheduled tasks, which is naturally supported by message queuing itself. We don't need to worry too much about the availability of the storage system itself. Don't worry about it. Give it to DBA boldly.

For queues that do not share storage, such as Kafka using partition plus active / standby mode, it is slightly more troublesome. It is necessary to ensure high availability in each partition, that is, each partition must have at least one master / slave and data synchronization. For details of this HA, please refer to the next pull model message system design.

The ability of the server to carry message accumulation

If a message arrives at the server and reaches the receiver without any processing, the broker loses its meaning. In order to meet a series of needs such as off-peak / flow control / final reachability, it makes sense to store the message and choose the right time to deliver it.

It's just that this storage can be made in many ways. For example, it is stored in memory, in distributed KV, on disk, in database, and so on. But to sum up, there are mainly two kinds: persistence and non-persistence.

The form of persistence can ensure the reliability of messages to a greater extent (such as power outages and other irresistible external forces), and theoretically can carry a greater amount of message accumulation (the space of external memory is much larger than that of memory).

But not every kind of message needs to be persisted. Many messages require more delivery performance than reliability, and there are a large number of messages (such as logs). At this time, the message is stored in memory directly, try failover several times, and finally deliver it.

Message queuing on the market generally supports both forms. Of course, the specific scene should be combined with the business of the company.

Selection of storage subsystem

Let's take a look at the choice of various storage subsystems if data is needed to land. In theory, in terms of speed, file system > distributed KV (persistence) > distributed file system > database, but reliability is quite the opposite. You still need to make the most reasonable choice based on the supported business scenarios. If your message queue is used to support payments / transactions and other requirements for reliability is very high, but the requirements for performance and volume are not so high, and you do not have the time and energy to specialize in file storage system research, DB is the best choice.

However, DB is subject to IOPS, and file-based storage is a better solution if QPS performance of more than 5 digits in a single broker is required. On the whole, we can use the way of data file + index file to deal with, the design of this part is more complex, we can refer to the design of storage subsystem in the second part.

Distributed KV (such as MongoDB,HBase), or persistent Redis, because of its friendly programming interface and considerable performance, is a good choice in scenarios where reliability requirements are not so high.

Analysis of consumption relationship

Now our message queue initially has the ability to dump messages. The next important thing is to parse the send-receive relationship and deliver the message correctly.

Message queues on the market define a bunch of confusing nouns, such as Topic/Queue,Kafka in the JMS specification, Exchange in Topic/Partition/ConsumerGroup,RabbitMQ, and so on. Apart from the phenomenon, there is nothing more than the difference between unicast and broadcasting. The so-called unicast is point-to-point, while broadcasting is point-to-many. Of course, for most Internet applications, inter-group broadcast and intra-group unicast are the most common situations.

Messages need to be notified to multiple business clusters, and there are many machines in a business cluster, as long as one machine consumes the message.

Of course, this is not absolute, and most of the time, intra-group broadcasts are also applicable to scenarios, such as local cache updates and so on. In addition, the consumption relationship may have a multi-level tree relationship in addition to intra-group and inter-group. This situation is too complex to be taken into consideration. Therefore, the general design is to support inter-group broadcasting, and different groups register different subscriptions. Different machines in the group will unicast if they register the same ID, or broadcast if they register a different ID (such as IP address + port).

As for the maintenance of broadcast relationship, generally, because the message queue itself is a cluster, it is maintained on public storage, such as config server, zookeeper and so on. What needs to be done to maintain a broadcasting relationship is basically the same:

Maintenance of the sending relationship.

Send a notification of a relationship change.

Queue advanced feature design

The above are all the implementations of the basic functions of the message queue. Let's take a look at some content related to the characteristics of the message queue, regardless of reliable delivery / message loss and duplication, transactions and even performance, not every message queue will take care of. Therefore, according to the needs of the business, to carefully weigh the costs, advantages and disadvantages of the implementation of various features, and finally make the most reasonable design.

Reliable delivery (final consistency)

This is an exciting topic. Is it possible not to lose news at all? The answer is, it is entirely possible, provided that the message may be duplicated and, in exceptional cases, accept the delay of the message.

The solution is simple and simple, that is, whenever something unreliable (RPC, etc.) happens, the message is landed and then sent. When you fail or are not aware of a successful failure (such as a timeout), the message status is waiting to be sent, and the timed task constantly polls all the messages to be sent, which will eventually be delivered.

Specifically:

Before producer sends a message to broker, it needs to do a landing.

After the request is sent to the server, the server ensures that the data is landed and then tells the client that it is sent successfully.

Message queues that support broadcasting need to persist a delivery state for each endpoint to be sent until all endpoint states are OK before the message can be deleted.

For all kinds of uncertainties (timeout, downtime, message not delivered, data not landing after delivery, data landing reply not received), it is one thing for the sender, that is, the message is not delivered.

The problem with repushing messages is that messages are duplicated. Repetition and loss are like two nightmares, you have to face one. Fortunately, there is still a chance to deal with the repetition of the message, so it is difficult to get it back if the message is lost.

Anyway, as a mature message queue, should try its best to reduce the possibility of repeated delivery in all links, and should not indulge in random delivery just because there is a solution.

Finally, not all systems require final consistency or reliable delivery, such as a forum system or a recruitment system. A duplicate resume or topic may seem more unacceptable to users than losing a post. Keep repeating the sentence that any underlying component should serve the business scenario.

Consumption confirmation

When broker delivers the message to the consumer, the consumer can immediately respond to me and receive the message. But receiving this message is only the first step, and I'm not sure whether I can deal with it or not. Maybe because of the consumption capacity, the load of the system can no longer handle this message, or the message mentioned in the state machine is not the message I want to receive, so I take the initiative to resend it.

Separate the delivery of messages from the processing of messages, so as to truly realize the essence of message queues-decoupling. Therefore, it is necessary to allow consumers to take the initiative to confirm consumption. Of course, the default Auto Ack is OK for messages without special logic, but the consumer must be allowed to take the initiative to ack.

There is nothing special about the correct consumption of ack. But for reject and error, special instructions are needed. The matter of reject is often imperceptible to the business side, and the assessment of system traffic and health, as well as the assessment of processing capacity is a very complex matter. To take an extreme example, the build index starts when a message is received. It may take half an hour to process the message, but the number of messages is very small. Therefore, reject is suggested to make a sliding window / thread pool similar model to control.

When the spending power does not match, refuse directly and send it again after a period of time to reduce the burden of the business.

But the business error is known only to the business side, like the state machine mentioned above, and so on. At this time, the business side should be allowed to take the initiative to ack error, and the next delivery time can be agreed with broker.

Repeat messages and sequential messages

As mentioned above, duplicate messages cannot be avoided at 100%. Unless loss is allowed, can sequential messages be 100% satisfied? The answer is yes, but the conditions are more stringent:

Allow messages to be lost.

It is a single point and single thread from the sender to the server to the recipient.

Therefore, absolute sequential messages can not be realized, of course, in the message queue of METAQ/Kafka and other pull models, single-thread production / consumption, excluding message loss, is also a solution of sequential messages.

Generally speaking, in the design paradigm of a mainstream message queue, we should minimize duplicate messages without losing messages, and do not guarantee the delivery order of messages.

When it comes to repetitive messages, there are two main topics:

How to identify duplicate messages and deal with duplicate messages idempotently.

How a message queue minimizes the delivery of duplicate messages.

Let's take a look at the first topic. Each message should have its unique identity. Whether the MessageId is customized by the business side or generated according to the IP/PID/ timestamp, if there is a place to record the MessageId, the message arrival can be compared.

Be able to complete the repeated identification. The unique key of the database / bloom filter/ key in the distributed KV is a good choice. Since messages cannot be permanently stored, it is theoretically possible that the upstream is still delivered at the moment the message is removed from the persistent storage (upstream delivery fails for various reasons, keeps retrying, and it is time to clean up the message downstream). This kind of thing only happens under abnormal circumstances, after all, it is a minority situation. If the message hasn't been delivered in two minutes, what if you send it one more time? Idempotent processing of messages is an art, repeated messages or messy messages for a variety of reasons have arrived, talking about two general solutions:

Version number.

State machine.

Business

Persistence is a feature of transactions, but only satisfying persistence does not necessarily satisfy the characteristics of transactions. Let's take the example of deducting money / adding money. To meet the consistency characteristics of transactions, either none of them must be carried out or all of them must be successful.

There are two general solutions:

Two-phase commit, distributed transaction.

Local transaction, local landing, compensation sending.

The biggest problem of distributed transactions is that the cost is too high, and the two-phase commit protocol is almost an unsolvable black hole for arbitration downmachine or single point of failure. For transaction-intensive or Istroke O-intensive applications, there is no way to withstand such high network latency and system complexity.

And mature distributed transactions must be built with more reliable commercial DB and commercial middleware, the cost is also too high.

So how do you use local transactions to solve the problem of distributed transactions? Take the local and the business to create a table in a database instance as an example, and insert the message into the local database in the same transaction as the deduction business operation. If the message fails, the business rolls back; if the message is successfully loaded, the transaction commits.

Then send the message (note that it can be sent in real time, there is no need to wait for timing tasks to check out, in order to improve the real-time performance of the message). The future problem is mentioned in the previous final consistency problem, which has been retried by scheduled tasks as long as the message is not sent successfully.

A key point here is that local transactions do business landing and message landing transactions, not business landing and RPC successful transactions. Many people here are easily confused. If it is the latter, it is undoubtedly a transaction nested RPC, which is a big taboo, and there will be various risks such as long transaction deadlocks.

As long as the message is successfully landed, there is to a large extent no risk of loss (except for physical disk damage). As long as the message is delivered to the server for confirmation before it is deleted locally, the reliable delivery of producer- > broker is completed, and when the message storage is abnormal, the business can be rolled back.

There are two biggest barriers to using local transactions:

The configuration is more complex, and the "kidnapping" business must provide a database table for the local database instance.

It is not suitable for services with high sensitivity to message latency.

Then again, not every business needs strong transactions. Deduction and increase of money require transaction guarantee, but issuing orders and generating text messages do not require transactions. You cannot request a rollback of the order business due to the failure of message storage and delivery. Therefore, a complete message queue should define the types of messages it can deliver, such as transactional messages, local non-persistent messages, and unreliable messages on the server side. Make different choices for different business scenarios. In addition, the use of transactions should be as low-cost and transparent as possible, and can be extended based on existing mature frameworks, such as Spring's declarative transactions. The business side only needs to use the @ Transactional tag.

Performance-related async / synchronization

First of all, to clarify the concept, async, synchronization and oneway are three things. Asynchronism, in the final analysis, you still need to care about the results, but you may not care about the results at that time. You can use polling or callback to deal with the results. Synchronization needs to be concerned at that time.

While oneway is sent regardless of life or death, this is applicable to some scenarios where there is no requirement for reliability at all, but it is not the focus of our discussion.

From a regression point of view, any RPC exists client-side async and server-side async, and can be arbitrarily combined: client-side synchronization to server-side async, client-side synchronization to server-side synchronization, client-side synchronization to server-side synchronization.

For the client, synchronous and asynchronous are mainly the difference between getting a Result or Future (Listenable). The implementation can be thread pooling, NIO, or other event mechanisms, which are not discussed here.

Server-side asynchrony may be a little more difficult to understand, which is supported by the RPC protocol. Referring to the servlet 3.0 specification, the server can spit a future to the client and notify the client when the future done occurs.

You can refer to the following code for the whole process:

Client synchronization server async.

Future future = request (server); / / server immediately returns futuresynchronized (future) {while (! future.isDone ()) {future.wait (); / / notify the future after server processing and modify the isdone flag}} return future.get ()

Client synchronization server synchronization.

Result result = request (server)

Client-side asynchronous server-side synchronization (in this case, thread pool).

Future future = executor.submit (new Callable () {public void call () {result = request (server);}}) return future

Client async server async.

Future future = request (server); / / server returns future return future immediately

What has been said above is actually intended to free us from two misunderstandings:

RPC can only be done asynchronously on the client side, not on the server side.

Asynchrony can only pass through the thread pool.

So what is the biggest benefit of using asynchrony on the server side? In the final analysis, it frees the thread and the Icano. Imagine that the server has a bunch of Imax O waiting to be processed. If every request needs to be synchronized and every message needs the result to be returned immediately, then it is almost impossible to do the Imax O merge.

(of course, the interface can be designed as batch, but the number of messages sent by batch may still be small.) If you return it to the client future asynchronously, you can have the opportunity to merge the messages sent in several batches together (especially for databases such as MySQL that allow batch insert), and completely release the thread. Without saying how many threads are requested, the amount of concurrency that can be supported is increased in a straight line.

Looking at the second misunderstanding, the way to return future is not necessarily just the thread pool. In other words, synchronous or asynchronous operations can be performed within the thread pool, or asynchronous operations (NIO, events) can be used without the thread pool.

Going back to the issue of message queuing, we certainly do not want the sending of messages to block the main process (as mentioned earlier, if the server side uses the asynchronous model, it may cause a certain degree of message delay due to message merging). So you can first use the thread pool to submit a send request, and the main process continues.

But do requests in the thread pool care about the result? Of course, you must wait for the server message to land successfully before the message is sent successfully. So the model here, to be exact, the client side is semi-synchronous and semi-asynchronous (using a thread pool does not block the main process, but tasks in the thread pool need to wait for the server side to return), and the server side is purely asynchronous. The client's thread pool wait spits back on the future on the server side, and the block is not unblocked until the server side has finished processing.

In a word, synchronization can guarantee the result, asynchronism can guarantee efficiency, and reasonable combination can achieve the best efficiency.

Push or pull?

Most of the message queues mentioned above are designed for the push model. Now there are many classic and mature pull model message queues on the market, such as Kafka, MetaQ and so on. This is very different from the traditional push method in JMS, and it can be said to find a different way.

We briefly analyze the advantages and disadvantages of push and pull models.

Slow consumption

Slow consumption is undoubtedly the biggest fatal wound of the push model. Dressed as an assembly line, if the speed of the consumer is much slower than that of the sender, it is bound to cause the accumulation of messages in the broker. Assuming that these messages are useful and cannot be discarded, the messages should always be saved on the broker side. Of course, this is not the most deadly, the most deadly is that broker pushed a bunch of messages to consumer that consumer could not handle, consumer was either reject or error, and then played the ball back and forth.

In contrast to the pull mode, consumer can consume on demand without worrying about harassing itself by messages that it cannot handle, and it is relatively easy for broker to stack messages. It is not necessary to record the status of each message to be sent, but only needs to maintain the queues and offsets of all messages. Therefore, for slow consumption such as indexing, the number of messages is limited and the speed of arrival is uneven, pull mode is more appropriate.

Message delay and busy, etc.

This is the biggest deficiency of the pull model. Since the initiative rests with the consumer, the consumer cannot accurately decide when to pull the latest news. If the pull gets the message, you can continue to go to pull. If there is no pull to get it, you need to wait a period of time to re-pull.

But it is hard to determine how long to wait. You might say that I can have a xx dynamic pull time adjustment algorithm, but the essence of the problem is that it is not up to the consumer to decide whether there is news or not. Maybe 1000 messages came in succession in one minute, and then no new messages were generated in half an hour.

Maybe your algorithm calculates that the next time is most likely to come in 31 minutes, or 60 minutes later, and the next message arrives in 10 minutes. Isn't that frustrating?

Of course, this is not to say that there is no solution to the delay, the industry's more mature approach is to start in a short time (not too much burden on broker), and then wait for exponential growth. For example, first wait for 5ms, then 10ms, then 20ms, then 40ms... Until news arrives, and then go back to 5ms.

Even so, there is still a delay problem: if the 50ms message between 40ms and 80ms arrives, the message delays the 30ms, and for messages that come every half an hour, the overhead is wasted.

In Ali's RocketMq, there is an optimized approach-long polling, to balance the shortcomings of the push-pull model. The basic idea is: if consumers try to pull failed, not directly return, but hang the connection there wait, if the server has a new message, the connection notify up, this is also a good idea. However, the cost of the massive long-connected block to the system should not be underestimated. It is still necessary to evaluate the time interval reasonably, and it is better to add a time limit to the wait.

At this point, I believe you have a deeper understanding of "when to use Java message queues". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.