Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what is the principle of Java implementation of the message management platform". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

A brief understanding of "message Management platform"

"message management platform" may have different names in different companies. Sometimes I call it "push system", sometimes I call it "message management platform", and some colleagues call it "touch platform". Even grandiose I can call it "message center".

But in any case, its function is to send messages to users. What is its position in the company? As long as the messages sent in the official name, all go to the message management platform.

Generally, if you sign up for an APP/ website, what message can you receive from that APP/ website? It's usually the following, right?

IM message: it is actually the chat message in APP.

Notification bar (PUSH) message: system pop-up message

Mail (Email) messages

Short message (Sms) message

Wechat service number message

WeChat Mini Programs (Service Notification) message

Why is there a message management platform?

It can be said that almost all companies that do APP will have a message management platform.

Most of the time, we want to send messages to users:

It is possible that the user wants such a feature (reservation activity reminder notification)

It is also possible that we want to "wake up" / "inform" and other operations by sending messages to tell users that we are still here.

So the question is, is it difficult to send a message? Is it complicated to send a message?

Obviously, sending a message is very simple and not complicated at all.

Texting is nothing more than calling the API of third-party text messages, sending e-mail is nothing more than calling email API, sending Wechat messages (mobile QQ / Mini Program / Wechat service number) is nothing more than calling Wechat's API, sending notice bar messages (Push) is nothing more than calling the API of APNS/ mobile phone manufacturers, sending IM messages, you can also use cloud services, and transfer the API... of cloud services.

It is possible that many people's projects are done in this way, which is nothing more than sending a message, and it is not impossible to implement it by yourself.

But the problem with this is that within a company, there will be many projects that have code implementations that "send messages". Suppose there is a problem with sending a message, and you have to solve it yourself.

First of all, the system is difficult to maintain, and secondly, it is not necessary. I am an advertiser, although I want to send a message, why should I do it myself?

When we write code, we may extract the common code into methods that can be called repeatedly by the current project. If the common code is used by multiple projects, we may extract component packages for use by multiple projects. As long as the common code is used by enough people, it is likely to rise from a component to something at the platform (system) level.

How to implement the message management platform?

Going back to the nature of the message management platform, it is a system that can send messages. Then how to design and implement it? Let's start with the interface.

Interface design

The message management platform is a platform that provides messaging services. If I were to implement it, my idea might be to write an interface for each type of message, and then expose these interfaces to the public.

Therefore, there may be the following interfaces:

/ * content: copy sent * receiver: recipient * / sendSms (String content,String receiver); sendIm (String content,String receiver); sendPush (String content,String receiver); sendEmail (String content,String receiver); sendTencent (String content,String receiver); / /....

It seems that this implementation is not impossible, anyway, each interface is very clear, what type of message to send, which interface you call is fine.

Suppose we define the above interface, and now that we are about to send a message, we will have the following scenario:

Copywriter: "Hello, I am three crooked", recipient: "woshisanwai" (sent to one person at a time)

Copywriter: "Hello, I am three crooked", recipient: "woshisanwai,java3y,javayyy" (the same copywriter is sent to multiple people)

If you are a novice, you may think: this is simple, I have two interfaces for each type, which are single and batch interfaces.

SendSingleSms (); sendBatchSms (); / /.

Is it necessary to design like this? It's not really necessary. I'll just define the recipient as an Array's size==1, then I'll send the copy to this person, Array's size > 1, and I'll send it to everyone in the Array.

So we still have only one interface:

/ * content: copy sent * receiver: recipient (multiple or single) * / sendSms (String content,Set receiver)

In fact, we do not define Array here, my interface receiver is still String, if there are multiple uses, the sign separation is fine.

/ * content: copy sent * receiver: recipients (multiple or single), separated by commas * / sendSms (String content,String receiver)

Now there is another scene. What if different copywriters are sent to different people? Some people say, hasn't this already been realized? Just call the above interface directly. It's not like you can't call it repeatedly, for example:

Copywriter: "Hello, I'm Java3y", recipient: "woshisanwai"

Copywriter: "Hello, I am three crooked", recipient: "3y"

Copywriter: "Hello, woshisanwai", recipient: "three crooked"

.

That's true. It could have been done. But not good enough.

Let's take a real scenario: now that an anchor is on the air, you have to send a message telling those who subscribe to the anchor to watch it. In order to improve the effect of this notice, we designed it as follows: {user nickname}, the anchor you subscribed to is already on the air, so go and see it!

This kind of message must be real-time (suppose the speed of pushing the message is too slow, and when the user receives the message and the VJ broadcasts, then the user must not hammer you?)

Voiceover: obviously this situation belongs to different copywriters and sent to different people.

How does this kind of message work at the business level? It may be to scan the DB table, traverse the fans who subscribe to the anchor, and then push messages to them.

Now we can only call the send () API to send a message for every fan who subscribes to the anchor. If the anchor has 500W fans, he will have to call the 500W send interface for many times. Isn't that terrible? The number of calls, the network overhead.

Therefore, we have to provide a "batch" interface that allows callers to pass in different people with different copywriters at a time. So how do you do that? It is also very simple, in fact, the above interface is encapsulated another layer, so that the caller can be "batch" passed in. So the code can look like this:

/ * multiple "groups" (copywriters and senders) are introduced at a time * List * SendParam defines content and receiver * / sendBatchSms (List sendParam)

Now that the "prototype" of the interface has emerged, we have implemented the most basic function of the message management platform: sending messages.

Regardless of the internal implementation, let's assume that we have adapted to the corresponding API, and now our API has sufficient and necessary conditions at the sending level: as long as you pass in the receiver and send content, I can send you a message.

But we call it a platform, ah, how can we make it look like it only encapsulates a few methods? the platform should look like a platform.

Let me cite the most basic function on a daily basis: someone called my interface to send a text message. The copywriter of this message is a CAPTCHA type. He asked me if the message was sent to the user.

If you have connected to SMS, you will know that sending SMS to the user is an asynchronous process.

Call the API of the SMS provider, assuming there is nothing wrong with your input parameter, and it will tell you that the call was successful. If you really want to know whether this message has been sent to the user, you have two ways: first, provide an interface for the SMS service provider to call, and when it is really processed, the SMS service will call your interface and tell you what the end result is. Second, you go to poll the interface of the SMS service provider to get the final result.

Back to the question, he wants him to call my interface and send the message successfully, then I just ask him to get his cell phone number and copywriting, and then take the following steps:

Determine whether the mobile phone number and copy are normal when they are sent (whether there is a real call to the API for sending text messages)

Assuming that the SMS API is successfully sent, check whether the returned receipt (delivery result) is normal.

So at present, we still perfectly support the above problems in the existing interfaces, right? As long as we record the results and receipt information, we can tell him whether the mobile phone number and copy provided have been sent to the user.

Well, today he came and asked: today, there are many people who feedback that they can't receive CAPTCHA messages (not all people can't receive them, but most people). I'd like to know the success rate of CAPTCHA messages sent today.

At this time, I can only match (like%%) how many people his copy has been sent to my API, how many people have successfully sent it to the API of the SMS service provider, and how many people have received the successful receipt (result).

You can finally tell him the result by matching the copywriter, but it's stupid. In the final analysis, it is because the services provided by the system are still too weak.

Then how to solve the problems mentioned above? In fact, it is also very simple, since matching copywriting is stupid, then I can just give him a unique Id for this batch of CAPTCHA messages.

Just like we go to the SMS service provider, we need to create a new SMS template, which represents the content you want to send. After creating the template, we will give you a template Id. You can specify this template Id when you send it.

Then our platform can also play in this way. You want to send a message, right? Yes, first come to my platform to create a "template", then send me the template Id.

Therefore, we have solved the problems mentioned above perfectly.

Now let's discuss whether it is necessary for different message types (SMS, email, IM, etc.) to separate different interfaces, but it is not necessary. Because as long as the concept of "template" is abstracted, we can naturally solidify the message type on the template. As long as the template Id is passed, I will know what type of message you are sending.

In this way, we will eventually have two interfaces: batch and single send interface.

/ * send message API * @ author java3y * / public interface SendService {/ * the same copywriter, send it to * @ param sendParam * / void send (SendParam sendParam); / * send different copywriters to different people, and receive multiple groups of * @ param sendParam * / void batchSend (BatchSendParam sendParam) at a time. } public class SendParam {/ * template Id * / private String templateId; / * message parameters * / private MsgParam msgParam;} public class MsgParam {/ * recipient: if there are multiple, separate * / private String receiver with "," / * * Custom parameters (copywriting) * / private Map variables;}

A single interface refers to sending messages to 1N people at a time, who receive the same copywriting.

Batch interface refers to sending one copy to one person at a time, but one call can send N individuals and the corresponding copy.

The individual and batch here are not defined by the sender's dimension, but by the message copy corresponding to the person.

To give another example, now I send a message to all the students who follow me: "Happy New year, brother and sister-in-law." in this case, I only need to use the send method. I send the same copywriting to a group of people, and the copywriters received by these people are exactly the same.

Request parameters for a single push API call:

{"templateId": 12345, "msgParam": {"receivers": "three crooked, ao C, eggs, rice beans", "variables": {"content": "Happy New year, Big Brother", "title": "give me a like, kiss"}

If I want to send a message to all my classmates who follow me: "{Wechat user name}, Happy New year, Big Brother". In this case, I usually use the batchSend method. Before sending it, the copy corresponding to the combination person is encapsulated into a List. One call to the API means a message to the List.size () group.

Request parameters for a batch API call:

{"templateId": 12345, "msgParam": [{"receivers": "Aobing", "variables": {"content": "Aobing, Happy New year, Big Brother", "title": "give me a like" Pro "}, {" receivers ":" variables "," variables ": {" content ":" Happy New year, Big Brother "," title ":" give me a like, kiss "}}]}

I didn't expect to write this article for so long just on the interface, mainly to take care of inexperienced students.

Review the ideas for designing interfaces:

At first, I wanted each message type to have a different interface.

Considering that the same copy will be sent to multiple people, the receiver parameter must support the input of "batch".

Considering that there will be a scenario of batch calling interface, a batch interface is required.

Considering the need to count the scenarios of sending messages, we need to abstract a "template", and all messages sent on the platform must have a "template".

With templates, you can solidify a lot of information into templates, so eventually we abstract two interfaces: single push and batch.

Let's talk about templates.

We have defined the interface earlier, and the main difference from the simple messaging function you have implemented is the concept of "template".

One thing mentioned above: with a "template", you can solidify a lot of information into a template. So what did we solidify into the template?

The types of messages that can be sent. The message management platform can send many types of messages, so our template needs fields to distinguish different message types. Don't think so hard, in fact, we use 1 for text messages and 2 for e-mail.

The information of the template creator (mobile phone number, name) has nothing to do with the substance of the message, but if there is an indescribable problem with the template, the scapegoat must be found. What if the template creator leaves? It's all right. I will find my department according to the creator, then find the department to take the blame (hehe)

The copywriter of the news. Summing up the message seen above, we can see that a message consists of nothing more than the following parts: content, title, picture, link, video. Different messages can send different copywriters, such as text messages can only have content and links, while notification bar messages (Push) can be composed of title, content, pictures and links. So, we will store the copy of the message in a field in json format.

The business rules of the message. The business rules we are talking about here are not really detailed business, but platform constraints on different message types. For example, at the product level, it is hoped that users will not receive notification bar tweets at night (after all, it will disturb users), and that users will not receive two notifications within an hour. A maximum of N notification bar tweets a day (also because of the user's experience). These platform constraints are suitable to be done on the message management platform, which you can understand as a backstop function.

Send the account number. What? The concept of sending a message and an account number? Are you mistaken, crooked? In fact, it is true. You can choose different email accounts when sending e-mail, different Wechat official accounts when sending Wechat official account messages (same as Mini Program), and different accounts when sending IM messages. In fact, there are two types when accessing SMS: notification and marketing. We will abstract these into accounts.

Recipient Id type. The IM messages in the station use the userId in the station, the notification bar messages (PUSH) use did, the short credit messages use mobile phone numbers, and Wechat messages use openId. Specify the Id type of the recipient, indicating which type of id you want to pass into the template. Suppose you specify userId, but if you want to send a text message, the message management platform needs to convert userId to a mobile phone number. It is also identified by a field, 1 for userId,2 and did for...

What we can find is that we have stuffed all the information (or even unnecessary information) needed for a message into the template, and when the caller passes in the template Id, I will be able to get all the information I want.

Is that all of a template? Of course not. The above mentioned are the common features of templates. We can also divide them into two types according to the usage scenarios of templates:

Operation template: the operator sends a message to a specified group of people at a certain time and time. This group of people are offline. Example: if a user registers to log in to APP, they can send messages to the user every other day (or even longer). This is a non-real-time (offline) push, which does not require technology to undertake, to circle the crowd and then set the corresponding time to push.

Technical template: the system automatically triggers a batch of messages according to business conditions, and the recipient list also depends on the business scenario (these people are usually real-time). Example: if a user registers and logs in to APP, he or she needs to send a message to that user immediately. This kind of push belongs to real-time push and needs corresponding technology to undertake.

As systems and businesses evolve, the boundaries between operational templates and technical templates become more and more blurred. It essentially provides two ways to send messages:

Circle a group of people and trigger it by using a scheduled task-to-point call interface (the recipient, copywriter, and sending time are all clear).

The technical call interface sends messages (the receiver, copywriting, and sending time are all generated by business logic). Example: welcome to follow Sanjie, your CAPTCHA is: 888. There's a mole, terminate the deal. When you focus on crooked, the system triggers a message. The sending time, verification code value and personnel are all uncertain)

When users create templates on the platform, different types of templates need to fill in different fields: the operation template needs to fill in the crowd and task trigger time, while the technical template does not need to fill in the crowd and task trigger time at all. So our template will have a field to identify whether the template is an operation type or a technology type. 1 indicates the type of operation, 2 indicates the type of technology.

Do you think it's over? nonono, not yet. We will also distinguish the types of messages, which are currently composed of three main categories: notification, marketing, and CAPTCHA.

The question is, why should we distinguish between the types of messages? For statistical purposes? Of course not, there is no good statistics for these types of granularity.

Let's take an example to illustrate: on 2020-02-30, a 5000W group of operators chose to send a text message at 8pm, basically telling users that the article had been updated regardless of blood loss. The system executes the task on time at 8: 00 p. M., reads the template information of the template and sends it. 5000W people, can the system send in seconds? Obviously, it won't work.

Voiceover: in addition to considering your own system capabilities, you also have to consider the ability of the downstream to bear. If you fool around, they won't take you to play.

Therefore, it must take a certain amount of time for these 5000W people to be fully distributed, and now we assume that it will be completed in 15 minutes. A CAPTCHA message was triggered at 08:02, and is it reasonable that the CAPTCHA message was delayed because of this 5000W population? It obviously doesn't make sense.

How did it happen? The reason is that the 5000W message and the CAPTCHA message go through the same channel, which causes the CAPTCHA message to be blocked. We can solve the above problem by taking different channels for different message types.

Therefore, at the design level, our system sets the operation template to the marketing type message by default, while the message type of the technical template is chosen by the caller. In the real world, the only thing that can be blocked is marketing news.

Voiceover: the practices mentioned above are all related to usage scenarios and specific business, and they certainly can't be come up with overnight.

The template is over, and I'm not going to dwell on some of the details. Let me briefly summarize it again:

We stuff into the template the information necessary to send a message (copy, sending account, incoming recipient Id type, message type: notification, marketing and CAPTCHA), platform information (business rules: whether to remove duplicates, block, display logic, etc.) and basic information (business party information, message name).

Due to the usage scenario, the template is divided into operational template and technical template. The main feature of the operation template is that it needs to fill in the crowd information and delivery time, and the operation template is scheduled and sent by the message management platform itself.

Interface implementation

BB for such a long time, many people may just want to see: three crooked this forced to write a secret in the title, send a message who can not, not just adjust an API, but also give you to play with flowers?

Don't worry, write it now. Now that we've laid the groundwork for what the interface design and template is, let's get back to the interface implementation.

First, let's take a brief look at the system architecture link diagram of the message management platform:

Voiceover: the interface we mentioned above is defined in the unified call layer (access layer)

When the caller calls our send/batchSend method, will it directly call the downstream API to send the message? No.

It is too risky to directly call the downstream API to send messages, and interface 1W+QPS is normal, so we only do simple parameter verification and information completion after receiving the message and send the message to the message queue. The advantage of this is that the interface access layer is very lightweight, and as long as the Kafka is resistant, the request will be fine.

When it is sent to the message queue, it will be sent to different topic according to different message types, and the sending layer will listen to the topic for consumption. The architecture is roughly as follows:

After the sender consumes the topic, it puts the message on its own memory queue, and multiple threads consume the message from the memory queue to send the message.

What you can see is that we have done sub-topic to achieve business isolation from the access layer to the message queue, and we also put it in our respective memory queues for consumption. This is achieved: different types of messages from different channels and the same channel do not interfere with each other.

Seeing the picture above, students who have thought about it will surely ask: why do you need a memory queue? Anyway, you have already assigned topic at the upper level, and you can achieve what you call "business isolation" without memory queues.

Indeed, the main reason for using memory queues here is to increase concurrency. Increased concurrency, which means that distribution can be faster (in the process of sending messages, the most time-consuming is the network interaction, such as SMS, which can be consumed by more threads).

The business rules mentioned above are done here in the lower layer, including night masking, 1-hour deduplication and Id conversion.

Night shielding is to determine whether it is at night. If night shielding is checked and filtered at night, it will be fine.

To remove duplicates in 1 hour is to use the userId+ message channel as Key to see if it exists on Redis. If so, filter it out.

We have made a system with the function of id conversion. Let me briefly talk about it below, so I won't repeat it.

Voiceover: it's best to use Pipeline to read and write Redis in this scenario.

Then it is adapted to the interfaces of various channels, call API to send messages, this piece is not much different from your individual implementation, calling an interface can also give you a flower? (the code style will be slightly better, and the template method pattern, responsibility chain, producer and consumer model, etc., all have corresponding applications in the project.)

Summarize the implementation of the interface:

When the caller invokes the interface, the interface does not synchronously call the downstream API to send messages, but puts it on the message queue (supports high concurrency)

When queued, it will be classified according to different channels and different types of messages, and put into different topic (service isolation)

When consuming queues, blocking queues are used locally to increase concurrency (speed up consumption)

Id conversion

(extension) as mentioned earlier, different types of messages require different id types: openId for Wechat, mobile number for SMS, and did for push notification bar push.

In most cases, the average caller passes userId to me, and I need to convert userId according to different message types.

So how do we implement the system on our side? The main steps and logic are as follows:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Monitor user changes and Wechat official account subscription / customs topic, clean out a unified data model in Flink, and write the cleaned data to another topic.

The Id mapping system listens to the topic cleaned by Flink and writes to the data source in real time (here we use a search engine)

It's not that hard to look at, is it?

Have you ever wondered why you should use an Id mapping system to listen to the topic washed out by Flink instead of writing directly to the data source in Flink?

In fact, it is no problem to write directly to the data source through Flink, and by encapsulating an Id mapping system, you can do this work in more detail.

What can be found from the description is that only real-time increments are implemented above. In many cases, we will worry that there are problems with increments, resulting in inaccuracy or loss of some data, we will write a full amount, and the same is true of Id mapping.

So how is the full amount of Id mapping done? User data will form a table in Hive through various relationships, and the full amount of Id mapping is based on this Hive table (the information of the Hive table will be read in the early hours of every morning and the data source will be written again).

Based on the above logic, a background management is specially done for Id mapping (full amount can be triggered manually, whether to enable increment / full amount, and modify the time of full trigger)

Data statistics

I think this is the most essential part of the message management platform.

Dreaming back to our original interface design, we introduced the concept of template because of the need for "data statistics". Now that we have a template Id, how do we implement data statistics on our side? Our statistics of messages are based on the dimensions of the template.

When a template is created, a template Id is generated. Based on this template Id, we generate a value called umpId: the first bit is technical / operational push, the last eight bits are dates, and the middle six digits are template Id.

Because all messages will pass through the access layer, as long as the message has a link, we will add a umpid parameter to the link, and the link will be sent through until the user clicks it.

When each system executes a message, it may cause the message not to be sent (it may be because the message is duplicated, the user's mobile phone number is incorrect, or the user has not logged in for too long, etc.). We log in these "key locations" to facilitate our investigation.

We give it a name with simple numbers for these "key locations". For example, we use "11" to indicate that the user is not bound with a mobile phone number, and "12" to represent that the user received an identical message 10 minutes ago, and use "13" to represent the user to block the message.

"11", "12", "13", "14", "15" and "16" these are called "points". If you log these points in key positions, this is called "burying points".

With buried sites, what we need to do is to collect these points, then uniformly process them into our data format and output them to the data source.

Collect logs

Cleaning log

Export to data source

There is logAgent to help us collect logs to Kafka, real-time cleaning logs we use Flink, after cleaning we output to Redis (real-time) / Hive (offline).

Data sample of Hive table (mainly used for offline report statistics):

Redis will be stored in multiple dimensions to support our business needs. For example, to find out why a message failed to be sent, search it through userId and finish it directly (real-time records are recorded in Redis, so what is read here is Redis data)

For example, check the overall distribution of a message through the template Id:

Why do I say this is the quintessence of the message management platform? umpId runs through all the systems that the message management platform passes through. As long as the messages sent on the message management platform are recorded and sent, you can quickly track the delivery of messages through points.

Summarize the statistics:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Design the business umpid, and add umpdId parameters to all message push links.

Get through the upstream and downstream, jointly design and maintain key points, and unify the log format to achieve cross-platform collection and cleaning

Taking into account real-time and offline requirements to write to different data sources, real-time multi-dimensional statistics to quickly locate the problem

Talk about the operation layer.

As mentioned earlier, the template of the operation needs to circle a group of people and then send messages, so where do these people come from?

A long time ago, the message management platform also eliminated the crowd. the general idea is that it can support file upload and hivesql upload to circle the crowd, circle it and upload it to hdfs for reading, and support the update / segmentation / export of the crowd.

With the concept of crowd, you will find that the messages you receive are actually closely related to you (not blindly pushed to you, you are inside, can circle you). It may be because you have been looking at the dress for several days, so the news of pushing the dress to you attracts you to buy it.

Later, due to the rise of the company's internal DMP system, the crowd was managed by DMP. But the idea of implementation is also similar, but it is still the same: what others do is a platform, and the function must be much better than being able to write a few interfaces.

When doing a push, it is inevitable to send the wrong message, especially on the operational side (pushing tens of thousands of people every minute). What measures does our platform do to avoid this problem as much as possible?

After the operation delineates the crowd, we will have a separate test function to "test whether a single user can send messages normally and whether there is a problem with the copywriting link."

This step must be done, and the message sent to the user must first be verified by its own. If you confirm that there is no problem with the link and copywriting, you can submit the task and send it only after the work order is approved.

If you find that there is a problem with the copy / link after startup, you can also intercept the remaining outstanding messages.

For (technical push), we have configured a "whitelist" in the pre-sent environment to receive the message.

Online messages have "de-duplicated" logic:

Filter out duplicate messages within a certain period of time

Operational message push (the way to circle the crowd to send messages) the same user can only send messages after a period of time.

Although we have made a lot of rules to avoid accidents as far as possible, we have to say that push is still an accident-prone function.

This is the end of the content of "what is the principle of the Java implementation of the message management platform". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.