Shulou(Shulou.com)06/03 Report--

Overview of Alipay system Architecture big data

Typical processing default big data

Big data, a fund processing platform

Financial accountant big data

Payment and settlement big data

Big data of Accounting Center

Trade big data

Flexible transaction

Alipay's open source distributed messaging middleware-- Metamorphosis (MetaQ)

Metamorphosis (MetaQ) is a high-performance, highly available and scalable distributed message middleware, similar to LinkedIn's Kafka, with the characteristics of message storage sequential write, large throughput and support for local and XA transactions. It is suitable for large throughput, sequential message, broadcast and log data transmission scenarios. It is widely used in Taobao and Alipay.

Metamorphosis is an open source Java message middleware on Taobao. With regard to messaging middleware, you should have heard of the JMS specification, as well as some open source implementations such as ActiveMQ and HornetQ. Metamorphosis is one of them.

The origin of Metamorphosis is from linkedin's open source MQ-- now transferred to apache's kafka learning, this is a very unique design of MQ system, it uses pull mechanism, rather than the general MQ push model, it makes extensive use of zookeeper to do service discovery and offset storage, its design concept I very much appreciate and agree, I strongly recommend that you read its design documents, generally speaking, the design of metamorphosis is completely consistent with it. But why do you need meta?

Briefly summarize the reasons why I rewrote meta:

1.Kafka is written by scala, I am not familiar with scala, and the development of the whole kafka community is too slow.

two。 There are some functions that kakfa does not implement, but we need: transactions, multiple offset storage, high availability scenarios (HA), etc.

Some of the features unique to 3.Meta relative to kafka:

-text protocol design, very transparent, supporting protocols like memcached stats to monitor broker

-Pure Java implementation, from communication to storage, from client to server.

-provide transaction support, including local transactions and XA distributed transactions

-support HA replication, including asynchronous replication and synchronous replication, to ensure the reliability of messages

-support for sending messages asynchronously

-failed to consume messages. Local recovery is supported.

-A variety of offset storage is supported, including database, disk and zookeeper. Custom implementations support group commit to improve data reliability and throughput.

-support message broadcast mode

-A series of supporting projects: python client, twitter storm spout, tail4j, etc.

Therefore, the improvement of meta compared to kafka is huge. Meta has been widely used in Taobao and Alipay. Now Alipay routes 12 billion messages via meta every day, and Taobao also has hundreds of millions of messages every day.

Suitable applications for Meta:

-Log transfer. High-throughput log transfer is a strong point of kafka.

-message broadcasting functions, such as invalid broadcast cache configuration

-Sequential synchronization of data, such as mysql binlog replication

-message routing in distributed environment (broker,producer,consumer are all clustered), scenarios that require extremely high order and reliability

-other features used as a general MQ.

Overall structure:

Internal structure:

For more information, please follow the official account of Wechat: it_

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.