Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to access the Apache Kafka server". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's ideas to study and learn "how to access the Apache Kafka server".

Dealmaking engine (Matching/Trading Engine), as its name implies, is a software used to match deals, which is widely used in finance, securities, cryptocurrency transactions and other fields. The transaction engine is responsible for managing all open order (Open Orders) in the encrypted asset market and automatically executing the transaction when a matching order pair (Trading Pair) is found. This article will first introduce the basic concepts of encrypted asset dealmaking engine, such as entrustment order, transaction entrustment ledger, etc., and then use Golang to implement a rationale matchmaking engine.

1. Basic concepts and terminology

Before we start to build a dealmaking engine, let's first familiarize ourselves with the basic concepts and terminology.

Matchmaking / transaction engine

As mentioned earlier, the dealmaking engine is the software used to make deals. You can first think of the dealmaking engine as a black box, which has some inputs and outputs.

For example, possible inputs include:

Create a new order (NewOrder): a new order can be used as input to the dealmaking engine, which will try to match it with an existing order.

Cancel an existing CancelOrder: the user can also cancel a previously entered order, that is, an open order if it has not been executed.

Of course, you can define other inputs, but for the sake of simplicity, we now define only the above two inputs.

The output of the dealmaking engine is some events in order to notify other applications for processing in a timely manner. For example, when the engine matches a transaction, it triggers a TradesGenerated event, and when an existing commission order is cancelled, the engine triggers rderCancelled. Similarly, you can define the output of the engine according to your own requirements, so let's keep it simple and define only these two output events.

Transaction entrustment book

The transaction entrustment book (Order Book) is a list of buyer's orders or buyer's orders, usually sorted by price and time.

When a new buyer (buyer) consignment order enters the engine, the engine will try to match it with the existing seller (buyer) entrustment book to see if it is possible to execute the transaction. If a matching counterparty order is found, the engine can execute the two commission orders, that is, the match is successful.

Entrustment order

In any transaction engine, there may be multiple types of commission orders for users to choose from. Common types include:

Price limit order

Limit order is the most commonly used type of commission in the current cryptocurrency trading environment. This entrustment allows the user to specify a price and execute the transaction only when the matchmaking engine finds an counterparty order with the same or better price.

For a buyer's order, this means that if your commission price is ¥100, the order will be sold at any price not higher than ¥100-a specified price or a lower price; for a seller's order, the same price means that the order will be sold at any price not less than ¥100-sell at the specified price or higher.

Market price entrustment order

The matchmaking of market order will completely ignore the price factor and focus on the limited completion of a specified number of transactions. The market price order has a higher priority in the transaction commission book, and the market price order can guarantee the transaction in the market with sufficient liquidity.

For example, when a user entrusts to buy two etheric coins, the order can be traded at ¥900,1000, ¥2000 or any other price, depending on the current exposure order in the market.

Stop-loss order

The stop-loss order is not activated until the market price reaches the specified price, so it is executed in the opposite way to the market order. Once stop-loss orders are activated, they can be automatically converted into market-price orders or limit orders.

If you want to build an advanced exchange, there are other concepts you need to understand, such as liquidity, long / short trading, FIX/FAST protocols, etc., but also for the sake of simplicity, we leave them to you to discover for yourself.

2. System architecture

Now that we have some understanding of the composition of the dealmaking engine, let's take a look at the architecture of the system and the technologies we are going to use:

As you can see above, our system will contain multiple clients of the engine, which can be other components of the exchange system, such as App that receives end-user delegation requests, and so on.

The communication between the client and the engine is implemented using Apache Kafka as the message bus, and each transaction pair corresponds to a topic of Kafka, so we can ensure that when the message queue receives the user order, the engine will process the order in the same order. This ensures that we can rebuild the transaction entrustment book even if the engine crashes and restarts.

The engine listens for Kafka topics, executes delegated account commands, and publishes the engine's output events to the message queue. Of course, it would be cooler to monitor the processing speed of the order and the execution of the transaction. We can use Prometheus to collect performance metrics and grafana to implement a monitoring dashboard.

3. Development language selection

You can choose the development language you are familiar with, but because the dealmaking engine has a large amount of computation, we should usually choose the underlying series of languages, such as Calgary +, GoLang, Rust, Java and so on. In this tutorial, we use Golang because it is fast, easy to understand, simple to implement concurrently, and I haven't used C++ for a long time.

4. Develop the dealmaking engine

We will follow these steps to develop the dealmaking engine:

Basic type definition

Consumer implementation

Order Book implementation

Producer implementation

Monitoring implementation

4.1 basic type definition

We need to first define some basic types, including Order, OrderBook, and Trade, which represent entrustment orders, transaction delegation books, and transactions, respectively:

The following is the contents of the engine/order.go file:

Package engineimport "encoding/json" type Order struct {Amount uint64 `json: "amount" `Price uint64 `json: "price" `ID string `json: "id" `Side int8 `json: "side" `} func (order * Order) FromJSON (msg [] byte) error {return json.Unmarshal (msg, order)} func (order * Order) ToJSON () [] byte {str, _: = json.Marshal (order) return str}

Here we simply create a structure to record the main information of the order, and then add a method for fast JSON transformation.

Similarly, the contents of the engine/trade.go file:

Package engineimport "encoding/json" type Trade struct {TakerOrderID string `json: "taker_order_id" `MakerOrderID string `json: "maker_order_id" `Amount uint64 `json: "amount" `Price uint64 `json: "price" `} func (trade * Trade) FromJSON (msg [] byte) error {return json.Unmarshal (msg, trade)} func (trade * Trade) ToJSON () [] byte {str _: = json.Marshal (trade) return str}

Now that we have defined the basic input and output types, let's look at the contents of the transaction delegation book engine/order_book.go file:

Package engine// OrderBook typetype OrderBook struct {BuyOrders [] Order SellOrders [] Order} / / Add a buy order to the order bookfunc (book * OrderBook) addBuyOrder (order Order) {n: = len (book.BuyOrders) var i int for i: = n-1; I > = 0; imuri-{buyOrder: = book.BuyOrders [I] if buyOrder.Price

< order.Price { break } } if i == n-1 { book.BuyOrders = append(book.BuyOrders, order) } else { copy(book.BuyOrders[i+1:], book.BuyOrders[i:]) book.BuyOrders[i] = order }}// Add a sell order to the order bookfunc (book *OrderBook) addSellOrder(order Order) { n := len(book.SellOrders) var i int for i := n - 1; i >

= 0 I sellOrder-{sellOrder: = book.SellOrders [I] if sellOrder.Price > order.Price {break}} if I = = n copy 1 {book.SellOrders = append (book.SellOrders, order)} else {copy (book.SellOrders [I + 1:] Book.SellOrders [I:]) book.SellOrders [I] = order}} / / Remove a buy order from the order book at a given indexfunc (book * OrderBook) removeBuyOrder (index int) {book.BuyOrders = append (book.BuyOrders [: index], book.BuyOrders [book.BuyOrders + 1:]...)} / / Remove a sell order from the order book at a given indexfunc (book * OrderBook) removeSellOrder (index int) {book.SellOrders = append (book.SellOrders [: index]) Book.SellOrders [index + 1:]...)}

In the transaction delegation book, in addition to creating a list of buy / seller orders, we also need to define a way to add new orders.

The list of entrustment orders should be arranged in ascending or descending order according to its type: the seller's order is arranged in descending order, so that the order with the largest serial number in the list has the lowest price; the buyer's order is arranged in ascending order, so the price of the last order in the list is the highest.

Since most transactions close to the market price, we can easily insert or remove members from these arrays.

4.2 entrust order processing

Now let's deal with the power of attorney.

In the following code, we add a command to implement the processing of the limit order.

Contents of the file engine/order_book_limit_order.go:

Package engine// Process an order and return the trades generated before adding the remaining amount to the marketfunc (book * OrderBook) Process (order Order) [] Trade {if order.Side = = 1 {return book.processLimitBuy (order)} return book.processLimitSell (order)} / / Process a limit buy orderfunc (book * OrderBook) processLimitBuy (order Order) [] Trade {trades: = make ([] Trade, 0 1) n: = len (book.SellOrders) / / check if we have at least one matching order if n! = 0 | | book.SellOrders [n-1] .Price = 0 I sellOrder-{sellOrder: = book.SellOrders [I] if sellOrder.Price > order.Price {break} / / fill the entire order if sellOrder.Amount > = order.Amount { Trades = append (trades Trade {order.ID, sellOrder.ID, order.Amount SellOrder.Price}) sellOrder.Amount-= order.Amount if sellOrder.Amount = = 0 {book.removeSellOrder (I)} return trades } / / fill a partial order and continue if sellOrder.Amount

< order.Amount { trades = append(trades, Trade{order.ID, sellOrder.ID, sellOrder.Amount, sellOrder.Price}) order.Amount -= sellOrder.Amount book.removeSellOrder(i) continue } } } // finally add the remaining order to the list book.addBuyOrder(order) return trades}// Process a limit sell orderfunc (book *OrderBook) processLimitSell(order Order) []Trade { trades := make([]Trade, 0, 1) n := len(book.BuyOrders) // check if we have at least one matching order if n != 0 || book.BuyOrders[n-1].Price >

= order.Price {/ / traverse all orders that match for i: = n-1; I > = 0; buyOrder-{buyOrder: = book.BuyOrders [I] if buyOrder.Price

< order.Price { break } // fill the entire order if buyOrder.Amount >

= order.Amount {trades = append (trades, Trade {order.ID, buyOrder.ID, order.Amount) BuyOrder.Price}) buyOrder.Amount-= order.Amount if buyOrder.Amount = = 0 {book.removeBuyOrder (I)} return trades } / / fill a partial order and continue if buyOrder.Amount < order.Amount {trades = append (trades Trade {order.ID, buyOrder.ID, buyOrder.Amount BuyOrder.Price}) order.Amount-= buyOrder.Amount book.removeBuyOrder (I) continue} / / finally add the remaining order to the list book.addSellOrder (order) return trades}

It seems that we have changed one method into two, dealing with the buyer's order and the seller's order respectively. The two methods are similar in every way, except that they deal with different market sides.

The algorithm is very simple. We match a buyer's order with all the seller's orders and find any seller's order that is the same or lower than the buyer's commission price. When this condition cannot be met, or after the buyer's order is completed, we will return to the transaction that will be brokered.

4.3 access to Kafka

Now that we are about to complete our trading engine, we still need to access the Apache Kafka server and start listening for orders.

Contents of the main.go file:

Package mainimport ("engine/engine"log"github.com/Shopify/sarama" cluster "github.com/bsm/sarama-cluster") func main () {/ / create the consumer and listen for new order messages consumer: = createConsumer () / / create the producer of trade messages producer: = createProducer () / / create the order book book: = engine.OrderBook { BuyOrders: make ([] engine.Order) 0100), SellOrders: make ([] engine.Order, 0100) } / / create a signal channel to know when we are done done: = make (chan bool) / / start processing orders go func () {for msg: = range consumer.Messages () {var order engine.Order / / decode the message order.FromJSON (msg .value) / / process the order trades: = book.Process (order) / / send trades to message queue for _ Trade: = range trades {rawTrade: = trade.ToJSON () producer.Input ()

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.