Shulou(Shulou.com)05/31 Report--

This article will explain in detail about the use of the framework or independent development and sharding implementation level considerations in the DB sub-table, the content of the article is of high quality, so the editor will share it for you to do a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

When the team combs the system business and database carefully and determines the sharding scheme, the next problem is how to realize the sharding scheme. At present, there are many open source frameworks and products for reference in sharding. At the same time, many teams will choose to develop and implement independently, and whether they choose the framework or independent development, they will face the problem of implementing sharding logic on which layer. The editor will analyze and consider this series of problems one by one.

I. the implementation level of sharding logic.

From the perspective of program architecture of a system, sharding logic can be implemented in four levels: DAO layer, JDBC API layer, Spring data access encapsulation layer between DAO and JDBC (template of various spring) and sharding proxy server between application server and database.

Figure 1. Sharding implementation level and related frameworks / products

Implemented in the DAO layer

When the team decides to implement sharding on its own, the DAO layer may be the * * location embedded in the sharding logic, because at this level, each DAO method clearly knows the data tables and query parameters that need to be accessed, and with this information can be directly located to the target shard, rather than having to parse the SQL and then route it according to the configured rules as the framework does. Another advantage is that it is not constrained by the ORM framework. Because most applications rely on some kind of ORM framework on the data access layer, and most shrading frameworks are often unable to support or can only support one orm framework, which makes the selection and application of a framework is greatly restricted, while self-implemented sharding has no such problems at all, and even different shard can work together using different orm frameworks. For example, today's java applications mostly use hibernate, but there is no very satisfactory hibernate-based sharding framework (hibernate hards will be described below), so many teams will choose to implement sharding on their own.

To sum up, the advantages of self-implementing sharding in the DAO layer are: it is not restricted by the ORM framework, it is relatively simple to implement, it is easy to customize flexibly according to the characteristics of the system, there is no need for SQL parsing and routing rule matching, and the performance will be slightly better. The disadvantage is that there is a certain technical threshold, the workload is larger than relying on the framework implementation (conversely, the framework will have learning costs), is not universal, and can only work in specific systems. Of course, in the DAO layer, sharding logic can also be extracted "externally" through XML configuration or annotations to form a general framework. However, there is no such framework yet.

Implemented in the ORM framework layer

There are two directions to implement sharding in the ORM framework layer. One is to provide sharding support on the premise of implementing Omurr Mapping, so it is positioned as a distributed data access framework. The representative of this type of framework is guzz. The other direction is to add the sharding mechanism by modifying and enhancing the existing ORM framework. The representative product of this type is hibernate shard. It should be said that with the mainstream position of hibernate, the industry's demand for a hibernate-oriented sharding framework is very urgent, but according to the current hibernate shards, the performance is not satisfactory, mainly because it has too many restrictions on the use of hibernate, for example, its support for HQL is very limited. In the aspect of mybatis, there is no mature framework at present. Some people propose to use the plug-in mechanism of mybatis to implement sharding, but unfortunately, the plug-in mechanism of mybatis can not control the connection level of multiple data sources, on the other hand, leaving the plug-in layer loses the opportunity to centrally parse and route sql, so there is no framework for reference on the mybatis framework, and the team may have to work on the DAO layer or Spring template classes.

Implemented in the JDBC API layer

The JDBC API layer is a place that many people will think of to implement sharding. If we can provide a JDBC API implementation that implements sharding logic, then sharding is completely transparent to the whole application, and such an implementation can be directly used as a general sharding product. But the technical threshold and workload of this solution is obviously beyond the average team's ability to do, so basically no team will implement sharding at this level, and there are even no such open source products. The only commercial product I know, dbShards, uses this scheme.

Implementation in Spring data access wrapper layer between DAO and JDBC

Today, with the popularity of springd, almost no application built on the java platform does not use spring. Between DAO and JDBC, spring provides a variety of template to manage the creation and release of resources and synchronization with transactions. Most spring-based applications will use template classes as the entry for data access, which gives us another opportunity to embed sharding logic, that is, to complete sharding work by providing a template class with embedded sharding logic. This scheme is basically consistent with the scheme based on JDBC API in effect, and is also transparent to the upper code, and can be smoothly overdone in the transformation of sharding, but its implementation is simpler than that based on JDBC API, so it has become the choice of many frameworks. Cobar Client of Ali Group Research Institute is an implementation of this kind of scheme.

Implemented by proxy between the application server and the database

When an agent is added between the application server and the database, the data request sent by the application to the data will first pass through the agent, and the agent will parse the SQL according to the configured routing rules and then route to the target shard. Because this scheme is completely transparent to the application and has good versatility, it has become the choice of many sharding products. The more well-known products in this regard are the official agent tool of mysql: Mysql Proxy and a product developed by Chinese people: amoeba. Mysql proxy itself does not implement any sharding logic, but as an agent for mysql databases, it provides developers with a place to embed sharding logic, using lua as the programming language, which is a problem for many teams to consider. Amoeba is a proxy product that specializes in read-write separation and sharding. It is very easy to use, does not use any programming language, and only needs to be configured through xml. However, amoeba does not support transactions (when a request from an application containing transaction information reaches amoeba, the transaction information is erased, so there will be no transaction even for a single point of data access). Of course, it depends on the positioning and design concept of the product. We can only say that amoeba is not suitable for systems that require very high transaction requirements.

Second, use the framework or develop it independently?

Many open source frameworks and products have been listed in the previous discussion. Here are MySQL Proxy and Amoeba based on proxy, Hibernate Shards based on Hibernate framework, and Cobar Client by rewriting the ibatis template class of spring. Each of these frameworks has its own advantages and disadvantages. Architects can choose according to the actual situation of the project after in-depth research, but generally speaking, I am cautious about the choice of framework. On the one hand, most frameworks lack the verification of successful cases, and their maturity and stability are questionable. On the other hand, whether some open source frameworks from successful commercial products (such as some open source projects from Ali and Taobao) are suitable for you requires in-depth research and analysis by architects. Of course, the final choice must be based on comprehensive factors such as project characteristics, team status, technical threshold and learning cost.

On the DB sub-library sub-table about the use of the framework or independent development and sharding implementation level considerations are shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.