Shulou(Shulou.com)06/01 Report--

This article mainly introduces the JPA multi-source distributed transaction example analysis, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, let the editor take you to understand it.

Problem background

When solving the desensitization of mysql fields, combined with the desensitization component function of sharding-jdbc, in order to improve the compatibility and minimization of sql applications, the blogger proposed a field desensitization solution based on the fusion of multiple data sources (only sharding-jdbc desensitization proxy data sources are used for operations that include desensitization field tables). This solution not only solves the problem, but also brings a new problem. The transaction of the data source is independent. As I said in this article, "JPA project multi-data source mode integrates sharding-jdbc to realize data desensitization". In the context of spring, each data source corresponds to an independent transaction manager, and the default transaction manager's data source uses the data source of the business itself, so when encrypted business is used. You need to specify that the transaction manager name in the @ Transactional annotation is the transaction manager name corresponding to desensitization. There is no problem for a simple business scenario to be used in this way, but in a general business scenario, there is always an operation in which a transaction covers two data sources. At this time, it is not possible to specify which transaction manager, so. Here you need a transaction manager with multiple data sources.

XA transaction scheme

XA protocol uses 2PC (two-phase commit) to manage distributed transactions. The XA interface provides a standard interface for communication between resource managers and transaction managers. In JDBC's XA transaction-related api abstraction, the relevant interfaces are defined as follows

The XADataSource,XA protocol data source public interface XADataSource extends CommonDataSource {/ * attempts to establish a physical database connection with the given username and password. The returned connection can retrieve a {@ code XAResource} object in a distributed transaction using non-critical methods such as * / XAConnection getXAConnection () throws SQLException; / / omitting getLogWriter} XAConnectionpublic interface XAConnection extends PooledConnection {/ *, which will be used by the transaction manager to manage the transaction behavior of the {@ code XAConnection} object in distributed transactions * / javax.transaction.xa.XAResource getXAResource () throws SQLException } XAResourcepublic interface XAResource {/ * commit the global transaction * / void commit (Xid xid, boolean onePhase) throws XAException; / * * specified by xid to end the work performed on behalf of the transaction branch. The resource manager detaches the XA resource from the specified transaction branch and lets the transaction complete. * / void end (Xid xid, int flags) throws XAException; / * notify the transaction manager to ignore this xid transaction branch * / void forget (Xid xid) throws XAException; / * determine whether the same resource manager * / boolean isSameRM (XAResource xares) throws XAException; / * specify the xid transaction preparation phase * / int prepare (Xid xid) throws XAException Get a list of prepared transaction branches from the resource manager. The transaction manager calls this method during recovery, * to get a list of transaction branches that are currently in the ready or preliminary completion state. * / Xid [] recover (int flag) throws XAException; / * informs the resource manager to roll back the work done on behalf of the transaction branch. * / void rollback (Xid xid) throws XAException; / * starts working on behalf of the transaction branch specified in xid. * / void start (Xid xid, int flags) throws XAException; / / omit non-critical methods}

Compared with ordinary transaction management, JDBC's XA protocol management has one more XAResource resource manager, and the behaviors related to XA transactions (open, prepare, commit, rollback, end) are controlled by this resource manager, these are all internal behaviors of the framework, and the data sources provided at the development level have also become XADataSource. In the abstraction of JTA, UserTransaction and TransactionManager are defined. To use JTA transactions, you must first implement these two interfaces. So, if we want to use JTA+XA to control transactions from multiple data sources, take Atomikos as an example in sprign boot

Introduce Atomikos dependency org.springframework.boot spring-boot-starter-jta-atomikos

Spring boot has helped us define the XA transaction manager autoload class, such as:

Create JTA transaction Manager @ Configuration (proxyBeanMethods = false) @ EnableConfigurationProperties ({AtomikosProperties.class, JtaProperties.class}) @ ConditionalOnClass ({JtaTransactionManager.class, UserTransactionManager.class}) @ ConditionalOnMissingBean (PlatformTransactionManager.class) class AtomikosJtaConfiguration {@ Bean (initMethod = "init", destroyMethod = "shutdownWait") @ ConditionalOnMissingBean (UserTransactionService.class) UserTransactionServiceImp userTransactionService (AtomikosProperties atomikosProperties, JtaProperties jtaProperties) {Properties properties = new Properties () If (StringUtils.hasText (jtaProperties.getTransactionManagerId () {properties.setProperty ("com.atomikos.icatch.tm_unique_name", jtaProperties.getTransactionManagerId ());} properties.setProperty ("com.atomikos.icatch.log_base_dir", getLogBaseDir (jtaProperties)); properties.putAll (atomikosProperties.asProperties ()) Return new UserTransactionServiceImp (properties);} @ Bean (initMethod = "init", destroyMethod = "close") @ ConditionalOnMissingBean (TransactionManager.class) UserTransactionManager atomikosTransactionManager (UserTransactionService userTransactionService) throws Exception {UserTransactionManager manager = new UserTransactionManager (); manager.setStartupTransactionService (false); manager.setForceShutdown (true); return manager } @ Bean @ ConditionalOnMissingBean (XADataSourceWrapper.class) AtomikosXADataSourceWrapper xaDataSourceWrapper () {return new AtomikosXADataSourceWrapper ();} @ Bean JtaTransactionManager transactionManager (UserTransaction userTransaction, TransactionManager transactionManager, ObjectProvidertransactionManagerCustomizers) {JtaTransactionManager jtaTransactionManager = new JtaTransactionManager (userTransaction, transactionManager); transactionManagerCustomizers.ifAvailable ((customizers)-> customizers.customize (jtaTransactionManager)) Return jtaTransactionManager;}}

Obviously, if you want to use XA transactions, you need to provide an implementation of UserTransaction and TransactionManager. There must also be a XADataSource, and the data source of the sharding-jdbc agent is DataSource. We need to wrap the XADataSource as a normal DataSource,spring that has provided a XA data source wrapper for AtomikosXADataSourceWrapper, and has been registered in the Spring context in AtomikosJtaConfiguration, so we can directly inject the wrapper instance when customizing the data source. Then, because it is a JPA environment, when creating an EntityManagerFactory instance, we need to specify the transaction management type of JPA as JTA. To sum up, the default data source configuration for a normal business is as follows:

/ * * @ author: kl @ kailing.pub * @ date: 2020-5-18 * / @ Configuration@EnableConfigurationProperties ({JpaProperties.class, DataSourceProperties.class}) public class DataSourceConfiguration {@ Primary @ Bean public DataSource dataSource (AtomikosXADataSourceWrapper wrapper, DataSourceProperties dataSourceProperties) throws Exception {MysqlXADataSource dataSource = dataSourceProperties.initializeDataSourceBuilder (). Type (MysqlXADataSource.class). Build (); return wrapper.wrapDataSource (dataSource) } @ Primary @ Bean (initMethod = "afterPropertiesSet") public LocalContainerEntityManagerFactoryBean entityManagerFactory (JpaProperties jpaProperties, DataSource dataSource, EntityManagerFactoryBuilder factoryBuilder) {return factoryBuilder.dataSource (dataSource) .destroy (Constants.BASE_PACKAGES) .properties (jpaProperties.getProperties ()) .persistenceUnit ("default") .j ta (true) .build () } @ Bean @ Primary public EntityManager entityManager (EntityManagerFactory entityManagerFactory) {/ / SharedEntityManager instance must be created using SharedEntityManagerCreator, otherwise the transaction in SimpleJpaRepository will not take effect return SharedEntityManagerCreator.createSharedEntityManager (entityManagerFactory);}}

Sharding-jdbc encrypted data source and ordinary business data source are actually the same data source, but the data source with encryption and decryption logic needs to be represented by the encryption component of sharding-jdbc, and the processing logic of encryption and decryption is added. So the configuration is as follows:

Author: kl @ kailing.pub * @ date: 2020-5-18 * / @ Configuration@EnableConfigurationProperties ({JpaProperties.class,SpringBootEncryptRuleConfigurationProperties.class, SpringBootPropertiesConfigurationProperties.class}) public class EncryptDataSourceConfiguration {@ Bean public DataSource encryptDataSource (DataSource dataSource,SpringBootPropertiesConfigurationProperties props,SpringBootEncryptRuleConfigurationProperties encryptRule) throws SQLException {return EncryptDataSourceFactory.createDataSource (dataSource, new EncryptRuleConfigurationYamlSwapper (). Swap (encryptRule), props.getProps ()) } @ Bean (initMethod = "afterPropertiesSet") public LocalContainerEntityManagerFactoryBean encryptEntityManagerFactory (@ Qualifier ("encryptDataSource") DataSource dataSource,JpaProperties jpaProperties, EntityManagerFactoryBuilder factoryBuilder) throws SQLException {return factoryBuilder.dataSource (dataSource) .resume (Constants.BASE_PACKAGES) .properties (jpaProperties.getProperties ()) .sten ceUnit ("encryptPersistenceUnit") .jta (true) .build () } @ Bean public EntityManager encryptEntityManager (@ Qualifier ("encryptEntityManagerFactory") EntityManagerFactory entityManagerFactory) {/ / you must create a SharedEntityManager instance using SharedEntityManagerCreator, otherwise the transaction in SimpleJpaRepository will not take effect return SharedEntityManagerCreator.createSharedEntityManager (entityManagerFactory);}}

Encounter problems 1.

Connection pool exhausted-try increasing 'maxPoolSize' and/or' borrowConnectionTimeout' on the DataSourceBean.

Resolve the problem: the maximum data source connection pool initialized by the default AtomikosXADataSourceWrapper wrapper is 1, so you need to add configuration parameters such as:

Spring.jta.atomikos.datasource.max-pool-size=20

Encounter problems 2.

XAER_INVAL: Invalid arguments (or unsupported command)

Solve the problem: this is the bug for mysql to implement XA. This problem occurs only if you access the same MySQL database multiple times in the same transaction. Add the following parameters to the mysql connection url, such as:

Spring.datasource.url = jdbc:mysql://127.0.0.1:3306/xxx?pinGlobalTxToPhysicalConnection=true

Mysql XA transaction behavior

In this scenario, although there are multiple data sources, the underlying link is the same mysql database, so the XA transaction behavior is, starting with the first sql executed (not the JTA transaction begin phase), generating xid and XA START transactions, and then XA END. When the sql of the second data source is executed, it will determine whether it is the same mysql resource. If it is the same, then re-XA START RESUME it with the newly generated xid, and then XA END. In the end, although there are two DataSource in the application layer, XA COMMIT will only be called once. The start of the XAResource implemented by the mysql driver is as follows:

Public void start (Xid xid, int flags) throws XAException {StringBuilder commandBuf = new StringBuilder (MAX_COMMAND_LENGTH); commandBuf.append ("XA START"); appendXid (commandBuf, xid); switch (flags) {case TMJOIN: commandBuf.append ("JOIN"); break; case TMRESUME: commandBuf.append ("RESUME") Break; case TMNOFLAGS: / / no-op break; default: throw new XAException (XAException.XAER_INVAL);} dispatchCommand (commandBuf.toString ()); this.underlyingConnection.setInGlobalTx (true);}

The first sql execution, flags=0, takes the TMNOFLAGS logic, and the second sql execution, flags=134217728, the TMRESUME, restarts the transaction logic. The above is the real transaction logic of Mysql XA, but the blogger found that msyql xa does not support XA START RESUME, and there are many restrictions on "Mysql XA transaction restrictions", so it is best to understand the shortcomings of mysql xa when using XA transactions in mysql databases.

Chain transaction scheme

Chain transaction is not my first term. Under the Transaction package of the spring-data-common project, there is already a default implementation of ChainedTransactionManager. In the previous article, "deeply understanding the @ Transactional working principle of spring" has analyzed the transaction abstraction of Spring, which is composed of PlatformTransactionManager (transaction manager), TransactionStatus (transaction state), TransactionDefinition (transaction definition) and other forms. ChainedTransactionManager also implements PlatformTransactionManager and TransactionStatus. The implementation principle is also very simple. The collection of transaction managers is maintained within ChainedTransactionManager. The real transaction managers are arranged by agents, and the transactions in the set are operated separately when the transaction is started, committed, and rolled back. In order to achieve the unified management of multiple transactions. This solution is relatively crude and flawed. In the commit phase, if the exception does not occur in the first data source, then the previous commit will not be rolled back, so when using ChainedTransactionManager, try to put the transaction manager that may cause problems at the back of the chain (open transactions, commit transactions in reverse order). Here is only a new idea of multi-data source transaction management, which can be managed by XA and XA as much as possible.

The default data source configuration for a normal business is as follows:

/ * * @ author: kl @ kailing.pub * @ date: 2020-5-18 * / @ Configuration@EnableConfigurationProperties ({JpaProperties.class, DataSourceProperties.class}) public class DataSourceConfiguration {@ Primary @ Bean public DataSource dataSource (DataSourceProperties dataSourceProperties) {return dataSourceProperties.initializeDataSourceBuilder () .type (HikariDataSource.class) .build () } @ Primary @ Bean (initMethod = "afterPropertiesSet") public LocalContainerEntityManagerFactoryBean entityManagerFactory (JpaProperties jpaProperties, DataSource dataSource, EntityManagerFactoryBuilder factoryBuilder) {return factoryBuilder.dataSource (dataSource) .persist (Constants.BASE_PACKAGES) .properties (jpaProperties.getProperties ()) .persistenceUnit ("default") .b uild () } @ Bean @ Primary public EntityManager entityManager (EntityManagerFactory entityManagerFactory) {/ / SharedEntityManager instance must be created using SharedEntityManagerCreator, otherwise transactions in SimpleJpaRepository will not take effect return SharedEntityManagerCreator.createSharedEntityManager (entityManagerFactory);} @ Primary @ Bean public PlatformTransactionManager transactionManager (EntityManagerFactory entityManagerFactory) {JpaTransactionManager txManager = new JpaTransactionManager (); txManager.setEntityManagerFactory (entityManagerFactory); return txManager;}}

The sharding-jdbc encrypted data source is configured as follows:

Author: kl @ kailing.pub * @ date: 2020-5-18 * / @ Configuration@EnableConfigurationProperties ({JpaProperties.class,SpringBootEncryptRuleConfigurationProperties.class, SpringBootPropertiesConfigurationProperties.class}) public class EncryptDataSourceConfiguration {@ Bean public DataSource encryptDataSource (DataSource dataSource,SpringBootPropertiesConfigurationProperties props,SpringBootEncryptRuleConfigurationProperties encryptRule) throws SQLException {return EncryptDataSourceFactory.createDataSource (dataSource, new EncryptRuleConfigurationYamlSwapper (). Swap (encryptRule), props.getProps ()) } @ Bean (initMethod = "afterPropertiesSet") public LocalContainerEntityManagerFactoryBean encryptEntityManagerFactory (@ Qualifier ("encryptDataSource") DataSource dataSource,JpaProperties jpaProperties, EntityManagerFactoryBuilder factoryBuilder) throws SQLException {return factoryBuilder.dataSource (dataSource) .resume (Constants.BASE_PACKAGES) .properties (jpaProperties.getProperties ()) .sten ceUnit ("encryptPersistenceUnit") .build () } @ Bean public EntityManager encryptEntityManager (@ Qualifier ("encryptEntityManagerFactory") EntityManagerFactory entityManagerFactory) {/ / you must use SharedEntityManagerCreator to create a SharedEntityManager instance, otherwise the transaction in SimpleJpaRepository will not take effect return SharedEntityManagerCreator.createSharedEntityManager (entityManagerFactory);} @ Bean public PlatformTransactionManager chainedTransactionManager (PlatformTransactionManager transactionManager) throws SQLException {JpaTransactionManager encryptTransactionManager = new JpaTransactionManager (); encryptTransactionManager.setEntityManagerFactory (encryptEntityManagerFactory ()) / / use a chained transaction manager to wrap the real transactionManager, txManager transaction ChainedTransactionManager chainedTransactionManager = new ChainedTransactionManager (encryptTransactionManager,transactionManager); return chainedTransactionManager;}}

With this scheme, when business from multiple data sources is involved, you need to specify which transaction manager to use, such as:

PersistenceContext (unitName = "encryptPersistenceUnit") private EntityManager entityManager; @ PersistenceContext private EntityManager manager; @ Transactional (transactionManager = "chainedTransactionManager") public AccountModel save (AccountDTO dto) {AccountModel accountModel = AccountMapper.INSTANCE.dtoTo (dto); entityManager.persist (accountModel); entityManager.flush (); AccountModel accountMode2 = AccountMapper.INSTANCE.dtoTo (dto); manager.persist (accountMode2); manager.flush (); return accountModel } Thank you for reading this article carefully. I hope the article "sample Analysis of JPA Multi-source distributed transactions" shared by the editor will be helpful to you. At the same time, I also hope that you will support and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.