Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to understand the life cycle of component". In daily operation, I believe many people have doubts about how to understand the life cycle of component. Xiaobian consulted all kinds of materials and sorted out simple and easy operation methods. I hope to help you answer the doubts about "how to understand the life cycle of component"! Next, please follow the small series to learn together!

Storm helps you manage the life cycle of spouts and bolts in storm. You just need to create a new instance and specify the parallel number of the component. So how to handle the lifecycle of tool classes used in topology, such as database connection pool, redis connection pool, where should these objects be initialized? This article discusses this problem using the jdbc connection pool as an example. Before reading this article, please understand how storm topology works.

Storm Component Life Cycle

Storm component refers to spouts and bolts in topoloty, which is a general term for spouts and bolts. As for the life cycle of component, I combine the answer of storm author Nathanmarz and my own understanding to describe it:

1. When the topology is submitted, instances of spouts and bolts are created locally on the machine where the storm submission is executed and serialized locally. In addition, all component constructors, the declareOutputFields method, run locally once.

The entire topology is uploaded to nimbus.

3. The corresponding supervisor will obtain the serialized topology code from nimbus and hand it to the corresponding worker for execution.

4. Each worker performs the following: deserialize the code, run prepare/open (initialize the corresponding component). For example, if a spout has 3 parallelism degrees set in the topology and 3 workers set in the topology, the open method will be called once on 3 different workers, for a total of 3 calls.

5, worker notifies executor to loop continuously to run nextTuple/execute method.

One thing to note is that the initialization of components should be performed in the prepare/open method, not when instantiating components or in the main function of topology.

Jdbc Pool initialization scheme

Referring to the lifecycle of a component, two scenarios can be thought of:

1, each component maintains a connection with the database, this scheme is suitable for write operations are very frequent, the number of components is relatively small, and there will not be a period of time without any write and read operations, otherwise mysql connection timeout will occur; In addition, you can omit the two steps of obtaining connections from the connection pool and returning to the connection pool after running out of connections. If you decide on this scenario, simply put the database configuration in config and initialize the database connection in the corresponding component.

2. Maintain a connection pool in each worker. When all executors in the worker need to use connections, they are obtained from the connection pool and returned after use. This program is generally recommended.

Let's talk about when the connection pool for the second scenario should be initialized.

First of all, in topology operation, a worker is equivalent to a jvm process, which is equivalent to N workers, and there will be N connection pools. Initializing the connection pool needs to be done on the worker side. There is only one instance of the connection pool in a worker, which is suitable for single-instance mode.

Initialization idea: Each component receives database configuration information, and initializes the connection pool when it is prepared/open. Here, it needs to ensure that it can be initialized only once in a single worker, that is, lock the initialized code to ensure that only one component can be initialized.

In the storm source package, storm-jdbc implementation is provided, which uses the JDBC connection pool component. The specific implementation idea is the second solution discussed above. The following is the section locked at initialization.

public class HikariCPConnectionProvider implements ConnectionProvider { private Map configMap; private transient HikariDataSource dataSource; public HikariCPConnectionProvider(Map hikariCPConfigMap) { this.configMap = hikariCPConfigMap; } @Override public synchronized void prepare() { if(dataSource == null) { Properties properties = new Properties(); properties.putAll(configMap); HikariConfig config = new HikariConfig(properties); this.dataSource = new HikariDataSource(config); this.dataSource.setAutoCommit(false); } } @Override public Connection getConnection() { try { return this.dataSource.getConnection(); } catch (SQLException e) { throw new RuntimeException(e); } } @Override public void cleanup() { if(dataSource != null) { dataSource.shutdown(); } }}

See that initialization prepare is locked to ensure that it is initialized only once. In fact, you can optimize it again here: don't lock the method, but lock the code of the specific initialization connection to reduce the scope of the lock effect.

At this point, the study of "how to understand the life cycle of components" is over, hoping to solve everyone's doubts. Theory and practice can better match to help everyone learn, go and try it! If you want to continue learning more relevant knowledge, please continue to pay attention to the website, Xiaobian will continue to strive to bring more practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.