Shulou(Shulou.com)05/31 Report--

This article is to share with you about the output operation on DStreams, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Dstream.foreachRDD is a powerful primitive for sending data to external systems. However, it is very important to understand how to use this primitive correctly and effectively. The following points describe how to avoid general errors.

Frequently writing data to an external system requires establishing a connection object (such as a TCP connection to a remote server) and using it to send data to the remote system. To achieve this, the developer may inadvertently create a connection object in the Spark driver, but try to call the connection object in Spark worker to save the record to RDD, as follows:

Dstream.foreachRDD (rdd = > {val connection = createNewConnection () / / executed at the driver rdd.foreach (record = > {connection.send (record) / / executed at the worker})})

This is incorrect because it requires serializing the connection object and then sending it from driver to worker. Such connection objects cannot be transferred between machines. It may appear as a serialization error (the connection object is not serializable) or an initialization error (the connection object should be initialized in worker), and so on. The correct solution is to create a connection object in worker.

However, this results in another common mistake-creating a connection object for each record. For example:

Dstream.foreachRDD (rdd = > {rdd.foreach (record = > {val connection = createNewConnection () connection.send (record) connection.close ()})})

In general, creating a connection object has the cost of resources and time. Therefore, creating and destroying connection objects for each record results in a very high cost, significantly reducing the overall throughput of the system. A better solution is to use the rdd.foreachPartition method. Create a connection object for RDD's partition and use these two objects to send all records in partition.

Dstream.foreachRDD (rdd = > {rdd.foreachPartition (partitionOfRecords = > {val connection = createNewConnection () partitionOfRecords.foreach (record = > connection.send (record)) connection.close ()}))

This allocates the cost of creating the connection object over all records in partition.

Finally, further optimizations can be made by reusing join objects between multiple RDD or batch data. Developers can maintain a static pool of connected objects and reuse objects in the pool to push batches of RDD to external systems to further save money.

Dstream.foreachRDD (rdd = > {rdd.foreachPartition (partitionOfRecords = > {/ / ConnectionPool is a static, lazily initialized pool of connections val connection = ConnectionPool.getConnection () partitionOfRecords.foreach (record = > connection.send (record)) ConnectionPool.returnConnection (connection) / / return to the pool for future reuse})})

It is important to note that connection objects in the pool should be delayed as needed and automatically time out after a period of idle time. This provides the most efficient way to generate data to an external system.

Other points to pay attention to:

The output operation manipulates DStreams by lazy execution, just as RDD action manipulates RDD by lazy execution. Specifically, RDD actions and DStreams output operations receive data processing. Therefore, if your application does not have any output operation or is used for the output operation dstream.foreachRDD (), but there is no RDD action operation in dstream.foreachRDD (), nothing will be performed. The system simply receives inputs and discards them.

By default, DStreams output operations are performed in time-sharing, and they are executed sequentially in the order defined by the application.

The above is what the output operation on DStreams looks like, and the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.