Shulou(Shulou.com)06/01 Report--

Being fully aware of the data changes in the database is the key point of data synchronization from MongoDB to other data services. Compared with directly querying collection to obtain data changes, streaming monitoring is much more effective and timely. This is a very powerful "responsive programming" mode. With the update of the version of MongoDB, the streaming access method will become easier to use.

Let's review it together. Before MongoDB3.6, if we wanted to listen for data changes in the MongoDB database, we usually used the mode of "listen and play back oplog" ("tail the oplog") (the oplog table will record data changes in the replication set). In a production environment, this approach ("monitor and playback oplog") is usually complex and difficult to guarantee its stability and reliability.

Change Streams and Collections

Change Streams, which has been supported since MongoDB3.6, broke the deadlock. Change Streams makes it easy to listen for changes in data. The following is an example that demonstrates the monitoring of changes to the "movieDetails" table through Node.js.

Javascriptconst MongoClient = require ("mongodb"). MongoClient;const uri = "MONGODBURL"; const client = new MongoClient (uri, {useNewUrlParser: true}); client.connect (). Then (db = > {const changeStream = client.db ("video"). Collection ("movieDetails"). Watch (); changeStream.on ("change", next = > {console.log (next);});})

The above code first connects to the data instance and establishes a change stream on the "movieDetails" table of the "video" library through the watch () function. And then through .on ("change", … An event trigger is created that listens for all changes on that change stream and calls the corresponding subsequent functions. In the above example, the change event will be printed after the change is detected. The following is an example of the output after corresponding modification in MongoDB Compass:

Javascript {_ id: {_ data: '825C51D03F0000000129295A1004E515B4338C574BA2B9603CB1C7FB3B0446645F696400645C0EC4B74B052F9E2EF0C3810004'}, operationType:' replace', clusterTime: Timestamp {_ bsontype: 'Timestamp', low_: 1, high_: 1548865599}, fullDocument: {_ id: 5c0ec4b74b052f9e2ef0c381, title:' PS I Love You', year: Awards: {wins: 2, nominations: 4, text:'2 wins & 4 nominations.'}, type: 'movie'}, ns: {db:' video', coll: 'movieDetails'}, documentKey: {_ id: 5c0ec4b74b052f9e2ef0c381}}

You can quickly find the important information about this change stream in the above return, that is, you can learn the type of change through operationType. For a complete description of the return, please see [Change Events documentation]. When listening to a collection, the value of operationType is usually insert, update, replace, delete or invalidate. The meaning of the first four types can be clearly known by name. The replace type returned above is the feedback of replace operation with collection through Compass.

When the collection we are listening to is drop, renamed, or the db to which it belongs is drop, we will see an operationType of type invalidate. At the same time, it also means it's time to shut down change stream. The rest of the above return is the details of the change, what namespace the change occurred on, what the data looks like, and when the change occurred.

The above example was generated in the MongoDB4.x version, and version 4.x adds a new _ data field compared to version 3.6. This field is a recovery token (resume token) from which the application can continue to listen after reconnecting.

Beyond Collections

If you only need to listen for a certain collection, MongoDB3.6 can meet your needs, but for those students who have previously used oplog for change monitoring, their demand is often to listen for all changes in the database in order to apply the changes to other systems. MongoDB4.0 satisfies this requirement very well. In version 4.0, we can listen for changes in several databases or entire instances (replication set or sharding). Unlike a collection in watch (), we can watch a database or an entire instance in 4. 0.

Javascriptconst MongoClient = require ("mongodb"). MongoClient;const uri = "MONGODBURL"; undefinedconst client = new MongoClient (uri, {useNewUrlParser: true}); client.connect (). Then (db = > {const changeStream = client.watch (); changeStream.on ("change", next = > {console.log (next);})

The above example will output any changes to any database, any table, and this is not all the information we have captured. Since we have broadened the scope of listening, we will also see delete events when collection is deleted, when the database is deleted, and events that rename collection.

What Next?

According to the actual need, we can choose to listen for changes in a particular collection, or all collection changes in a database, or all database and collection changes in the entire instance. It is important to note that changes to the database will not be directly monitored when a new collection is created, but we can learn indirectly from the contents of the changes.

Of course, this is not a big deal, if we want to monitor the creation of the database or collection, we can change the collection in the content to determine whether the table is a new table that has not been created before. In addition, the creation of the index will not be captured by listening because it is not for table data changes.

MongoDB4.0 brings us a new and powerful way to monitor data changes, especially for real-time change capture. We highly recommend that you try this feature. For more information on Change Stream, please refer to [Change Streams]. If you have not installed the MongoDB4.0 instance, you can also [register] in MongoDB Atlas and get the free cluster node of M0 for learning and testing.

Translator: Zhou Liyang

Director of Operation and maintenance of Teambition

Member of MongoDB Chinese Community Organizing Committee

Article navigation

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.