Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to migrate JSON data from MongoDB to MaxCompute". In daily operation, I believe many people have doubts about how to migrate JSON data from MongoDB to MaxCompute. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts about "how to migrate JSON data from MongoDB to MaxCompute". Next, please follow the editor to study!

{"store": {"book": [{"category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95}, {"category": "fiction" "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99}, {"category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings" "isbn": "0395-19395-8", "price": 22.99}], "bicycle": {"color": "red", "price": 19.95}}, "expensive": 10}

Log in to the DMS console of MongoDB. In this example, the database is admin and the collection is userlog. You can view the uploaded data in the query window using the command db.userlog.find (). Limit (10), as shown in the following figure.

In addition, you need to create new users in the database ahead of time for DataWorks to add data sources. In this example, using the command db.createUser ({user: "bookuser", pwd: "123456", roles: ["root"]}), the new user name is bookuser, the password is 123456, and the permission is root.

Use DataWorks to extract data to MaxCompute

New MongoDB data source

Go to the DataWorks data integration console and add a new MongoDB data source.

The specific parameters are shown below. You can click to test the connectivity of the data source. Since the MongoDB in this article is in the VPC environment, the data source type needs to be a public network IP.

The access address and port number can be obtained by clicking the instance name in the MongoDB management console, as shown in the following figure.

Create a new data synchronization task

Create a new data synchronization type node on DataWorks.

At the same time, create a new table creation task in DataWorks to store JSON data. In this case, the new table is named mqdata.

The table parameters can be completed through the graphical interface. In this example, the mqdata table has only one column, of type string, and the column name is MQ data.

After completing the above new creation, you can initially configure the parameters of the data synchronization task in the graphical interface, as shown in the following figure. Select the target data source name as odps_first, and select the target table as the newly created mqdata. The data source type is MongoDB. Select the data source mongodb_userlog we just created. After completing the above configuration, click convert to script and jump to script mode.

An example of script mode code is as follows.

{"type": "job", "steps": [{"stepType": "mongodb", "parameter": {"datasource": "mongodb_userlog", / / data source name "column": [{"name": "store.bicycle.color" / / JSON field path In this example, extract the color value "type": "document.document.string" / / the number of fields in this column should be the same as name. If the JSON field you selected is a first-level field, such as expensive in this example, you can simply enter string. }], "collectionName / / Collection name": "userlog"}, "name": "Reader", "category": "reader"}, {"stepType": "odps", "parameter": {"partition": "" "isCompress": false, "truncate": true, "datasource": "odps_first", "column": [/ / MaxCompute table column name "mqdata"], "emptyAsNull": false, "table": "mqdata"} "name": "Writer", "category": "writer"}], "version": "2.0", "order": {"hops": [{"from": "Reader", "to": "Writer"}]} "setting": {"errorLimit": {"record": "}," speed ": {" concurrent ": 2," throttle ": false," dmu ": 1}

After the above configuration is completed, click run and then. An example of running the success log is shown below.

Result verification

Create a new ODPS SQL node in your business process.

You can enter the SELECT * from mqdata; statement to view the data in the current mqdata table. Of course, you can also enter a command to run this step directly in the MaxCompute client.

At this point, the study on "how to migrate JSON data from MongoDB to MaxCompute" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.