Shulou(Shulou.com)06/02 Report--

This article introduces why MongoDB is 7 times slower than PostgreSQL when operating list (). The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

Recently, there is a need to migrate a LAN Web data platform online, and by the way, the PostgreSQL used by the local service will also be replaced with the MongoDB of a domestic cloud. Because of the Django selected by the previous Web framework, in order to make minimal changes to the code and connect the existing databases and data on MongoDB, after a comparison (Djongo and MongoEngine), MongoEngine is selected as an object document mapping (ODM) module to replace the original ORM of Django. Compared with the MongoEngine document, the first page migration is complete. Open it quickly. If you don't look at it, you will cry, and the page will be loaded with 1,2,3,4,5, 6, 7, 8, and so on. 27 seconds.

(online MongoDB service)

On the previous system, it only takes a few seconds to open this page:

(local PostgreSQL service)

This is also good, online is not a disgrace to yourself, quickly check the test.

On this page, the main data operation is to query the original data from the database, then load it into the DataFrame of Pandas, then process all kinds of data in Pandas, and finally return the JSON data to the front end for rendering. The total amount of data in the database is nearly 5 million, and the final query result is about 30, 000. After testing each link, it is found that the main time is spent in the process of reading data from the database and then loading it into Pandas. So the process is specifically extracted and tested separately for the time consumed. It turns out that it takes dozens of seconds to query the data using MongoEngine and then load it into Pandas:

The 21-second time is probably due to the poor performance of MongoEngine after encapsulating PyMongo, so test it directly with PyMongo: sure enough, after one layer of encapsulation, MongoEngine is not as efficient as PyMongo, but it takes 14 seconds to use PyMongo directly. The Django ORM test using PostgreSQL locally as the database backend took only 3 seconds:

Making the data query results processed with list () is a regular pre-operation to load it into P andas, which is equivalent to traversing the results of the query set and adding each piece of data to a list. Similar to the following:

Res = [] for i in xx: res.append (I) is just a list () operation, which makes MongoDB so much slower than PostgreSQL. What's wrong with me?

On why the list () operation MongoDB is 7 times slower than PostgreSQL to share here, I hope the above content can be of some help to you, you can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.