Shulou(Shulou.com)05/31 Report--

What is the application of MongoDB? I believe many inexperienced people don't know what to do about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

Preface

Chance to see mongo Chinese community held an essay activity, I feel very interesting, although I am still on the way to become a boss, but it is not impossible to participate. So there is this article.

The activity has stipulated the framework of topic selection. I thought about it for a while and felt that it was from 0 to 1 +.

Web app

The scope is too wide. However, it is true that MongoDB is widely used in web applications. Web applications require high expansibility, flexible and rich queries, dynamically adding fields, and so on.

Agile development

The main emphasis here is that the lack of a fixed schema makes it very suitable for agile development methodology.

Analytical and logging

Capped collection is suitable for log data, but I don't see much analytical data. Could it be better than dedicated OLAP db?

Caching

Variable schema

Script building

Personal advice is to write a script to build a cluster, which is convenient to generate with one click, rather than typing one command at a time. My script is for reference [7]

Later I'll talk about debugging the kernel with gdb on the basis of quickly creating a custom cluster.

Distributed concept and principle

This field is too big!

MongoDB is a distributed database, compared with a stand-alone database, there is a network distance between nodes, so all kinds of unreliable things will happen (google "8 common fallacies in distributed systems"). According to the actual situation of mongo, I will tell you a little bit about _ ^ ^ _ more interesting materials must refer to DDIA [6], which is by far the most popular version.

I talk very simply in my own language from three aspects: the background, why I need it, and how Mongo does it. The vocabulary I talk about suggests readers to google a lot.

Consensus Agreement (Consensus)

Background

The simple understanding is to reach agreement in many ways. Anyone who has a little contact with this piece knows that raft, created by Stanford professor John Ousterhout and his doctoral student Diego Ongaro, has been applied to a variety of distributed databases such as TiDB and PolarDB.

Of course, there are other protocols in the industry, Lamport's Paxos (applied to Chubby) and Zookeeper's ZAB,MongoDB 's pv1.

Why do you need it?

To put it simply, when multiple nodes make a decision together, if you say yours and I say mine, how can you decide? It's like a group of people having a meeting in the room, talking all the time, but there is no unity, and in the end, the meeting can only be held in vain. By the same token, in a distributed system, we need a set of rules for nodes to agree on events and results in order to work properly. This is actually very consistent with the real-world model.

How did Mongo do it?

Mongo uses MongoDB pv1, which is a kind of raft protocol, but it has a rich extension, such as rs.conf () can configure the priority,hidden, vote and other attributes of each node, which has great flexibility; added PreVote, DryRun and other actions. For details, readers can refer to the relevant documentation.

Isolation level / consistency (ACID/Consistency/CAP)

Background

These concepts are similar, so they are put together. It seems that we generally don't talk about ACID in distributed systems, which is a common word in stand-alone relational databases, and C in this is not the same thing as Consistency in distributed systems!

CAP is a term proposed by Brewer in 1992, and it is not recommended in many papers because of its ambiguity.

In many papers, there are many words that are consistent with each other, such as

-causal consistency, causal consistency, in Mongo

-linearizability, linear consistency, for single object, always read the latest data

-serializability, serialization, emphasizing that multiple transactions operate multiple object, which is the strongest isolation level in relational db.

-strict serializability,linearizability + serializability, mentioned in google spanner

-sequence consistency: sequence consistency, which is weaker than linearizability, for example, x86 CPU default consistency is it. We often see `std::memory_order_ seq` in C++ Memory Model

In terms of data security, to ensure persistence, the common skill is to do checkpoint on a regular basis and have write-ahead log, which is natively supported in the WiredTiger engine layer.

Why do you need it?

Whenever there are copies and reads and writes, there must be the question of whether you can read the latest data, which belongs to the question of consistency. Some businesses require that the newly written data must be read, this is strongconsistency, but some businesses do not require that the database can let go of this strong constraint, so there is a final consistent eventualconsistency, which means that given a certain period of time, all replicas will eventually be the same. This implementation is much less complex than strong consistency.

How did Mongo do it?

With regard to consistency, I have to talk about my long-standing misunderstanding. It turns out that the quorom in mongo is not the kind of quorum we often talk about!

Before in-depth understanding of Cassandra, and its C++ product Scylladb, their prototype is amazon's Dynamo, the paper talked about the quorum model: when there are N nodes, if you write most of them, that is, W > N _ Scylladb 2, and most of them > N _ quorum 2, you must be able to read the newly written data. However, although mongo also has the theory of majority, its connotation is completely different.

When writing mongo, the client can only write master, not slave, which is different from the leader-less system (no master system, each node is peer-to-peer). The slave pulls data from the master; both master and slave nodes are maintaining a point in time of majoritycommitted, which will move forward when writing has reached most of the time.

When the client specifies readConcern: majority, whether it can be read successfully depends on whether the time point at which the operation was initiated is after the majoritycommitted time point. If so, the majority read is successful.

Mongo transactions support snapshot isolation, the point at which the transaction is readable and most recently stable, which may be old data, but it is consistent with other data, thus avoiding read-write conflicts.

Replication and failure redundancy (Replication, Fault Tolerance)

Background

In distributed systems, replication is an important and conventional means to improve availability. In a complex distributed environment, there are always individual components that will collapse, get stuck, and do not respond. In order not to affect the user's request, it is necessary to transfer the request to a normal node, so there must be multiple copies of data. Otherwise, how do you access the previously accessed data?

Fault redundancy is a classic concept, distributed faults are strange, software, hardware, man-made; in a typical single-master system, if the master node does not have, it will affect the user's reading and writing. therefore, in the very short time when the previous master node is gone, there must be a new master to replace it, and when it is perfect, the user will not feel cut-off at all.

Why do you need it?

As mentioned earlier, ensure system availability and data security.

How did Mongo do it?

Mongo is a single master system, so it can only write to the master node, so it has an election mechanism and relies on the raft-like protocol mentioned above. This is to ensure fault redundancy.

In terms of replication, the slave node pulling oplog,oplog from the master node can be understood as the log in the raft, which reflects the mutation of the master node. If the slave node apply this locally, it can achieve the same state as the master node.

For a very detailed description, see the official source code [12].

Kernel

Personal contact with the core is not long, here to throw a brick to attract jade.

The kernel is actually divided into Server layer and Storage Engine layer. As the Server contact is incomplete, we will only talk about the engine layer for the time being.

Storage engine

Here is a document generated by doxygen [11], which is worth reading.

Engine layer technology is the core technology of database system, which involves the implementation of the core principles of database. First of all, we have to understand that data can be organized in many ways, which one is better, I am afraid it is impossible to say before the code is implemented.

Obviously here we need to plug and unplug features, the database layer (that is, dry sql,cql, query optimization, execution plan, etc.) can be flexibly connected to a variety of storage engines, so that in the end, we will know who is good and who is bad. Therefore, the engine layer must be very independent, providing the most primitive interface for the upper layer to call, which is also the perfect embodiment of the idea of computer layering in the database field.

The MongoDB engine has been WiredTiger since 3.x, and officials don't seem to have considered putting RocksDB-compatible code in it, so MongoRocks is a third-party existence; there is, of course, an in-memory engine.

WiredTiger

Here it is referred to as WT [8]. WT was originally founded by a tycoon Michael Cahill, but it was acquired by MongoDB one year and has been mongo's default storage engine ever since. We can see the basic introduction of WT here [2]. It's very rich, and you can check it out more.

First of all, WT is a kv storage engine, which is the same as Rocksdb in category, but it is much less famous, probably because it is relatively small, it seems that it is only used by mongo, and the code does not look easy to read.

There is also a lot of discussion on the Internet that the engine index implementation is B tree, not B + tree. As for why B tree is used, as far as I know:

1.mongo focuses on improving point query performance rather than range query, so unlike B + tree, you don't have to go to leaf nodes to get data every time, on average, take a shorter path.

two。 Optimize the scenario of reading more and writing less.

3. Others.

The use of WT API

WT is used in mongo, but there are only a few basic calls:

1. Create a connection conn

Wiredtiger_open (home, NULL, "create,cache_size=**, transaction_sync=**, checkpoint_sync=**,...", & conn)

This needs to be called at startup to generate a WT_CONN pointing to db as a private member of WiredTigerKVEngine.

two。 Create session

All operations in mongo have a session context, and the session in the document actually corresponds to the WT_SESSION in the engine layer. In the code, in order to make efficient use of session, there is a sessionCache to use, so you don't have to go to open every time.

Conn- > open_session (conn,NULL, "isolation=**", & session)

3. Create a table / index

When the mongo layer executes createCollection/createIndex, there are:

Sesssion- > create (session, "table::access", "key_format=S,value_format=S"))

4. Create a cursor on session

Session- > open_cursor (session, "table:mytable", NULL,NULL,&cursor)

5. When transactions are supported, start transactions on session

Session- > begin_transaction (session, "isolation=**, read_timestamp=**,sync=**,...")

6. Use cursor set/get key/value

The BSON that users see in the json,mongo server layer is actually converted to (key, value) pair at the bottom.

Cursor- > set_key (cursor, "key")

Cursor- > set_value (cursor, "value")

Cursor- > update (cursor)

7. Commit / roll back the transaction

Session- > commit_transaction (session, "commit_timestamp=**, durable_timestamp=**, sync=**,...")

Session- > rollback_transaction (session,NULL)

For the above steps, I would like to clarify a few points:

WT API calls are like that style, and it is particularly obvious that there will be a parameter char* config in which various configuration parameters are specified in the format of adirecb. Although it is quite primitive.

The parameters related to timestamps are complex and require in-depth documentation.

The meaning of the parameter still has to refer to [2].

Timestamp mechanism

From the official documentation and video [14], the introduction of logical session from 3.6and the addition of timestamp field to WT's update structure are gradually paving the way for supporting transactions and distributed transactions.

I've been exposed to some of the concepts of WT timestamps to familiarize myself with MongoRocks's support for transactions, and I can't yet systematically discuss how timestamps work between them. There can be a lot of reference in this respect [2], and I will not talk about it here.

MongoRocks

Listening to the name, you can guess that it has something to do with rocksdb, and it's natural to think of it. Since the bottom layer is connected to kv engine,rocksdb and KV, it can be connected completely, just like MyRocks. Look at the source code [3] stars also has 300 versions, originally implemented by developer Igor Canadi and other MongoRocks versions of 3.2,3.4. The project was put on hold for a while. A few months ago, Igor Canadi accepted the MR of wolfkdy to MongoRocks 4.0 [16], in which I participated in the relevant PR submission as [4].

The implementation of the 4.0 mongo-rocks driver layer is mainly focused on the transaction part. As Igor said, after 3.6.x, the internal transactions of mongo have a large jump, if the correct implementation of version 4.0 requires a large part of effort [5].

MongoRocks 4.0has just been released, so it will take more time to stabilize. For example, I found that there was a hole in oplog reading [13], which has been fixed by the author [15]. Personally, I still look forward to Rocksdb being able to connect to Mongo. I believe there will be a brighter spot than WT! Individuals should invest more time in this area and look forward to more domestic developers to join us!

Kernel gdb debugging

Large code, if you use gdb to step to learn is certainly not good, single step is only suitable for debugging bug. Why am I talking about gdb debugging here? Get runtimepath!

I have always thought that to get a large C++ project, in addition to staring at the code for half a day to understand code flow, using gdb bt is a sharp weapon! Add a breakpoint on the server side, and the client sends a command, and then a bt immediately knows the core path of server, which is very convenient!

Highlight: please use > = 8.x version of gdb. The advantage is that bt has its own color display, which looks much more comfortable than before.

Let's talk about how to use it.

First, start a copy set or sharding cluster (depending on which one you follow), and make the following settings for the master:

Cfg=rs.conf (); cfg.settings.heartbeatTimeoutSecs=3600; cfg.settings.electionTimeoutMillis=3600000;rs.reconfig (cfg)

Let's assume that we are going to debug the Lord. To prevent failover within the default time when debugging, increase the heartbeat,election timeout so that the master is still the master all the time (of course, if you want to debug the code of the master-slave code, don't do so)

When we want to see the request path of the insert command

Just take a look at the code and search for the insert keyword. I believe it is not difficult to find words like CmdInsert. If you take a closer look, you can see that it inherits a base class, and it also has a run method, so developers with feelings can actually guess: it is very likely that when server receives an insert request, run will be called!

So we can add a breakpoint at run, or we can find the word insertRecords in grep, which makes it more likely that we went here when inserting the document, so we have this:

Can continue enter, this path from libc.so start_thread to run, to insertRecords is very long, this section of the path is enough for us to analyze how to go.

Similarly, for find, update,delete is a similar approach.

For transaction operations, you can go to grep transaction, and you will also find functions that can be used as breakpoints. When you encounter begin_transaction,commit_transaction, rollback_transaction is actually a very familiar function name, which is suitable to add breakpoints.

Conclusion

A brief talk on MongoDB technology here, the amount of knowledge in this area is so huge that it can not be explained by an article. For myself, the connotation itself is fascinating, because it is a database, it is a distributed system, and it still has many defects. Although Mongo has officially tightened its agreement, some cloud vendors can no longer play high-end versions. But I think it's still a relief for engineers as long as it's open source and as long as the code is real. From shallow to deep, from now on!

After reading the above, have you mastered the application of MongoDB? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.