Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to achieve log reading and writing by wal". In daily operation, I believe many people have doubts about how to achieve log reading and writing in wal. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to achieve log reading and writing in wal". Next, please follow the editor to study!

Etcd raft introduction

Etcd raft is the most widely used raft library at present. Etcd raft is used in etcd, Kubernetes, Docker Swarm, Cloud Foundry Diego, CockroachDB, TiDB, Project Calico, Flannel and other distributed systems, and has been verified in the generation environment. The implementation of traditional raft library is a single design (integrated storage layer, message serialization, network layer, etc.). Etcd raft inherits the simple design concept and only implements the core raft algorithm, which is more flexible. Etcd separates the network, log storage, snapshot and other functions, through a separate module, users can call when needed. Etcd has implemented a set of raft libraries of its own: etcd-wal (for storing logs), snap (for storing snapshots), and MemoryStorage (for storing current log, snapshot, status, and other information for use by raft core programs).

Etcd wal introduction

WAL is an acronym for write ahead log. Etcd uses the wal module to persist raft logs, and all etcd implementations of wal are placed in the wal directory.

Wal data structure

Type WAL struct {

Lg * zap.Logger

Dir string / / the living directory of the underlay files

/ / dirFile is a fd for the wal directory for syncing on Rename

DirFile * os.File

Metadata [] byte / / metadata recorded at the head of each WAL

State raftpb.HardState / / hardstate recorded at the head of WAL

Start walpb.Snapshot / / snapshot to start reading

Decoder * decoder / / decoder to decode records

ReadClose func () error / / closer for decode reader

Mu sync.Mutex

Enti uint64 / / index of the last entry saved to the wal

Encoder * encoder / / encoder to encode records

Locks [] * fileutil.LockedFile / / the locked files the WAL holds (the name is increasing)

Fp * filePipeline

}

The above is the data structure of wal, and an instance of wal is obtained by using the Create () method in the wal.go file. Wal will first create a temporary directory and initialize related variables, and create and initialize the first wal file. After all the initialization is completed, directly change the name of the temporary directory to complete the initialization of the wal instance.

Document organization

All wal logs are placed in a specified directory with the file name ending with .wal and the format-.wal. Both seq and index are in 6x format, for example: 00000000000001-0000000000000001.wal. Index represents the index,seq of the first raft log in this file is the serial number of the file (incremented in turn).

The size of each file defaults to 64m, and when the file is greater than 64m, wal automatically generates a new log file to store the log. Each log file will use flock to lock the file with the parameter LOCK_EX, which is a unique lock. Only one process can manipulate the log file at a time, so when wal owns the file, the file cannot be deleted through the process.

Log logical organization

Wal logs can store many types of data, as follows.

CrcType the first record of each new log file will be a record of type crcType, and crcType will only be written at the beginning of each log file to record the last crc of the last file.

MetadataType metadataType follows the crcType record in every new log file, and each log file appears only once

The log type of stateType is added in two cases:

When the log file is split automatically, a log of stateType will be saved immediately after metadataType in the new log file.

This type of log is also stored when hard state is returned in the raft core program ready.

Only the term and index of snapshot are stored in the snapshotType wal log, and the specific data is stored in a special snapshot. Each snapshot stored will store a snapshot of wal in the wal log. When a snapshot is stored, all index log files prior to the snapshot are released. The snapshot stored in wal is mainly used to check whether the snapshot is correct.

Log reading and writing

Wal implements log reading and writing through encapsulated encoder and decoder modules.

Write a journal

The encoder module writes the incremental calculation crc and data to the wal file. The following is the encoder data structure

Type encoder struct {

Mu sync.Mutex

Bw * ioutil.PageWriter

Crc hash.Hash42

Buf [] byte / / cache space. Default is 1m, reducing the pressure on data allocation.

Uint64buf [] byte

}

Wal writes logs through the encoder implementation, and the calculation of crc is completed in this module. Wal to better manage data, each piece of data in the log is aligned with 8 bytes (wal automatically aligns bytes). The process of log writing is as follows.

The crc in the figure is an incremental calculation based on all previous log data. Wal only cares about writing logs and does not check whether the index of the log is duplicated, but if the Node is restarted, the system will automatically filter out the mixed logs.

Log segmentation

Wal implements automatic log segmentation. When the log data is greater than the default 64m, a new file will be generated and written to the log. Log segmentation is realized by the cut method in the wal.go file. The cut method triggers the call only when the Save method in wal is called, and the first record of the new file is the last crc of the previous wal file.

Log compact

Wal does not implement the automatic compact of logs, and the system only provides the log compact method of MemoryStorage (which needs to be called actively by the user).

File_pipeline module

When wal creates a new file, it always creates a new tmp file, and renames the file when all operations are completed. Wal uses the file_pipeline module to start a co-program in the background to prepare a temporary file for use, thus avoiding the overhead of temporary file creation.

Etcd snap introduction

Etcd raft comes with a go.etcd.io/etcd/etcdserver/api/snap module to store snapshots.

Document organization

In the snap module, a snapshot is stored in a file with the suffix .snap, and the file format is-.snap, term and index represent the term and index, respectively, where the snapshot log resides. The specific storage structure of each snapshot is shown below:

Detailed introduction

The system can have multiple snapshots, and the snap module uses the Snapshotter structure to uniformly manage snapshots.

Type Snapshotter struct {

Lg * zap.Logger

Dir string

}

The above snapshotter structure code, snapshotter is mainly used to store and read snapshots.

The specific storage content of the snapshot needs to be specified by the user, for example, in the official example of raft, the current kv data is directly stored in the snapshot after Marshal.

Func (s * kvstore) getSnapshot () ([] byte, error) {

S.mu.RLock ()

Defer s.mu.RUnlock ()

Return json.Marshal (s.kvStore)

}

When to take a snapshot

In etcd-raft, users can choose when to take a snapshot. In the official case of etcd, the method to take a snapshot is maybeTriggerSnapshot (), which is called when the node's Ready () method returns. When the index value of the current submission is greater than 10000 of the last large snapshot, a new snapshot will be taken.

Etcd MemoryStorage introduction

MemoryStorage is used to store temporary data of raft nodes, including entrys, snapshots, and so on. The user stores the data in memoryStorage, and the raft node uses the data as well. Including the delivery of entrys, the sending of snapshots, and so on, are all sent from memoryStorage.

/ / MemoryStorage implements the Storage interface backed by an

/ / in-memory array.

Type MemoryStorage struct {

/ / Protects access to all fields. Most methods of MemoryStorage are

/ / run on the raft goroutine, but Append () is run on an application

/ / goroutine.

Sync.Mutex

HardState pb.HardState

Snapshot pb.Snapshot

/ / ents [i] has raft log position i+snapshot.Metadata.Index

Ents [] pb.Entry

}

MemoryStorage stores the latest entrys (including those without commit), snapshots, and status, and users need to store data in memorystorage when they receive relevant data from other nodes.

At this point, the study of "how to read and write logs in wal" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.