Shulou(Shulou.com)06/03 Report--

Online full and incremental backup of QEMU/KVM disk is a concrete implementation of "packaging" scheme, which can realize point-in-time backup, support both local and remote backup methods, and can specify backup files for recovery.

Tag: qemu, kvm, disk, live, backup, incremental, bitmap

Brother slow's original article, welcome to reprint.

Catalogue

▪ 1. The reason of this article

▪ 2. What can be achieved in the end

▪ 3. Environment description

▪ 4. A list of commonly used commands

▪ 5. Four ways of disk online backup

▪ 6. Query the device name corresponding to the disk

▪ 7. What is bitmap?

▪ 8. Create and query bitmap

▪ 9. Make incremental backup based on bitmap

▪ 10. Delete bitmap

▪ 11. Reset bitmap

▪ 12. Backup logic

▪ 13. Remote backup

▪ 14. Backup task management

▪ 15. Event View

▪ 16. Backup chain, data recovery, merger

▪ 17. Reference documentation

1. The reason of this article

In the underlying technology of virtualization, storage is a top priority, and no one wants to lose data. Therefore, backup is particularly important.

According to the implementation scheme of backup, it can be divided into two categories:

▷ copy: it can be implemented directly from back-end storage without the help of computing. For example, Ceph can make multiple copies, and local disk files can be synchronized in real time through DRBD technology. The replica scheme improves the overall reliability of storage.

▷ packaging: means to package and export disk data periodically. When you need to restore, you can specify backup files to restore.

The copy and packaging are complementary, the copy cannot be backed up and restored according to the point in time, and the packaging does not have the data immediately before the downtime. All the contents described in this article are "packaging".

two。 What can be achieved in the end

After investigation and testing, the final content of this paper can achieve the following major functions:

▷ online full backup, online incremental backup

▷ event View

▷ task management

▷ is unlimited in storage form, which can be local qcow2, raw, ceph rbd and other forms.

▷ remote backup

3. Environment description

All content of this article, except for special instructions, is based on the following circumstances

▷ operating system: CentOS 7.5x86x64

▷ qemu: qemu-2.12.0 based on ovirt-4.3 yum source

▷ libvirt: libvirt-4.5.0 based on centos-update yum source

4. A list of commonly used commands

Get disk device name

# View virsh qemu-monitor-command DOMAIN via qmp-- pretty'{"execute": "query-block"}'# View virsh qemu-monitor-command DOMAIN-- hmp 'info block' through hmp

-- pretty indicates that the output json is formatted in newline format, otherwise there is only one line, and this parameter and-- hmp are mutually exclusive.

Backup command

# full backup (including backing file): "full" virsh qemu-monitor-command DOMAIN'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "full", "target": "/ opt/backup/full.img"}'# full backup (excluding backing file Only backup current files): "top" virsh qemu-monitor-command DOMAIN'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "top", "target": "/ opt/backup/top.img"}'# Real-time export new IO: "none" virsh qemu-monitor-command DOMAIN'{"execute": "drive-backup" "arguments": {"device": "drive-virtio-disk0", "sync": "none", "target": "/ opt/backup/none.img"}}'# incremental backup: "bitmap" virsh qemu-monitor-command DOMAIN'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "incremental" "bitmap": "bitmap0", "target": "/ opt/backup/inc.0.qcow2"}}'# incremental backup: "bitmap" Remote export virsh qemu-monitor-command DOMAIN'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "incremental", "bitmap": "bitmap0", "target": "iscsi://192.168.1.100:3260/iqn.2019-01.com.iaas/0"}} 'via iscsi

Bitmap operation

# create a non-persistent bitmap (qemu > = 2.4) virsh qemu-monitor-command DOMAIN'{"execute": "block-dirty-bitmap-add", "arguments": {"node": "drive-virtio-disk0", "name": "bitmap0"}'# # create a persistent bitmap (qemu > = 2.10) virsh qemu-monitor-command DOMAIN'{"execute": "block-dirty-bitmap-add", "arguments": {"node": "drive-virtio-disk0" "name": "bitmapY", "persistent": true}'# Delete bitmapvirsh qemu-monitor-command DOMAIN'{"execute": "block-dirty-bitmap-remove", "arguments": {"node": "drive-virtio-disk0", "name": "bitmap0"}'# reset bitmapvirsh qemu-monitor-command DOMAIN'{"execute": "block-dirty-bitmap-clear" "arguments": {"node": "drive-virtio-disk0", "name": "bitmap0"}'# query the block information of all disks in the virtual machine Contains bitmapvirsh qemu-monitor-command DOMAIN-- pretty'{"execute": "query-block"}'# query the bitmap of the specified disk of the virtual machine (query the usage of the first disk [0], the second disk [1], and so on) virsh qemu-monitor-command DOMAIN-- pretty'{"execute": "query-block"}'| jq .return [0] | sed-n'/ dirty\-bitmaps/,/] / p'

Event monitoring

# always listen to event virsh qemu-monitor-event DOMAIN-- timestamp-- loop# stop listening to virsh qemu-monitor-event DOMAIN-- event BLOCK_JOB_COMPLETED after receiving a specific event

Backup task management

# View running task virsh qemu-monitor-command DOMAIN via qmp-- pretty'{"execute": "query-block-jobs"}'# View running task virsh qemu-monitor-command DOMAIN via hmp-- hmp 'info block-jobs'# stop backup task virsh qemu-monitor-command DOMAIN' {"execute": "block-job-cancel", "arguments": {"device": "drive-virtio-disk0" "force": true}}'# pause backup task virsh qemu-monitor-command DOMAIN'{"execute": "block-job-pause", "arguments": {"device": "drive-virtio-disk0"}}'# # restore the paused backup task virsh qemu-monitor-command DOMAIN'{"execute": "block-job-resume" "arguments": {"device": "drive-virtio-disk0"}'5. Four ways of disk online backup

▷ full: full backup, which merges and outputs all master disks (backing file) on the specified disk and its chain to the local

▷ top: only the current disk is backed up, not backing file. This item is valid only for local disks, and has the same effect on ceph rbd,top and full as full backup.

▷ none:new I job O, that is, from the execution of the backup command, whenever there is a change in IO, the changed data will be exported in real time until the virtual machine is shut down or the job is stopped manually. Only block devices in qcow2 format are supported, not raw format (note that this refers to qcow2 format, not qcow2 local disk)

▷ incremental: specify dirty bitmap for backup. At present, incremental backup of qemu refers to dirty bitmap-based backup.

The above four backups are mutually exclusive: when any backup job is running, no other type of backup can be performed and a new job for the same backup can no longer be opened.

The operation methods of full, top and none are very simple. They have been mentioned in the previous "list of commonly used commands". I will not repeat them, but incremental backup will be a little more complicated. This article will focus on incremental backup.

6. Query the device name corresponding to the disk

Since qemu does not support one-click backup of all disks, no matter how many disks there are, you must first find the disk device name to back up.

The device name can be found through qmp or hmp. Qmp is the native method of qemu, and hmp is a layer of translator added to simplify the use of qmp (where h is human, indicating friendly readability). Although hmp is easy to use, it is not sure whether it can support all qmp functions, so here is a brief demonstration of the use of hmp, and then mainly through qmp

# View virsh qemu-monitor-command DOMAIN via qmp-- pretty'{"execute": "query-block"}'# View virsh qemu-monitor-command DOMAIN-- hmp 'info block' through hmp

The drive-virtio-disk0 and drive-virtio-disk1 in the output are the device names

For more information on the detailed use of qmp and hmp, see the author's another article, "interaction between qemu virtual machines based on QMP".

Https://mp.weixin.qq.com/s/HUo0HO6Sl9xn5-FWiexXoA7. What is bitmap?

This article will focus on bitmap backup

In qemu, "incremental backup" refers to dirty bitmap-based incremental backup. Please note that two words, dirty bitmap and incremental, are mentioned here. Next, we will explain in detail what bitmap is, how to use bitmap, and what features and precautions there are. Only when we have a clear understanding of bitmap can we do a good job of incremental backup.

What is bitmap?

Online incremental backup of qemu disks can be achieved through dirty bitmap without creating a long list of snapshots

The incremental backup of qemu usually refers to the backup achieved through dirty bitmap.

Dirty bitmap is implemented by qemu and tracks the data that needs to be backed up incrementally next time.

Since ▷ v2.4, dirty bitmap is supported, but the bitmap is only recorded in the emulator memory of qemu, not in the underlying block device (the underlying block device refers to ceph rbd, local disk qcow2, etc.). This kind of bitmap is called in-memory bitmap, and the author is used to calling it non-persistent bitmap.

Starting from ▷ v2.10, qemu began to support persistent dirty bitmap, which I used to call persistent bitmap

Differences among bitmap, dirty bitmap and persistent dirty bitmap

▷ bitmap: an advanced feature supported by qemu, until the writing of this article (v3.1.0), there is only one type of bitmap, which is dirty bitmap

▷ dirty bitmap: is a type of bitmap and the only type, so dirty bitmap and bitmap are the same thing. Because dirty bitmap is recorded in memory rather than on disk, it can support any type of underlying disk, such as local, ceph rbd, as well as disk formats of any format, such as qcow2, raw, and so on.

▷ persistent dirty bitmap: an improved version of dirty bitmap, because the dirty bitmap is recorded in memory, and when the qemu virtual machine shuts down, dirty bitmap disappears, resulting in the need to make a full backup again. Persistent dirty bitmap is not supported until v2.10.

Persistent bitmap only supports qcow2 and does not support raw format (such as local raw files or ceph-rbd)

The name of the bitmap

▷ is unique to nodes, but bitmap attached to different nodes can share the same name. For the same node, the names of persistent bitmap and non-persistent bitmap cannot conflict)

The bitmap created by ▷ for internal use of qemu may be anonymous and have no name. But the user-created bitmap is certainly not anonymous. Each node can have any number of bitmap

The bitmap name created by the ▷ user cannot be empty (that is, ")

The working principle of incremental backup based on bitmap

▷ first, in the startup state of the virtual machine, type bitmap for a disk of the virtual machine (called node), then the bitmap is in qemu memory, and the count attribute of bitmap is 0

▷ second, when there is a change in disk data, bitmap will record the change, and you can see that count continues to increase (no more than disk virtual size)

▷ finally, when you specify bitmap for backup, qemu will export the corresponding incremental data to local or remote based on the bitmap record.

8. Create and query bitmap# create non-persistent bitmap (qemu > = 2.4) virsh qemu-monitor-command DOMAIN-- pretty'{"execute": "block-dirty-bitmap-add", "arguments": {"node": "drive-virtio-disk0", "name": "bitmap0"}'# create persistent bitmap (qemu > = 2.10) virsh qemu-monitor-command DOMAIN-- pretty'{"execute": "block-dirty-bitmap-add" "arguments": {"node": "drive-virtio-disk0", "name": "bitmap1" "persistent": true}}'# query bitmapvirsh qemu-monitor-command DOMAIN--pretty'{"execute": "query-block"}'# query the bitmapvirsh qemu-monitor-command DOMAIN--pretty'{"execute": "query-block"}'of the first disk of the virtual machine | jq .return [0] | sed-n'/ dirty\-bitmaps/ /] / pause # query the bitmapvirsh qemu-monitor-command DOMAIN--pretty'{"execute": "query-block"}'of the second disk of the virtual machine | jq. Return [1] | sed-n'/ dirty\-bitmaps/,/] / pause 9. Do incremental backup based on bitmap # backup virsh qemu-monitor-command DOMAIN-- pretty'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "incremental", "bitmap": "bitmap0", "target": "/ opt/backup/inc.0.qcow2"}'# during the backup process You can view the task virsh qemu-monitor-command DOMAIN-- pretty'{"execute": "query-block-jobs"}'10. Delete bitmapvirsh qemu-monitor-command DOMAIN-- pretty'{"execute": "block-dirty-bitmap-remove", "arguments": {"node": "drive-virtio-disk0", "name": "bitmap0"}'11. Reset bitmap

To reset bitmap is to set the count value of bitmap to 0 and re-track disk changes.

Virsh qemu-monitor-command DOMAIN-pretty'{"execute": "block-dirty-bitmap-clear", "arguments": {"node": "drive-virtio-disk0", "name": "bitmap0"}'12. Backup logic

At this point, you already know how to do full and incremental backups, but how do you combine the two?

"wait a minute, combined with what, make a full backup first, then create a bitmap, and then do an incremental backup after a period of time, okay?"

"I wish it were that simple, but you missed a key point. In the boot state, the disk data may change in real time. While you are still doing a full backup, the disk data may have changed, and then it is too late to create a bitmap."

"then I'll create a bitmap first, and then make a full backup, so that there will be no shortage of data."

"it looks OK, but in theory, it doesn't work either. You see, execute two QMP commands respectively, first execute the creation of bitmap, and then perform a full backup. Although the creation of bitmap is completed in milliseconds, if the disk IO changes quickly, it is possible to produce changes in data within a millimeter."

"what are we gonna do?"

"just look down and see."

Make sure to back up the virtual machine before starting

If you want to back up the disk before the virtual machine starts for the first time, you can start the virtual machine in paused status, which ensures that the disk IO will not change, and there will be less worries as mentioned above. The specific actions are as follows:

1 ️start the virtual machine in paused state (virsh start DOMAIN-- paused)

2 ️backup a full backup of the disk ("sync": "top")

3 ️create bitmap (block-dirty-bitmap-add) on disk

4 ️restore virtual machine state (virsh resume DOMAIN)

5 ️backup can then do incremental backup ("sync": "incremental")

Examples are as follows

Virsh start DOMAIN-- pausedvirsh qemu-monitor-command DOMAIN'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "top", "target": "/ opt/backup/top.img"} 'virsh qemu-monitor-command DOMAIN-- pretty' {"execute": "block-dirty-bitmap-add" "arguments": {"node": "drive-virtio-disk0", "name": "bitmap0"} 'virsh resume DOMAIN#. Run for a period of time... virsh qemu-monitor-command DOMAIN-pretty'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "incremental", "bitmap": "bitmap0", "target": "/ opt/backup/inc.0.qcow2"}}'#. Run virsh qemu-monitor-command DOMAIN-- pretty'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "incremental", "bitmap": "bitmap0", "target": "/ opt/backup/inc.1.qcow2"}}'

Make sure you want to back up while the virtual machine is running

If the virtual machine has been running for some time before deciding to make a backup, you need to use the "transaction" of QMP.

1 ️backup creates a bitmap (block-dirty-bitmap-add) and makes a full backup of the disk in a transactional manner ("sync": "top")

2 ️backup can then do incremental backup ("sync": "incremental")

Some of the QMP features support transactionality (the purpose of the transaction is to automatically roll back when one of the things fails to ensure data consistency, but it can also be used to ensure that there is no missing data between the creation of the bitmap and the start of the backup), so the above 1 ️transactions use the transaction operation

{"execute": "transaction", "arguments": {"actions": [{"type": "block-dirty-bitmap-add", "data": {"node": "drive-virtio-disk0", "name": "bitmap0"}}, {"type": "drive-backup", "data": {"device": "drive-virtio-disk0" "target": "/ path/to/full_backup.img", "sync": "top"}]}}

Examples are as follows

Virsh qemu-monitor-command DOMAIN-- pretty'{"execute": "transaction", "arguments": {"actions": [{"type": "block-dirty-bitmap-add", "data": {"node": "drive-virtio-disk0", "name": "bitmap0"}}, {"type": "drive-backup", "data": {"device": "drive-virtio-disk0" "target": "/ opt/backup/top.img", "sync": "top"}'#. Run for a period of time... virsh qemu-monitor-command DOMAIN-pretty'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "incremental", "bitmap": "bitmap0", "target": "/ opt/backup/inc.0.qcow2"}}'#. Run virsh qemu-monitor-command DOMAIN-- pretty'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "incremental", "bitmap": "bitmap0", "target": "/ opt/backup/inc.1.qcow2"}}'

If you have previously created bitmap and want to ignore bitmap, do a full backup again, and reset bitmap (for subsequent incremental backups), you can

{"execute": "transaction", "arguments": {"actions": [{"type": "block-dirty-bitmap-clear", "data": {"node": "drive-virtio-disk0", "name": "bitmap0"}}, {"type": "drive-backup", "data": {"device": "drive-virtio-disk0" "target": "/ path/to/new_full_backup.img", "sync": "top"}]}} 13. Remote backup

After testing, qemu supports only one remote backup method: iscsi, which uses the iscsi format in the target of drive-backup:

# target content format iscsi:// [[%] @] [:] / example iscsi://192.168.1.100:3260/iqn.2019-01.com.iaas/0virsh qemu-monitor-command DOMAIN'{"execute": "drive-backup", "arguments": {"device": "drive-virtio-disk0", "sync": "incremental", "bitmap": "bitmap0" "target": "iscsi://192.168.1.100:3260/iqn.2019-01.com.iaas/0"} 14. Backup task management

If the event is not received for a long time, how to check whether the backup task is still in progress, or if you want to interrupt the backup, how to do it?

View backup tasks

# View virsh qemu-monitor-command DOMAIN via qmp-- pretty'{"execute": "query-block-jobs"}'# View virsh qemu-monitor-command DOMAIN-- hmp 'info block-jobs' through hmp

Output

# View via qmp Output {"return": [{"auto-finalize": true, "io-status": "ok", "device": "drive-virtio-disk1", "auto-dismiss": true, "busy": true, "len": 1073741824, "offset": 2424832, "status": "running", "paused": false, "speed": 0 "ready": false, "type": "backup"}, {"auto-finalize": true, "io-status": "ok", "device": "drive-virtio-disk0", "auto-dismiss": true, "busy": true, "len": 21474836480, "offset": 163840000, "status": "running", "paused": false "speed": 0, "ready": false, "type": "backup"}], "id": "libvirt-45"} # View via qmp Output without task {"return": [], "id": "libvirt-360"} # viewed through hmp, output with task Type backup, device drive-virtio-disk1: Completed 20185088 of 1073741824 bytes, speed limit 0 bytes/sType backup, device drive-virtio-disk0: Completed 181403648 of 21474836480 bytes, speed limit 0 bytes/s# viewed via hmp, output No active jobs without task

Stop backup task

Virsh qemu-monitor-command DOMAIN-pretty'{"execute": "block-job-cancel", "arguments": {"device": "drive-virtio-disk1", "force": true}}'

If the force parameter is not taken, the default is false, which cannot be stopped when the task is paused in the case of false

Output

{"return": {}, "id": "libvirt-5880"}

Receive the event at the same time

2019-02-03 13 aborting 02id 58.535: 0000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "aborting", "id": "drive-virtio-disk1"} 2019-02-03 13V 02Switzerland 58.541: 0000: event BLOCK_JOB_CANCELLED for domain DOMAIN: {"device": "drive-virtio-disk1", "len": 2147483648, "offset": 29687808, "speed": 0 "type": "backup"} 2019-02-03 13 event JOB_STATUS_CHANGE for domain DOMAIN 58.5411400000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "concluded", "id": "drive-virtio-disk1"} 2019-02-03 1314140000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "null", "id": "drive-virtio-disk1"}

▷ according to official documents, the JOB_STATUS_CHANGE event only appeared from qemu-3.0, but it can be seen at qemu- 2.12 (not seen at 2.9.9) and has not been delved into.

▷ therefore, if qemu is 2.12, when a JOB_STATUS_CHANGE event is received, it should be ignored, and I will not repeat it later.

Pause backup task

Virsh qemu-monitor-command DOMAIN'{"execute": "block-job-pause", "arguments": {"device": "drive-virtio-disk0"}'

Output

{"return": {}, "id": "libvirt-5882"}

Receive the event at the same time (assuming the previous task is in running state)

2019-01-22 02 paused 42V 55.503: 0000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "paused", "id": "drive-virtio-disk0"}

Resume a paused backup task

Virsh qemu-monitor-command DOMAIN'{"execute": "block-job-resume", "arguments": {"device": "drive-virtio-disk0"}'

Output

{"return": {}, "id": "libvirt-5999"}

Receive the event at the same time

2019-01-22 02 running 4615: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "running", "id": "drive-virtio-disk0"} 15. Event View

Monitor event

Before the backup operation, you can open another shell window to listen for qmp events in real time.

# always listen to event virsh qemu-monitor-event DOMAIN-- timestamp-- loop# stop listening to virsh qemu-monitor-event DOMAIN-- event BLOCK_JOB_COMPLETED after receiving a specific event

Events that will be received when you start the backup

2019-01-29 03 created 1415 54.516: 0000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "created", "id": "drive-virtio-disk0"} 2019-01-29 0315 1415 54.5160000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "running", "id": "drive-virtio-disk0"}

Events received when the backup is complete

2019-01-24 062406VOV 25id 21.6290000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "created", "id": "drive-virtio-disk0"} 2019-01-2406VOV 25MAV 21.6290000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "running", "id": "drive-virtio-disk0"} 2019-01-24 0626V 34.93500000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "waiting" "id": "drive-virtio-disk0"} 2019-01-24 06event JOB_STATUS_CHANGE for domain DOMAIN 34.93500000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "pending", "id": "drive-virtio-disk0"} 2019-01-2406Vera 26purl 34.93500000: event BLOCK_JOB_COMPLETED for domain DOMAIN: {"device": "drive-virtio-disk0", "len": 21474836480, "offset": 21474836480, "speed": 0 "type": "backup"} 2019-01-24 06 event JOB_STATUS_CHANGE for domain DOMAIN: 34.93500000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "concluded", "id": "drive-virtio-disk0"} 2019-01-2406Vera 26concluded 34.93500000: event JOB_STATUS_CHANGE for domain DOMAIN: {"status": "null", "id": "drive-virtio-disk0"}

▷ focuses on the above BLOCK_JOB_COMPLETED event, which has been around since qemu-1.1

▷ unfortunately, the event content is not detailed, can not identify whether it is full, top, none or bitmap, nor can you see the file path generated by the backup

16. Backup chain, data recovery, merger

In fact, this chapter should be placed at the back of this article, but there is a lot of space in this article, so I'm afraid it's easy to be left out.

Backup chain

After an incremental backup, a backup chain is formed: full.qcow2

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.