Skip to contentSkip to conversation

incremental backup a running vm

question Zhang Haoyu 4 participants 9 messages Jan 21, 2015 – Jan 26, 2015 KVM

lore
Share & tools
lore
#0 lore raw

Hi,

Does drive_mirror support incremental backup a running vm? Or other mechanism does?

incremental backup a running vm requirements:
First time backup, all of the allocated data will be mirrored to destination, then a copied bitmap will be saved to a file, then the bitmap file will log dirty for the changed data.
Next time backup, only the dirty data will be mirrored to destination. Even the VM shutdown and start after several days,
the bitmap will be loaded while starting vm.
Any ideas?

Thanks,
Zhang Haoyu

maintainer
#1 lore raw
Quoted 5 lines · Zhang Haoyu

Hi,

Does drive_mirror support incremental backup a running vm? Or other mechanism does?

Quoted 8 lines

incremental backup a running vm requirements:
First time backup, all of the allocated data will be mirrored to destination, then a copied bitmap will be saved to a file, then the bitmap file will log dirty for the changed data.
Next time backup, only the dirty data will be mirrored to destination. Even the VM shutdown and start after several days,
the bitmap will be loaded while starting vm.
Any ideas?

Drive-mirror is for storage migration. For backup there is another job, drive-backup. drive-backup copies a point-in-time snapshot of one or more disks corresponding to when the backup was started.

Incremental backup is being worked on. You can see patches on the list.

Paolo

#2 lore raw
Quoted 17 lines · Paolo Bonzini
Quoted 5 lines · Zhang Haoyu

Hi,

Does drive_mirror support incremental backup a running vm? Or other mechanism does?

Quoted 8 lines

incremental backup a running vm requirements:
First time backup, all of the allocated data will be mirrored to destination, then a copied bitmap will be saved to a file, then the bitmap file will log dirty for the changed data.
Next time backup, only the dirty data will be mirrored to destination. Even the VM shutdown and start after several days,
the bitmap will be loaded while starting vm.
Any ideas?

Quoted 3 lines

Drive-mirror is for storage migration. For backup there is another job, drive-backup. drive-backup copies a point-in-time snapshot of one or more disks corresponding to when the backup was started.

Zhang, I've been testing the drive-backup command via QMP for a little while, and it works reasonably well. If you'd like to test it you can quickly try as below, once you have a QEMU instance runing with QMP (I invoke my QEMU instances with '-qmp tcp:localhost:4444,server').

The below script invokes the 'drive-backup' QMP command (Ensure you're using the correct disk, your disk ID might be 'drive-virtio-disk1' :-) )
-----------------
#!/bin/bash
set -x
exec 3<>/dev/tcp/localhost/4444
echo -e "{ 'execute': 'qmp_capabilities' }" >&3
read response <&3
echo $response
echo -e "{ 'execute': 'drive-backup', 'arguments':

      { 'device': 'drive-virtio-disk0', 'sync': 'full', 'target':
      '/export/backup-of-vm1.qcow2', 'mode': 'absolute-paths', 'format': 'qcow2' } }" >&3

read response <&3
echo $response
-----------------

Once the above is invoked succesfully, you can see the success of the
command on the shell where your QMP server is running:
-----------------
[. . .]
char device redirected to /dev/pts/8 (label charserial0)
QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
Formatting '/export/backup-of-vm1.qcow2', fmt=qcow2 size=53687091200 encryption=off cluster_size=65536 lazy_refcounts=off
-----------------

Quoted 1 line

Incremental backup is being worked on. You can see patches on the list.

Signature · 1 line
/kashyap
#8 lore raw
In reply to Paolo Bonzini
Quoted 17 lines · Paolo Bonzini
Quoted 5 lines · Zhang Haoyu

Hi,

Does drive_mirror support incremental backup a running vm? Or other mechanism does?

Quoted 8 lines

incremental backup a running vm requirements:
First time backup, all of the allocated data will be mirrored to destination, then a copied bitmap will be saved to a file, then the bitmap file will log dirty for the changed data.
Next time backup, only the dirty data will be mirrored to destination. Even the VM shutdown and start after several days,
the bitmap will be loaded while starting vm.
Any ideas?

Quoted 3 lines

Drive-mirror is for storage migration. For backup there is another job, drive-backup. drive-backup copies a point-in-time snapshot of one or more disks corresponding to when the backup was started.

Zhang, I've been testing the drive-backup command via QMP for a little while, and it works reasonably well. If you'd like to test it you can quickly try as below, once you have a QEMU instance runing with QMP (I invoke my QEMU instances with '-qmp tcp:localhost:4444,server').

The below script invokes the 'drive-backup' QMP command (Ensure you're using the correct disk, your disk ID might be 'drive-virtio-disk1' :-) )
-----------------
#!/bin/bash
set -x
exec 3<>/dev/tcp/localhost/4444
echo -e "{ 'execute': 'qmp_capabilities' }" >&3
read response <&3
echo $response
echo -e "{ 'execute': 'drive-backup', 'arguments':

      { 'device': 'drive-virtio-disk0', 'sync': 'full', 'target':
      '/export/backup-of-vm1.qcow2', 'mode': 'absolute-paths', 'format': 'qcow2' } }" >&3

read response <&3
echo $response
-----------------

Once the above is invoked succesfully, you can see the success of the
command on the shell where your QMP server is running:
-----------------
[. . .]
char device redirected to /dev/pts/8 (label charserial0)
QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
Formatting '/export/backup-of-vm1.qcow2', fmt=qcow2 size=53687091200 encryption=off cluster_size=65536 lazy_refcounts=off
-----------------

Quoted 5 lines

Incremental backup is being worked on. You can see patches on the list.

Paolo

Signature · 1 line
/kashyap
#3 lore raw
In reply to Kashyap Chamarthy
Quoted 18 lines · Kashyap Chamarthy
Quoted 17 lines · Paolo Bonzini
Quoted 13 lines · Zhang Haoyu

Hi,

Does drive_mirror support incremental backup a running vm? Or other mechanism does?

incremental backup a running vm requirements:
First time backup, all of the allocated data will be mirrored to destination, then a copied bitmap will be saved to a file, then the bitmap file will log dirty for the changed data.
Next time backup, only the dirty data will be mirrored to destination. Even the VM shutdown and start after several days,
the bitmap will be loaded while starting vm.
Any ideas?

Quoted 9 lines
Quoted 3 lines

Drive-mirror is for storage migration. For backup there is another job, drive-backup. drive-backup copies a point-in-time snapshot of one or more disks corresponding to when the backup was started.

Zhang, I've been testing the drive-backup command via QMP for a little while, and it works reasonably well. If you'd like to test it you can quickly try as below, once you have a QEMU instance runing with QMP (I invoke my QEMU instances with '-qmp tcp:localhost:4444,server').

Hi, Kashyap
I've tried ‘drive_backup’ via QMP,
but the snapshots were missed to backup to destination, I think the reason is that backup_run() only copy the guest data regarding qcow2 image.

Thanks,
Zhang Haoyu

Quoted 31 lines

The below script invokes the 'drive-backup' QMP command (Ensure you're using the correct disk, your disk ID might be 'drive-virtio-disk1' :-) )
-----------------
#!/bin/bash
set -x
exec 3<>/dev/tcp/localhost/4444
echo -e "{ 'execute': 'qmp_capabilities' }" >&3
read response <&3
echo $response
echo -e "{ 'execute': 'drive-backup', 'arguments':

      { 'device': 'drive-virtio-disk0', 'sync': 'full', 'target':
      '/export/backup-of-vm1.qcow2', 'mode': 'absolute-paths', 'format': 'qcow2' } }" >&3

read response <&3
echo $response
-----------------

Once the above is invoked succesfully, you can see the success of the
command on the shell where your QMP server is running:
-----------------
[. . .]
char device redirected to /dev/pts/8 (label charserial0)
QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
Formatting '/export/backup-of-vm1.qcow2', fmt=qcow2 size=53687091200 encryption=off cluster_size=65536 lazy_refcounts=off
-----------------

Quoted 1 line

Incremental backup is being worked on. You can see patches on the list.

Signature · 1 line
/kashyap
maintainer
#4 lore raw
Quoted 5 lines · Zhang Haoyu

Hi, Kashyap
I've tried ‘drive_backup’ via QMP,
but the snapshots were missed to backup to destination, I think the reason is that backup_run() only copy the guest data regarding qcow2 image.

Yes, that's the case.

QEMU cannot still access internal snapshots while the file is open. External snapshots are opened read-only, and can be copied with "cp" while QEMU is running.

Paolo

#5 lore raw
In reply to Zhang Haoyu
Quoted 8 lines · Paolo Bonzini
Quoted 5 lines · Zhang Haoyu

Hi, Kashyap
I've tried ‘drive_backup’ via QMP,
but the snapshots were missed to backup to destination, I think the reason is that backup_run() only copy the guest data regarding qcow2 image.

Quoted 5 lines

Yes, that's the case.

QEMU cannot still access internal snapshots while the file is open. External snapshots are opened read-only, and can be copied with "cp" while QEMU is running.

Thanks, Paolo,
but too many internal snapshots were saved by customers, switching to external snapshot mechanism has significant impaction on subsequent upgrade.

Another problem:
drive_backup just implement one time backup,
but I want VMWare's VDP-like backup mechanism.
The initial backup of a virtual machine takes comparatively more time, because all of the data for that virtual machine is being backed up. Subsequent backups of the same virtual machine take less time, because changed block tracking (log dirty) mechanism is used to only backup the dirty data. After inittial backup done, even the VM shutdown, but subsequent backup also only copy the changed data.

Thanks,
Zhang Haoyu

Quoted 2 lines

Paolo

maintainer
#6 lore raw
Quoted 4 lines · Zhang Haoyu

Thanks, Paolo,
but too many internal snapshots were saved by customers, switching to external snapshot mechanism has significant impaction on subsequent upgrade.

In that case, patches are welcome. :)

Quoted 9 lines

Another problem:
drive_backup just implement one time backup,
but I want VMWare's VDP-like backup mechanism.
The initial backup of a virtual machine takes comparatively more time, because all of the data for that virtual machine is being backed up. Subsequent backups of the same virtual machine take less time, because changed block tracking (log dirty) mechanism is used to only backup the dirty data. After inittial backup done, even the VM shutdown, but subsequent backup also only copy the changed data.

As mentioned before, patches for this are on the list.

Paolo

#7 lore raw
Quoted 19 lines · Paolo Bonzini
Quoted 4 lines · Zhang Haoyu

Thanks, Paolo,
but too many internal snapshots were saved by customers, switching to external snapshot mechanism has significant impaction on subsequent upgrade.

In that case, patches are welcome. :)

Quoted 9 lines

Another problem:
drive_backup just implement one time backup,
but I want VMWare's VDP-like backup mechanism.
The initial backup of a virtual machine takes comparatively more time, because all of the data for that virtual machine is being backed up. Subsequent backups of the same virtual machine take less time, because changed block tracking (log dirty) mechanism is used to only backup the dirty data. After inittial backup done, even the VM shutdown, but subsequent backup also only copy the changed data.

Quoted 2 lines

As mentioned before, patches for this are on the list.

I see, thanks, Paolo.

Quoted 1 line

Paolo