Skip to contentSkip to conversation

e2image: Add support for QCOW2 image format

RFC In review Lukas Czerner 3 participants 13 messages Dec 1, 2010 – Dec 6, 2010

lore
Share & tools
lore
#0 lore raw

cover e2image: Add support for QCOW2 image format

Hi, all

As Ted suggested on LPC I started to play with e2image to add support for exporting images in QCOW2 format (http://en.wikipedia.org/wiki/Qcow2). The advantage of this format is mainly that the resulting image is quite small (because we write only the metadata blocks) and it is not sparse unlike "sparse" image format.

I am presenting you an initial parch, which works for me quite nicely, however so far it did received just limited testing. Nevertheless you can try it simply by invoking e2image with '-q' parameter like this:

e2image -q /dev/sda1 myimage

or you can add '-s' option so the file names are scrambled. To mount that qcow2 image I usually use 'qemu-nbd':

modprobe nbd max_part=8
qemu-nbd --connect=/dev/nbd0 myimage
mount -o ro /dev/nbd0 /mnt/test/

However I have experienced some problems with qemu-nbd from time to time. Also for some reason, when you mount the image read-write, you'll destroy it. So far I was not able to figure out why (I did not tried at all to be honest), it may be problem in my image format or even in qemu itself, so if you have any idea, please let me know.

QCOW2 format supports other neat features like compression, encryption and snapshots. None of those feature are supported in this patch and the question is, do we need it ? I can imagine for example snapshots to be useful for roll-back changes made by e2fsck, but this is hardly problem of e2image itself.

So, please look at the patch, try it and let me know what else you thing is useful to implement, or what would you like to change.

Anny comments appreciated.

Thanks!

-Lukas

#1 lore raw

PATCH 1/1 e2image: Add support for qcow2 format

This commit adds support for exporting filesystem into QCOW2 image format. Like sparse format this saves space, by writing only necessary (metadata blocks) into image. Unlike sparse image, QCOW2 image is NOT sparse, hence does not change its size by copying with not-sparse-aware tools.

New options '-q' has been added to tell the e2image to use QCOW2 as an output image format. QCOW2 supports encryption and compression, however e2image so far does no support such features, however you can still scramble filenames with '-s' option.

---
 lib/ext2fs/bitops.h  |    4 +
 lib/ext2fs/e2image.h |   59 +++++
 misc/e2image.c       |  587 +++++++++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 599 insertions(+), 51 deletions(-)
lib/ext2fs/bitops.h +4 -0
index 3ded002..83a01e4 100644
@@ -31,6 +31,8 @@ extern __u64 ext2fs_swab64(__u64 val);
 #define ext2fs_le32_to_cpu(x) ext2fs_swab32((x))
 #define ext2fs_cpu_to_le16(x) ext2fs_swab16((x))
 #define ext2fs_le16_to_cpu(x) ext2fs_swab16((x))
+#define ext2fs_cpu_to_be64(x) ((__u64)(x))
+#define ext2fs_be64_to_cpu(x) ((__u64)(x))
 #define ext2fs_cpu_to_be32(x) ((__u32)(x))
 #define ext2fs_be32_to_cpu(x) ((__u32)(x))
 #define ext2fs_cpu_to_be16(x) ((__u16)(x))
@@ -42,6 +44,8 @@ extern __u64 ext2fs_swab64(__u64 val);
 #define ext2fs_le32_to_cpu(x) ((__u32)(x))
 #define ext2fs_cpu_to_le16(x) ((__u16)(x))
 #define ext2fs_le16_to_cpu(x) ((__u16)(x))
+#define ext2fs_cpu_to_be64(x) ext2fs_swab64((x))
+#define ext2fs_be64_to_cpu(x) ext2fs_swab64((x))
 #define ext2fs_cpu_to_be32(x) ext2fs_swab32((x))
 #define ext2fs_be32_to_cpu(x) ext2fs_swab32((x))
 #define ext2fs_cpu_to_be16(x) ext2fs_swab16((x))
lib/ext2fs/e2image.h +59 -0

80 lines not on this page · raw message

misc/e2image.c +536 -51

697 lines not on this page · raw message

Signature · 1 line
1.7.2.3
#2 lore raw
In reply to Lukas Czerner
Quoted 27 lines · Lukas Czerner

Hi, all

As Ted suggested on LPC I started to play with e2image to add support for exporting images in QCOW2 format (http://en.wikipedia.org/wiki/Qcow2). The advantage of this format is mainly that the resulting image is quite small (because we write only the metadata blocks) and it is not sparse unlike "sparse" image format.

I am presenting you an initial parch, which works for me quite nicely, however so far it did received just limited testing. Nevertheless you can try it simply by invoking e2image with '-q' parameter like this:

       e2image -q /dev/sda1 myimage

or you can add '-s' option so the file names are scrambled. To mount that qcow2 image I usually use 'qemu-nbd':

       modprobe nbd max_part=8
       qemu-nbd --connect=/dev/nbd0 myimage
       mount -o ro /dev/nbd0 /mnt/test/

However I have experienced some problems with qemu-nbd from time to time. Also for some reason, when you mount the image read-write, you'll destroy it. So far I was not able to figure out why (I did not tried at all to be honest), it may be problem in my image format or even in qemu itself, so if you have any idea, please let me know.

Quoted 8 lines

QCOW2 format supports other neat features like compression, encryption and snapshots. None of those feature are supported in this patch and the question is, do we need it ? I can imagine for example snapshots to be useful for roll-back changes made by e2fsck, but this is hardly problem of e2image itself.

So, please look at the patch, try it and let me know what else you thing is useful to implement, or what would you like to change.

Hi Lukas,

This is very cool :-)
My wish is to export next3/ext4 snapshots as qcow2 snapshots.
I actually thought of implementing e2image -x <snapid> as a way to export a snapshot image,
but using qcow2, file system can be exported with all of it's snapshots. It should be quite trivial since next3 snapshots contain a map of changed blocks and I suppose qcow2 snapshots should be the same.
Now I only need to find the time to do it...

Amir.

Quoted 10 lines

Anny comments appreciated.

Thanks!

-Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 48 lines · Amir Goldstein
Quoted 27 lines · Lukas Czerner

Hi, all

As Ted suggested on LPC I started to play with e2image to add support for exporting images in QCOW2 format (http://en.wikipedia.org/wiki/Qcow2). The advantage of this format is mainly that the resulting image is quite small (because we write only the metadata blocks) and it is not sparse unlike "sparse" image format.

I am presenting you an initial parch, which works for me quite nicely, however so far it did received just limited testing. Nevertheless you can try it simply by invoking e2image with '-q' parameter like this:

       e2image -q /dev/sda1 myimage

or you can add '-s' option so the file names are scrambled. To mount that qcow2 image I usually use 'qemu-nbd':

       modprobe nbd max_part=8
       qemu-nbd --connect=/dev/nbd0 myimage
       mount -o ro /dev/nbd0 /mnt/test/

However I have experienced some problems with qemu-nbd from time to time. Also for some reason, when you mount the image read-write, you'll destroy it. So far I was not able to figure out why (I did not tried at all to be honest), it may be problem in my image format or even in qemu itself, so if you have any idea, please let me know.

Quoted 8 lines

QCOW2 format supports other neat features like compression, encryption and snapshots. None of those feature are supported in this patch and the question is, do we need it ? I can imagine for example snapshots to be useful for roll-back changes made by e2fsck, but this is hardly problem of e2image itself.

So, please look at the patch, try it and let me know what else you thing is useful to implement, or what would you like to change.

Hi Lukas,

This is very cool :-)
My wish is to export next3/ext4 snapshots as qcow2 snapshots.
I actually thought of implementing e2image -x <snapid> as a way to export a snapshot image,
but using qcow2, file system can be exported with all of it's snapshots. It should be quite trivial since next3 snapshots contain a map of changed blocks and I suppose qcow2 snapshots should be the same.
Now I only need to find the time to do it...

Quoted 1 line

Amir.

Hi Amir,

I seems like a cool feature, so when QCOW2 infrastructure is in place adding snapshots to it should not be a big deal (not that QCOW2 infrastructure is :)), however I was wondering are you planning to export only metadata, or even data itself ?

Thanks!

-Lukas

Quoted 13 lines
Quoted 12 lines

Anny comments appreciated.

Thanks!

-Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#4 lore raw
Quoted 24 lines · Lukas Czerner
Quoted 22 lines · Amir Goldstein
Quoted 9 lines · Lukas Czerner

QCOW2 format supports other neat features like compression, encryption and snapshots. None of those feature are supported in this patch and the question is, do we need it ? I can imagine for example snapshots to be useful for roll-back changes made by e2fsck, but this is hardly problem of e2image itself.

So, please look at the patch, try it and let me know what else you thing is useful to implement, or what would you like to change.

Hi Lukas,

This is very cool :-)
My wish is to export next3/ext4 snapshots as qcow2 snapshots.
I actually thought of implementing e2image -x <snapid> as a way to export a snapshot image,
but using qcow2, file system can be exported with all of it's snapshots. It should be quite trivial since next3 snapshots contain a map of changed blocks and I suppose qcow2 snapshots should be the same.
Now I only need to find the time to do it...

Quoted 9 lines
Quoted 1 line

Amir.

Hi Amir,

I seems like a cool feature, so when QCOW2 infrastructure is in place adding snapshots to it should not be a big deal (not that QCOW2 infrastructure is :)), however I was wondering are you planning to export only metadata, or even data itself ?

There are 2 applications I can think of for exporting snapshots. One needs only the metadata and the other requires the data as well.

1. Rollback to snapshot (metadata only)
----------------------------------------------------------
Unlike ZFS/Btrfs, there is no inherent way to "rollback to snapshot" with Ext4 snapshots.
Instead, all changed metadata needs to be copied over from snapshot. The only plausible way to do this is to un-mount the file system (or re-mount read-only),
use e2image to export a metadata image of a snapshot to a different location and then
overwrite the filesystem with snapshot metadata (data blocks are already in-place).

This method can be applied today, even without qcow2 support, but with qcow2 snapshots,
you can generate a single e2image, which can be used to rollback to any snapshot and to
restore the original filesystem (as long as the rolled back version wasn't modifed).

2. Filesystem replication (data + metadata)
--------------------------------------------------------------
This application is inspired by ZFS replication:
http://wikitech-static.wikimedia.org/articles/z/f/s/Zfs_replication.html
The idea is to start with a remote replica by transferring a full copy of snapshot N1
and then push incremental differences N1..N2 to roll the replica to snapshot N2. So if e2image has the ability to export a full snapshot image (including data) and the capability to export incremental qcow2 snapshots, those could be transferred
to the remote location and be used to create and update the replicated filesystem.

So to answer your original question ("what else you thing is useful to implement"),
e2image -d would be nice (export data blocks).

Amir.

#5 lore raw
Quoted 38 lines · Amir Goldstein
Quoted 24 lines · Lukas Czerner
Quoted 22 lines · Amir Goldstein
Quoted 9 lines · Lukas Czerner

QCOW2 format supports other neat features like compression, encryption and snapshots. None of those feature are supported in this patch and the question is, do we need it ? I can imagine for example snapshots to be useful for roll-back changes made by e2fsck, but this is hardly problem of e2image itself.

So, please look at the patch, try it and let me know what else you thing is useful to implement, or what would you like to change.

Hi Lukas,

This is very cool :-)
My wish is to export next3/ext4 snapshots as qcow2 snapshots.
I actually thought of implementing e2image -x <snapid> as a way to export a snapshot image,
but using qcow2, file system can be exported with all of it's snapshots. It should be quite trivial since next3 snapshots contain a map of changed blocks and I suppose qcow2 snapshots should be the same.
Now I only need to find the time to do it...

Quoted 9 lines
Quoted 1 line

Amir.

Hi Amir,

I seems like a cool feature, so when QCOW2 infrastructure is in place adding snapshots to it should not be a big deal (not that QCOW2 infrastructure is :)), however I was wondering are you planning to export only metadata, or even data itself ?

There are 2 applications I can think of for exporting snapshots. One needs only the metadata and the other requires the data as well.

Quoted 7 lines

1. Rollback to snapshot (metadata only)
----------------------------------------------------------
Unlike ZFS/Btrfs, there is no inherent way to "rollback to snapshot" with Ext4 snapshots.
Instead, all changed metadata needs to be copied over from snapshot. The only plausible way to do this is to un-mount the file system (or re-mount read-only),

So, you're saying that it would not be possible to implement in-kernel snapshot roll-back for ext4 with snapshots ? I find this hard to believe, because we can always freeze the filesystem a do whatever we want to do, or am I missing something ? If it would be possible this would be much better solution then exporting and restoring it from QCOW2 image.

Quoted 12 lines

use e2image to export a metadata image of a snapshot to a different location and then
overwrite the filesystem with snapshot metadata (data blocks are already in-place).

This method can be applied today, even without qcow2 support, but with qcow2 snapshots,
you can generate a single e2image, which can be used to rollback to any snapshot and to
restore the original filesystem (as long as the rolled back version wasn't modifed).

Quoted 12 lines

2. Filesystem replication (data + metadata)
--------------------------------------------------------------
This application is inspired by ZFS replication:
http://wikitech-static.wikimedia.org/articles/z/f/s/Zfs_replication.html
The idea is to start with a remote replica by transferring a full copy of snapshot N1
and then push incremental differences N1..N2 to roll the replica to snapshot N2. So if e2image has the ability to export a full snapshot image (including data) and the capability to export incremental qcow2 snapshots, those could be transferred
to the remote location and be used to create and update the replicated filesystem.

I am sorry but I do not really get it, can't you just export filesystem data + metadata into qcow2 image without any qcow2 snapshot functionality and then recreate the whole filesystem from this image ? It seems to me that you do not need qcow2 snapshot functionality to do this, however I might be missing something.

Quoted 4 lines

So to answer your original question ("what else you thing is useful to implement"),
e2image -d would be nice (export data blocks).

This is really easy thing to do, so I can do this :).

Thanks!

-Lukas

Quoted 7 lines

Amir.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
Quoted 48 lines · Lukas Czerner
Quoted 38 lines · Amir Goldstein
Quoted 24 lines · Lukas Czerner
Quoted 22 lines · Amir Goldstein
Quoted 9 lines · Lukas Czerner

QCOW2 format supports other neat features like compression, encryption and snapshots. None of those feature are supported in this patch and the question is, do we need it ? I can imagine for example snapshots to be useful for roll-back changes made by e2fsck, but this is hardly problem of e2image itself.

So, please look at the patch, try it and let me know what else you thing is useful to implement, or what would you like to change.

Hi Lukas,

This is very cool :-)
My wish is to export next3/ext4 snapshots as qcow2 snapshots.
I actually thought of implementing e2image -x <snapid> as a way to export a snapshot image,
but using qcow2, file system can be exported with all of it's snapshots. It should be quite trivial since next3 snapshots contain a map of changed blocks and I suppose qcow2 snapshots should be the same.
Now I only need to find the time to do it...

Quoted 9 lines
Quoted 1 line

Amir.

Hi Amir,

I seems like a cool feature, so when QCOW2 infrastructure is in place adding snapshots to it should not be a big deal (not that QCOW2 infrastructure is :)), however I was wondering are you planning to export only metadata, or even data itself ?

There are 2 applications I can think of for exporting snapshots. One needs only the metadata and the other requires the data as well.

Quoted 7 lines

1. Rollback to snapshot (metadata only)
----------------------------------------------------------
Unlike ZFS/Btrfs, there is no inherent way to "rollback to snapshot" with Ext4 snapshots.
Instead, all changed metadata needs to be copied over from snapshot. The only plausible way to do this is to un-mount the file system (or re-mount read-only),

Quoted 7 lines

So, you're saying that it would not be possible to implement in-kernel snapshot roll-back for ext4 with snapshots ? I find this hard to believe, because we can always freeze the filesystem a do whatever we want to do, or am I missing something ? If it would be possible this would be much better solution then exporting and restoring it from QCOW2 image.

The key problem is how to do this rollback in an atomic manner. And it would be hard to achieve reliable results without using an external storage space to hold the "rollback patch".
While Btrfs can hold 2 perfects trees on disk and switch the root in an atomic operations, Ext4 needs to copy over a large number of fixed positioned metadata block, which makes the rollback feature "non-trivial".

Quoted 25 lines
Quoted 12 lines

use e2image to export a metadata image of a snapshot to a different location and then
overwrite the filesystem with snapshot metadata (data blocks are already in-place).

This method can be applied today, even without qcow2 support, but with qcow2 snapshots,
you can generate a single e2image, which can be used to rollback to any snapshot and to
restore the original filesystem (as long as the rolled back version wasn't modifed).

Quoted 12 lines

2. Filesystem replication (data + metadata)
--------------------------------------------------------------
This application is inspired by ZFS replication:
http://wikitech-static.wikimedia.org/articles/z/f/s/Zfs_replication.html
The idea is to start with a remote replica by transferring a full copy of snapshot N1
and then push incremental differences N1..N2 to roll the replica to snapshot N2. So if e2image has the ability to export a full snapshot image (including data) and the capability to export incremental qcow2 snapshots, those could be transferred
to the remote location and be used to create and update the replicated filesystem.

Quoted 5 lines

I am sorry but I do not really get it, can't you just export filesystem data + metadata into qcow2 image without any qcow2 snapshot functionality and then recreate the whole filesystem from this image ? It seems to me that you do not need qcow2 snapshot functionality to do this, however I might be missing something.

you are missing the cost of transferring a full image over a slow network link. snapshots can provide the minimal set of blocks, which needs to be transferred.

So e2image -x 1..3 will not generate a full image, but only the changed metadata blocks between snapshot 1..3 and the data blocks allocated (or deleted and re-allocated) between snapshots 1..3. It should really also include the information
about deleted blocks between 1..3 (to issue discard on target), but I wouldn't know
how to utilize a qcow2 incremental snapshot to describe that.

Perhaps the use of qcow2 format for this application is not the perfect choice after all.

Quoted 8 lines
Quoted 5 lines

So to answer your original question ("what else you thing is useful to implement"),
e2image -d would be nice (export data blocks).

This is really easy thing to do, so I can do this :).

Cool :-)

Quoted 12 lines

Thanks!

-Lukas

Quoted 7 lines

Amir.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
Quoted 57 lines · Amir Goldstein
Quoted 48 lines · Lukas Czerner
Quoted 38 lines · Amir Goldstein
Quoted 24 lines · Lukas Czerner
Quoted 22 lines · Amir Goldstein
Quoted 9 lines · Lukas Czerner

QCOW2 format supports other neat features like compression, encryption and snapshots. None of those feature are supported in this patch and the question is, do we need it ? I can imagine for example snapshots to be useful for roll-back changes made by e2fsck, but this is hardly problem of e2image itself.

So, please look at the patch, try it and let me know what else you thing is useful to implement, or what would you like to change.

Hi Lukas,

This is very cool :-)
My wish is to export next3/ext4 snapshots as qcow2 snapshots.
I actually thought of implementing e2image -x <snapid> as a way to export a snapshot image,
but using qcow2, file system can be exported with all of it's snapshots. It should be quite trivial since next3 snapshots contain a map of changed blocks and I suppose qcow2 snapshots should be the same.
Now I only need to find the time to do it...

Quoted 9 lines
Quoted 1 line

Amir.

Hi Amir,

I seems like a cool feature, so when QCOW2 infrastructure is in place adding snapshots to it should not be a big deal (not that QCOW2 infrastructure is :)), however I was wondering are you planning to export only metadata, or even data itself ?

There are 2 applications I can think of for exporting snapshots. One needs only the metadata and the other requires the data as well.

Quoted 7 lines

1. Rollback to snapshot (metadata only)
----------------------------------------------------------
Unlike ZFS/Btrfs, there is no inherent way to "rollback to snapshot" with Ext4 snapshots.
Instead, all changed metadata needs to be copied over from snapshot. The only plausible way to do this is to un-mount the file system (or re-mount read-only),

Quoted 7 lines

So, you're saying that it would not be possible to implement in-kernel snapshot roll-back for ext4 with snapshots ? I find this hard to believe, because we can always freeze the filesystem a do whatever we want to do, or am I missing something ? If it would be possible this would be much better solution then exporting and restoring it from QCOW2 image.

Quoted 1 line

The key problem is how to do this rollback in an atomic manner.

As I said, we can freeze the filesystem.

Quoted 37 lines

And it would be hard to achieve reliable results without using an external storage space to hold the "rollback patch".
While Btrfs can hold 2 perfects trees on disk and switch the root in an atomic operations, Ext4 needs to copy over a large number of fixed positioned metadata block, which makes the rollback feature "non-trivial".

Quoted 25 lines
Quoted 12 lines

use e2image to export a metadata image of a snapshot to a different location and then
overwrite the filesystem with snapshot metadata (data blocks are already in-place).

This method can be applied today, even without qcow2 support, but with qcow2 snapshots,
you can generate a single e2image, which can be used to rollback to any snapshot and to
restore the original filesystem (as long as the rolled back version wasn't modifed).

Quoted 12 lines

2. Filesystem replication (data + metadata)
--------------------------------------------------------------
This application is inspired by ZFS replication:
http://wikitech-static.wikimedia.org/articles/z/f/s/Zfs_replication.html
The idea is to start with a remote replica by transferring a full copy of snapshot N1
and then push incremental differences N1..N2 to roll the replica to snapshot N2. So if e2image has the ability to export a full snapshot image (including data) and the capability to export incremental qcow2 snapshots, those could be transferred
to the remote location and be used to create and update the replicated filesystem.

Quoted 5 lines

I am sorry but I do not really get it, can't you just export filesystem data + metadata into qcow2 image without any qcow2 snapshot functionality and then recreate the whole filesystem from this image ? It seems to me that you do not need qcow2 snapshot functionality to do this, however I might be missing something.

Quoted 8 lines

you are missing the cost of transferring a full image over a slow network link. snapshots can provide the minimal set of blocks, which needs to be transferred.

So e2image -x 1..3 will not generate a full image, but only the changed metadata blocks between snapshot 1..3 and the data blocks allocated (or deleted and re-allocated) between snapshots 1..3. It should really also include the information
about deleted blocks between 1..3 (to issue discard on target), but I

Ok, but you do not need qcow2 snapshot support for it, at all. qcow2 format does not need to know about the snapshots, because you know what the data in the qcow images means and what to do with them.

You can export just the diff between some snapshots as you mentioned and then, when installing it on another filesystem the utility doing the job (e2image probably) would know what to do even without qcow2 snapshots, since all the data needed are in the image anyway, qcow2 snapshot is just useless abstraction we do not need in this case.

Quoted 34 lines

wouldn't know
how to utilize a qcow2 incremental snapshot to describe that.

Perhaps the use of qcow2 format for this application is not the perfect choice after all.

Quoted 8 lines
Quoted 5 lines

So to answer your original question ("what else you thing is useful to implement"),
e2image -d would be nice (export data blocks).

This is really easy thing to do, so I can do this :).

Cool :-)

Quoted 12 lines

Thanks!

-Lukas

Quoted 7 lines

Amir.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw
Quoted 59 lines · Lukas Czerner
Quoted 57 lines · Amir Goldstein
Quoted 55 lines · Lukas Czerner
Quoted 38 lines · Amir Goldstein
Quoted 24 lines · Lukas Czerner
Quoted 22 lines · Amir Goldstein
Quoted 9 lines · Lukas Czerner

QCOW2 format supports other neat features like compression, encryption and snapshots. None of those feature are supported in this patch and the question is, do we need it ? I can imagine for example snapshots to be useful for roll-back changes made by e2fsck, but this is hardly problem of e2image itself.

So, please look at the patch, try it and let me know what else you thing is useful to implement, or what would you like to change.

Hi Lukas,

This is very cool :-)
My wish is to export next3/ext4 snapshots as qcow2 snapshots.
I actually thought of implementing e2image -x <snapid> as a way to export a snapshot image,
but using qcow2, file system can be exported with all of it's snapshots. It should be quite trivial since next3 snapshots contain a map of changed blocks and I suppose qcow2 snapshots should be the same.
Now I only need to find the time to do it...

Quoted 9 lines
Quoted 1 line

Amir.

Hi Amir,

I seems like a cool feature, so when QCOW2 infrastructure is in place adding snapshots to it should not be a big deal (not that QCOW2 infrastructure is :)), however I was wondering are you planning to export only metadata, or even data itself ?

There are 2 applications I can think of for exporting snapshots. One needs only the metadata and the other requires the data as well.

Quoted 7 lines

1. Rollback to snapshot (metadata only)
----------------------------------------------------------
Unlike ZFS/Btrfs, there is no inherent way to "rollback to snapshot" with Ext4 snapshots.
Instead, all changed metadata needs to be copied over from snapshot. The only plausible way to do this is to un-mount the file system (or re-mount read-only),

So, you're saying that it would not be possible to implement in-kernel snapshot roll-back for ext4 with snapshots ? I find this hard to believe, because we can always freeze the filesystem a do whatever we want to do, or am I missing something ? If it would be possible this would be much better solution then exporting and restoring it from QCOW2 image.

Quoted 3 lines
Quoted 1 line

The key problem is how to do this rollback in an atomic manner.

As I said, we can freeze the filesystem.

Theoretically, yes, it is possible, but very very fragile, not a quality you would want when you are trying to recover your filesystem.

Let me explain with a simple example of rolling back to latest snapshot.

On mount with option 'rollback', after snapshots load (and before orphan cleanup),
start rollback to latest snapshot:
1. store backup copy of latest snapshot inode in super block
2. set ROLLBACK_FS flag in super block
3. start copying blocks from snapshot inode to filesystem (skip super block)
4. copy super block from snapshot inode to filesystem and update relevant fields from cache.

On mount, if ROLLBACK_FS flag is set:
6. load latest snapshot inode from backup copy
goto step 3.

Now if at any point in this process there is a problem, it will leave the filesystem nowhere near a consistent state.
For example, a corrupted snapshot file may result in copying blocks over the snapshot inode indirect blocks, sending the entire rollback operation down the drain and your filesystem with it.
This is why I think it would be wiser to have a full rollback patch put aside, which can be used to replay or rewind the rollback operation.

Quoted 51 lines
Quoted 38 lines

And it would be hard to achieve reliable results without using an external storage space to hold the "rollback patch".
While Btrfs can hold 2 perfects trees on disk and switch the root in an atomic operations, Ext4 needs to copy over a large number of fixed positioned metadata block, which makes the rollback feature "non-trivial".

Quoted 25 lines
Quoted 12 lines

use e2image to export a metadata image of a snapshot to a different location and then
overwrite the filesystem with snapshot metadata (data blocks are already in-place).

This method can be applied today, even without qcow2 support, but with qcow2 snapshots,
you can generate a single e2image, which can be used to rollback to any snapshot and to
restore the original filesystem (as long as the rolled back version wasn't modifed).

Quoted 12 lines

2. Filesystem replication (data + metadata)
--------------------------------------------------------------
This application is inspired by ZFS replication:
http://wikitech-static.wikimedia.org/articles/z/f/s/Zfs_replication.html
The idea is to start with a remote replica by transferring a full copy of snapshot N1
and then push incremental differences N1..N2 to roll the replica to snapshot N2. So if e2image has the ability to export a full snapshot image (including data) and the capability to export incremental qcow2 snapshots, those could be transferred
to the remote location and be used to create and update the replicated filesystem.

Quoted 5 lines

I am sorry but I do not really get it, can't you just export filesystem data + metadata into qcow2 image without any qcow2 snapshot functionality and then recreate the whole filesystem from this image ? It seems to me that you do not need qcow2 snapshot functionality to do this, however I might be missing something.

Quoted 8 lines

you are missing the cost of transferring a full image over a slow network link. snapshots can provide the minimal set of blocks, which needs to be transferred.

So e2image -x 1..3 will not generate a full image, but only the changed metadata blocks between snapshot 1..3 and the data blocks allocated (or deleted and re-allocated) between snapshots 1..3. It should really also include the information
about deleted blocks between 1..3 (to issue discard on target), but I

Ok, but you do not need qcow2 snapshot support for it, at all. qcow2 format does not need to know about the snapshots, because you know what the data in the qcow images means and what to do with them.

Quoted 5 lines

You can export just the diff between some snapshots as you mentioned and then, when installing it on another filesystem the utility doing the job (e2image probably) would know what to do even without qcow2 snapshots, since all the data needed are in the image anyway, qcow2 snapshot is just useless abstraction we do not need in this case.

I agree. I was forcing my problem over this solution, trying to squeeze every last drop of coolness from qcow2 snapshots. it doesn't fit in here.

Still for the first problem (rollback) I think that qcow2 snapshots can be a perfect and simple solution.

e2image starts with exporting filesystem to full qcow2 image, then takes a qcow2 snapshot and proceeds writing blocks modified by latest snapshot and so forth until the oldest snapshot.

The resulting qcow2 snapshot list is an inversion of the Ext4 snapshots list and it gives you both the ability to rollback to any older snapshot and to undo all rollbacks by restoring the base image.

Quoted 30 lines
Quoted 30 lines

wouldn't know
how to utilize a qcow2 incremental snapshot to describe that.

Perhaps the use of qcow2 format for this application is not the perfect choice after all.

Quoted 8 lines
Quoted 5 lines

So to answer your original question ("what else you thing is useful to implement"),
e2image -d would be nice (export data blocks).

This is really easy thing to do, so I can do this :).

Cool :-)

Quoted 12 lines

Thanks!

-Lukas

Quoted 7 lines

Amir.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 5 lines
Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
Quoted 14 lines · Amir Goldstein
Quoted 6 lines · Lukas Czerner

You can export just the diff between some snapshots as you mentioned and then, when installing it on another filesystem the utility doing the job (e2image probably) would know what to do even without qcow2 snapshots, since all the data needed are in the image anyway, qcow2 snapshot is just useless abstraction we do not need in this case.

I agree. I was forcing my problem over this solution, trying to squeeze every last drop of coolness from qcow2 snapshots. it doesn't fit in here.

Still for the first problem (rollback) I think that qcow2 snapshots can be a perfect and simple solution.

Quoted 8 lines

e2image starts with exporting filesystem to full qcow2 image, then takes a qcow2 snapshot and proceeds writing blocks modified by latest snapshot and so forth until the oldest snapshot.

The resulting qcow2 snapshot list is an inversion of the Ext4 snapshots list and it gives you both the ability to rollback to any older snapshot and to undo all rollbacks by restoring the base image.

Hi Lukas,

I think I may have failed to explain my reasoning for using qcow2 snapshots properly and I understand your confusion.

Of course, if e2image would produce a full image of the filesystem, Ext4 snapshots information would be a part of the information encoded in the image.

For rollback to snapshot application, at some point, either on image creation or on image install, the snapshot diff patches (which are stored in the snapshot inodes) need to be decoded.

If the snapshot diffs are decoded on e2image install, there is no need for qcow2 snapshots, like you said.

My suggestion to decode the snapshot diffs on e2image create and to store them in qcow2 snapshots format
has 2 advantages over decoding on e2image install:

1. Should we want to implement create of snapshot image using e2image -x <snapid>,
the easiest implementation would involve creating a full filesystem image and then applying snapshot diffs on top of it. With this implementation, we can get the full filesystem image and an image of every snapshot on the way to <snapid> with no extra cost, if we just start a qcow2 snapshot before applying every snapshot diff.

2. At the moment, the only way to access next3 snapshots is to mount the filesystem
and then mount the snapshot via a loop device. Now if there is a problem to mount the filesystem
the snapshots are not accessible as well, which is a shame if you want to backup some data
before attempting to fix the filesystem.
Using the qcow2 exported image (with snapshots converted to qcow2 snapshots), it is possible
to mount every one of the snapshots, directly from the qcow2 image, and recover some files,
without the need to rollback the entire filesystem.

When the qcow2 infrastructure is ready, I will rebase my e2fsprogs snapshot patches,
implement e2image -x and consult you about using qcow2 snapshots.

Cheers,
Amir.

#10 lore raw
Quoted 65 lines · Amir Goldstein
Quoted 14 lines · Amir Goldstein
Quoted 6 lines · Lukas Czerner

You can export just the diff between some snapshots as you mentioned and then, when installing it on another filesystem the utility doing the job (e2image probably) would know what to do even without qcow2 snapshots, since all the data needed are in the image anyway, qcow2 snapshot is just useless abstraction we do not need in this case.

I agree. I was forcing my problem over this solution, trying to squeeze every last drop of coolness from qcow2 snapshots. it doesn't fit in here.

Still for the first problem (rollback) I think that qcow2 snapshots can be a perfect and simple solution.

Quoted 8 lines

e2image starts with exporting filesystem to full qcow2 image, then takes a qcow2 snapshot and proceeds writing blocks modified by latest snapshot and so forth until the oldest snapshot.

The resulting qcow2 snapshot list is an inversion of the Ext4 snapshots list and it gives you both the ability to rollback to any older snapshot and to undo all rollbacks by restoring the base image.

Hi Lukas,

I think I may have failed to explain my reasoning for using qcow2 snapshots properly and I understand your confusion.

Of course, if e2image would produce a full image of the filesystem, Ext4 snapshots information would be a part of the information encoded in the image.

For rollback to snapshot application, at some point, either on image creation or on image install, the snapshot diff patches (which are stored in the snapshot inodes) need to be decoded.

If the snapshot diffs are decoded on e2image install, there is no need for qcow2 snapshots, like you said.

My suggestion to decode the snapshot diffs on e2image create and to store them in qcow2 snapshots format
has 2 advantages over decoding on e2image install:

1. Should we want to implement create of snapshot image using e2image -x <snapid>,
the easiest implementation would involve creating a full filesystem image and then applying snapshot diffs on top of it. With this implementation, we can get the full filesystem image and an image of every snapshot on the way to <snapid> with no extra cost, if we just start a qcow2 snapshot before applying every snapshot diff.

2. At the moment, the only way to access next3 snapshots is to mount the filesystem
and then mount the snapshot via a loop device. Now if there is a problem to mount the filesystem
the snapshots are not accessible as well, which is a shame if you want to backup some data
before attempting to fix the filesystem.
Using the qcow2 exported image (with snapshots converted to qcow2 snapshots), it is possible
to mount every one of the snapshots, directly from the qcow2 image, and recover some files,
without the need to rollback the entire filesystem.

Quoted 3 lines

When the qcow2 infrastructure is ready, I will rebase my e2fsprogs snapshot patches,
implement e2image -x and consult you about using qcow2 snapshots.

Fair enough :).

-Lukas

Quoted 4 lines

Cheers,
Amir.

#11 lore raw
In reply to Lukas Czerner
Quoted 4 lines · Lukas Czerner

I am presenting you an initial parch, which works for me quite nicely, however so far it did received just limited testing. Nevertheless you can try it simply by invoking e2image with '-q' parameter like this:

e2image -q /dev/sda1 myimage

Probably "-q" isn't the best name, since that normally means "quiet". Maybe "-Q"?

Cheers, Andreas

#12 lore raw
Quoted 10 lines · Andreas Dilger
Quoted 4 lines · Lukas Czerner

I am presenting you an initial parch, which works for me quite nicely, however so far it did received just limited testing. Nevertheless you can try it simply by invoking e2image with '-q' parameter like this:

e2image -q /dev/sda1 myimage

Probably "-q" isn't the best name, since that normally means "quiet". Maybe "-Q"?

Cheers, Andreas

Fair enough, I can change it to "-Q".

Thanks!

-Lukas