qcow2: persistent dirty bitmaps
v7 cover qcow2: persistent dirty bitmaps
v7:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v7
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
- a lot of refactoring and reordering of patches.
- dead code removed (bdrv_dirty_bitmap_load, etc.)
- do not maintain extra data for now
- do not store dirty bitmap directory in memory (as we use it seldom, we can just reread if needed)
By Kevin's review:
01 - commit message changed: fix->improvement (as it was not a bug) 03 - r-b
04 - r-b
05 - add 21 patch to fix spec, also, removed all (I hope) mentions of
"Bitmap Header", switch to one unified name for it - "Bitmap Directory Entry", to avoid misunderstanding with Qcow2 header. (also, add patch 22, to fix it in spec)
v6.06 - improve for_each_dir_entry loop, reorder patches, other small fixes v6.07 ~> v7.09 - dead code removed, I've moved to one function
.bdrv_store_persistent_bitmaps and have wrapper and callback in one patch (with also some other staff. If it not ok, I can split them)
v6.08 - about keeping bitmap directory instead of bitmap list: no I don't keep
it at all.
v6.16 - old bdrv_ bitmap-storing related functions are removed. The new one is
bdrv_store_persistent_bitmaps.
v6:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v6
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
There are a lot of changes, reorderings and additions in comparement with v5. One principal thing: now bitmaps are removed from image after loading instead of marking them in_use. It is simpler and we do not need to store superfluous data. Also, we are no more interested in command line interface to dirty bitmaps. So it is dropped. If someone needs it I can add it later.
Vladimir Sementsov-Ogievskiy (22):
hbitmap: improve dirty iter
tests: add hbitmap iter test
block: fix bdrv_dirty_bitmap_granularity signature block/dirty-bitmap: add deserialize_ones func
qcow2-bitmap: structs and consts
qcow2: add dirty bitmaps extension
qcow2-bitmap: introduce auto-loading bitmaps block/dirty-bitmap: add autoload field to BdrvDirtyBitmap
block: introduce persistent dirty bitmaps block/dirty-bitmap: add bdrv_dirty_bitmap_next()
qcow2-bitmap: add qcow2_store_persistent_bitmaps()
qcow2-bitmap: add IN_USE flag
qcow2-bitmap: check constraints
qcow2: delete bitmaps on truncate
qcow2-bitmap: add autoclear bit
qmp: add persistent flag to block-dirty-bitmap-add
qmp: add autoload parameter to block-dirty-bitmap-add
qapi: add md5 checksum of last dirty bitmap level to query-block
iotests: test qcow2 persistent dirty bitmap
qcow2-dirty-bitmap: refcounts
specs/qcow2: fix bitmap granularity qemu-specific note specs/qcow2: do not use wording 'bitmap header'
block.c | 30 ++ block/Makefile.objs | 2 +- block/dirty-bitmap.c | 59 ++- block/qcow2-bitmap.c | 963 +++++++++++++++++++++++++++++++++++++++++++ block/qcow2-refcount.c | 14 +- block/qcow2.c | 105 ++++- block/qcow2.h | 54 +++ blockdev.c | 30 +- docs/specs/qcow2.txt | 8 +- include/block/block.h | 2 + include/block/block_int.h | 2 + include/block/dirty-bitmap.h | 16 +- include/qemu/hbitmap.h | 47 ++- qapi/block-core.json | 17 +- qmp-commands.hx | 8 +- tests/qemu-iotests/165 | 87 ++++ tests/qemu-iotests/165.out | 5 + tests/qemu-iotests/group | 1 + tests/test-hbitmap.c | 19 + util/hbitmap.c | 48 ++- 20 files changed, 1475 insertions(+), 42 deletions(-) create mode 100644 block/qcow2-bitmap.c create mode 100755 tests/qemu-iotests/165 create mode 100644 tests/qemu-iotests/165.out
Signature · 1 line
1.8.3.1
PATCH v7 1/22 hbitmap: improve dirty iter
Make dirty iter resistant to resetting bits in corresponding HBitmap.
--- include/qemu/hbitmap.h | 24 ++---------------------- util/hbitmap.c | 23 ++++++++++++++++++++++- 2 files changed, 24 insertions(+), 23 deletions(-)
include/qemu/hbitmap.h +2 -22
@@ -243,10 +243,7 @@ void hbitmap_free(HBitmap *hb);* the lowest-numbered bit that is set in @hb, starting at @first. * * Concurrent setting of bits is acceptable, and will at worst cause the -* iteration to miss some of those bits. Resetting bits before the current -* position of the iterator is also okay. However, concurrent resetting of -* bits can lead to unexpected behavior if the iterator has not yet reached -* those bits. +* iteration to miss some of those bits. Concurrent bits resetting is ok too. */ void hbitmap_iter_init (HBitmapIter *hbi,const HBitmap *hb,uint64_t first);
@@ -285,24 +282,7 @@ void hbitmap_free_meta(HBitmap *hb);* Return the next bit that is set in @hbi's associated HBitmap, * or -1 if all remaining bits are zero. */ -static inline int64_t hbitmap_iter_next (HBitmapIter *hbi) -{ -unsigned long cur = hbi->cur[HBITMAP_LEVELS -1 ]; -int64_t item; - -if (cur ==0 ) { - cur =hbitmap_iter_skip_words (hbi); -if (cur ==0 ) { -return -1 ; - } - } - -/* The next call will resume work from the next bit. */ - hbi->cur[HBITMAP_LEVELS -1 ] = cur & (cur -1 ); - item = ((uint64_t )hbi->pos <<BITS_PER_LEVEL ) +ctzl (cur); - -return item << hbi->granularity; -} +int64_t hbitmap_iter_next (HBitmapIter *hbi);/** * hbitmap_iter_next_word:
util/hbitmap.c +22 -1
@@ -106,8 +106,9 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)unsigned long cur;do { - cur = hbi->cur[--i]; + i--; pos >>=BITS_PER_LEVEL ; + cur = hbi->cur[i] & hb->levels[i][pos]; }while (cur ==0 );/* Check for end of iteration. We always use fewer than BITS_PER_LONG
@@ -139,6 +140,26 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)return cur; } +int64_t hbitmap_iter_next (HBitmapIter *hbi) +{ +unsigned long cur = hbi->cur[HBITMAP_LEVELS -1 ] & + hbi->hb->levels[HBITMAP_LEVELS -1 ][hbi->pos]; +int64_t item; + +if (cur ==0 ) { + cur =hbitmap_iter_skip_words (hbi); +if (cur ==0 ) { +return -1 ; + } + } + +/* The next call will resume work from the next bit. */ + hbi->cur[HBITMAP_LEVELS -1 ] = cur & (cur -1 ); + item = ((uint64_t )hbi->pos <<BITS_PER_LEVEL ) +ctzl (cur); + +return item << hbi->granularity; +} +void hbitmap_iter_init (HBitmapIter *hbi,const HBitmap *hb,uint64_t first) {unsigned i, bit;
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 01/22] hbitmap: improve dirty iter
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
Make dirty iter resistant to resetting bits in corresponding HBitmap.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/qemu/hbitmap.h | 24 ++---------------------- util/hbitmap.c | 23 ++++++++++++++++++++++- 2 files changed, 24 insertions(+), 23 deletions(-)include/qemu/hbitmap.h +0
-0
include/qemu/hbitmap.h void hbitmap_free(HBitmap *hb); · line 246
Quoted 8 lines
include/qemu/hbitmap.h +1
-4@@ -243,10 +243,7 @@ void hbitmap_free(HBitmap *hb);* the lowest-numbered bit that is set in @hb, starting at @first. * * Concurrent setting of bits is acceptable, and will at worst cause the -* iteration to miss some of those bits. Resetting bits before the current -* position of the iterator is also okay. However, concurrent resetting of -* bits can lead to unexpected behavior if the iterator has not yet reached -* those bits. +* iteration to miss some of those bits. Concurrent bits resetting is ok too.
I'd prefer:
Concurrent resetting of bits is OK, too.
Or:
Concurrently resetting bits is OK, too.
Or:
Resetting bits concurrently is OK, too.
With or without that change:
Quoted 72 lines
include/qemu/hbitmap.h +1
-18*/ void hbitmap_iter_init (HBitmapIter *hbi,const HBitmap *hb,uint64_t first);@@ -285,24 +282,7 @@ void hbitmap_free_meta(HBitmap *hb);* Return the next bit that is set in @hbi's associated HBitmap, * or -1 if all remaining bits are zero. */ -static inline int64_t hbitmap_iter_next (HBitmapIter *hbi) -{ -unsigned long cur = hbi->cur[HBITMAP_LEVELS -1 ]; -int64_t item; - -if (cur ==0 ) { - cur =hbitmap_iter_skip_words (hbi); -if (cur ==0 ) { -return -1 ; - } - } - -/* The next call will resume work from the next bit. */ - hbi->cur[HBITMAP_LEVELS -1 ] = cur & (cur -1 ); - item = ((uint64_t )hbi->pos <<BITS_PER_LEVEL ) +ctzl (cur); - -return item << hbi->granularity; -} +int64_t hbitmap_iter_next (HBitmapIter *hbi);/** * hbitmap_iter_next_word: util/hbitmap.c +22
-1@@ -106,8 +106,9 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)unsigned long cur;do { - cur = hbi->cur[--i]; + i--; pos >>=BITS_PER_LEVEL ; + cur = hbi->cur[i] & hb->levels[i][pos]; }while (cur ==0 );/* Check for end of iteration. We always use fewer than BITS_PER_LONG @@ -139,6 +140,26 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)return cur; } +int64_t hbitmap_iter_next (HBitmapIter *hbi) +{ +unsigned long cur = hbi->cur[HBITMAP_LEVELS -1 ] & + hbi->hb->levels[HBITMAP_LEVELS -1 ][hbi->pos]; +int64_t item; + +if (cur ==0 ) { + cur =hbitmap_iter_skip_words (hbi); +if (cur ==0 ) { +return -1 ; + } + } + +/* The next call will resume work from the next bit. */ + hbi->cur[HBITMAP_LEVELS -1 ] = cur & (cur -1 ); + item = ((uint64_t )hbi->pos <<BITS_PER_LEVEL ) +ctzl (cur); + +return item << hbi->granularity; +} +void hbitmap_iter_init (HBitmapIter *hbi,const HBitmap *hb,uint64_t first) {unsigned i, bit;
PATCH v7 2/22 tests: add hbitmap iter test
In reply to Vladimir Sementsov-OgievskiyTest that hbitmap iter is resistant to bitmap resetting.
--- tests/test-hbitmap.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
tests/test-hbitmap.c +19 -0
@@ -881,6 +881,22 @@ static void hbitmap_test_add(const char *testpath, hbitmap_test_teardown); } +static void test_hbitmap_iter_and_reset (TestHBitmapData *data, +const void *unused) +{ +HBitmapIter hbi; + +hbitmap_test_init (data,L1 *2 ,0 ); +hbitmap_set (data->hb,0 , data->size); + +hbitmap_iter_init (&hbi, data->hb,BITS_PER_LONG -1 ); + +hbitmap_iter_next (&hbi); + +hbitmap_reset_all (data->hb); +hbitmap_iter_next (&hbi); +} +int main (int argc,char **argv) {g_test_init (&argc, &argv,NULL );
@@ -938,6 +954,9 @@ int main(int argc, char **argv) test_hbitmap_serialize_part);hbitmap_test_add ("/hbitmap/serialize/zeroes" , test_hbitmap_serialize_zeroes); + +hbitmap_test_add ("/hbitmap/iter/iter_and_reset" , + test_hbitmap_iter_and_reset);g_test_run ();return 0 ;
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 02/22] tests: add hbitmap iter test
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
Test that hbitmap iter is resistant to bitmap resetting.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- tests/test-hbitmap.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
PATCH v7 3/22 block: fix bdrv_dirty_bitmap_granularity signature
In reply to Vladimir Sementsov-OgievskiyMake getter signature const-correct. This allows other functions with const dirty bitmap parameter use bdrv_dirty_bitmap_granularity().
--- block/dirty-bitmap.c | 2 +- include/block/dirty-bitmap.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
block/dirty-bitmap.c +1 -1
@@ -388,7 +388,7 @@ uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs)return granularity; } -uint32_t bdrv_dirty_bitmap_granularity (BdrvDirtyBitmap *bitmap) +uint32_t bdrv_dirty_bitmap_granularity (const BdrvDirtyBitmap *bitmap) {return BDRV_SECTOR_SIZE <<hbitmap_granularity (bitmap->bitmap); }
include/block/dirty-bitmap.h +1 -1
@@ -29,7 +29,7 @@ void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap);void bdrv_enable_dirty_bitmap (BdrvDirtyBitmap *bitmap);BlockDirtyInfoList *bdrv_query_dirty_bitmaps (BlockDriverState *bs);uint32_t bdrv_get_default_bitmap_granularity (BlockDriverState *bs); -uint32_t bdrv_dirty_bitmap_granularity (BdrvDirtyBitmap *bitmap); +uint32_t bdrv_dirty_bitmap_granularity (const BdrvDirtyBitmap *bitmap);uint32_t bdrv_dirty_bitmap_meta_granularity (BdrvDirtyBitmap *bitmap);bool bdrv_dirty_bitmap_enabled (BdrvDirtyBitmap *bitmap);bool bdrv_dirty_bitmap_frozen (BdrvDirtyBitmap *bitmap);
Signature · 1 line
1.8.3.1
PATCH v7 4/22 block/dirty-bitmap: add deserialize_ones func
In reply to Vladimir Sementsov-OgievskiyAdd bdrv_dirty_bitmap_deserialize_ones() function, which is needed for qcow2 bitmap loading, to handle unallocated bitmap parts, marked as all-ones.
--- block/dirty-bitmap.c | 7 +++++++ include/block/dirty-bitmap.h | 3 +++ include/qemu/hbitmap.h | 15 +++++++++++++++ util/hbitmap.c | 17 +++++++++++++++++ 4 files changed, 42 insertions(+)
block/dirty-bitmap.c +7 -0
@@ -499,6 +499,13 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,hbitmap_deserialize_zeroes (bitmap->bitmap, start, count, finish); } +void bdrv_dirty_bitmap_deserialize_ones (BdrvDirtyBitmap *bitmap, +uint64_t start,uint64_t count, +bool finish) +{ +hbitmap_deserialize_ones (bitmap->bitmap, start, count, finish); +} +void bdrv_dirty_bitmap_deserialize_finish (BdrvDirtyBitmap *bitmap) {hbitmap_deserialize_finish (bitmap->bitmap);
include/block/dirty-bitmap.h +3 -0
@@ -70,6 +70,9 @@ void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap,void bdrv_dirty_bitmap_deserialize_zeroes (BdrvDirtyBitmap *bitmap,uint64_t start,uint64_t count,bool finish); +void bdrv_dirty_bitmap_deserialize_ones (BdrvDirtyBitmap *bitmap, +uint64_t start,uint64_t count, +bool finish);void bdrv_dirty_bitmap_deserialize_finish (BdrvDirtyBitmap *bitmap);#endif
include/qemu/hbitmap.h +15 -0
@@ -216,6 +216,21 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count, bool finish);/** +* hbitmap_deserialize_ones +* @hb: HBitmap to operate on. +* @start: First bit to restore. +* @count: Number of bits to restore. +* @finish: Whether to call hbitmap_deserialize_finish automatically. +* +* Fills the bitmap with ones. +* +* If @finish is false, caller must call hbitmap_serialize_finish before using +* the bitmap. +*/ +void hbitmap_deserialize_ones (HBitmap *hb,uint64_t start,uint64_t count, +bool finish); + +/** * hbitmap_deserialize_finish * @hb: HBitmap to operate on. *
util/hbitmap.c +17 -0
@@ -531,6 +531,23 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count, } } +void hbitmap_deserialize_ones (HBitmap *hb,uint64_t start,uint64_t count, +bool finish) +{ +uint64_t el_count; +unsigned long *first; + +if (!count) { +return ; + } +serialization_chunk (hb, start, count, &first, &el_count); + +memset (first,0xff , el_count *sizeof (unsigned long )); +if (finish) { +hbitmap_deserialize_finish (hb); + } +} +void hbitmap_deserialize_finish (HBitmap *bitmap) {int64_t i, size, prev_size;
Signature · 1 line
1.8.3.1
PATCH v7 5/22 qcow2-bitmap: structs and consts
In reply to Vladimir Sementsov-OgievskiyCreate block/qcow2-bitmap.c
Add data structures and constraints accordingly to docs/specs/qcow2.txt
--- block/Makefile.objs | 2 +- block/qcow2-bitmap.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+), 1 deletion(-) create mode 100644 block/qcow2-bitmap.c
block/Makefile.objs +1 -1
@@ -1,5 +1,5 @@ block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o dmg.o -block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o +block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-bitmap.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.o
block/qcow2-bitmap.c new file +47 -0
47 lines not on this page · raw message
block/qcow2.h +29 -0
@@ -52,6 +52,10 @@* space for snapshot names and IDs */ #define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS) +/* Bitmap header extension constraints */ +#define QCOW_MAX_DIRTY_BITMAPS 65535 +#define QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE (1024 * QCOW_MAX_DIRTY_BITMAPS) +/* indicate that the refcount of the referenced cluster is exactly one. */ #define QCOW_OFLAG_COPIED (1ULL << 63)/* indicate that the cluster is compressed (they never have the copied flag) */
@@ -142,6 +146,22 @@ typedef struct QEMU_PACKED QCowSnapshotHeader {/* name follows */ } QCowSnapshotHeader; +/* Qcow2BitmapDirEntry is actually a bitmap directory entry */ +typedef struct QEMU_PACKED Qcow2BitmapDirEntry { +/* header is 8 byte aligned */ +uint64_t bitmap_table_offset; + +uint32_t bitmap_table_size; +uint32_t flags; + +uint8_t type; +uint8_t granularity_bits; +uint16_t name_size; +uint32_t extra_data_size; +/* extra data follows */ +/* name follows */ +} Qcow2BitmapDirEntry; + typedef struct QEMU_PACKED QCowSnapshotExtraData {uint64_t vm_state_size_large;uint64_t disk_size;
@@ -222,6 +242,15 @@ typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,typedef void Qcow2SetRefcountFunc (void *refcount_array,uint64_t index,uint64_t value); +/* Be careful, Qcow2BitmapHeaderExt is not an extension of Qcow2BitmapDirEntry, it +* is Qcow2 header extension */ +typedef struct Qcow2BitmapHeaderExt { +uint32_t nb_bitmaps; +uint32_t reserved32; +uint64_t bitmap_directory_size; +uint64_t bitmap_directory_offset; +} QEMU_PACKED Qcow2BitmapHeaderExt; + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size;
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 05/22] qcow2-bitmap: structs and consts
Quoted 52 lines · Vladimir Sementsov-Ogievskiy
Create
block/qcow2-bitmap.c
Add data structures and constraints accordingly todocs/specs/qcow2.txtSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/Makefile.objs | 2 +- block/qcow2-bitmap.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+), 1 deletion(-) create mode 100644 block/qcow2-bitmap.cblock/Makefile.objs +1
-1@@ -1,5 +1,5 @@ block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o dmg.o -block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o +block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-bitmap.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.oblock/qcow2-bitmap.c new file +24
-0@@ -0,0 +1,47 @@ +/* +* Bitmaps for the QCOW version 2 format +* +* Copyright (c) 2014-2016 Vladimir Sementsov-Ogievskiy +* +* This file is derived from qcow2-snapshot.c, original copyright: +* Copyright (c) 2004-2006 Fabrice Bellard +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
block/qcow2-bitmap.c line 34
Quoted 10 lines
block/qcow2-bitmap.c new file +10
-0+* THE SOFTWARE. +*/ + +/* NOTICE: BME here means Bitmaps Extension and used as a namespace for +* _internal_ constants. Please do not use this _internal_ abbreviation for +* other needs and/or outside of this file. */ + +/* Bitmap directory entry constraints */ +#define BME_MAX_TABLE_SIZE 0x8000000 +#define BME_MAX_PHYS_SIZE 0x20000000/* 512 mb */
I suppose BME_MAX_TABLE_SIZE (8M) is greater than BME_MAX_PHYS_SIZE (512 MB) divided by the cluster size (>= 512; 512 MB / cluster_size <= 1 MB) because fully zero or one clusters do not require any physical space?
Makes some sense, but I can see that this might make give some trouble when trying to serialize overly large bitmaps. But I guess that comes later in this series, so I'll wait for that point.
Another thing is that 512 MB is rather big. It gets worse: The bitmap may only require 512 MB on disk, but with a maximum table size of 8 MB, it can require up to 8M * cluster_size in memory (with just 64 MB of disk space!) by using the "read as all zeroes" or "read as all ones" flags. With the default cluster size of 64 kB, this would be 512 GB in RAM. That sounds bad to me.
Well, it is probably fine as long as the bitmap is not auto-loaded... But we do have a flag for exactly that. So it seems to me that a manipulated image can easily consume huge amounts of RAM on the host.
So I think we also need some sane limitation on the in-RAM size of a bitmap (which is BME_MAX_TABLE_SIZE * cluster_size, as far as I understand). The question of course is, what is sane? For a server system with no image manipulation possible from the outside, 1 GB may be completely fine. But imagine you download some qcow2 image to your laptop. Then, 1 GB may not be fine, actually.
Maybe it would make sense to use a runtime-adjustable limit here?
block/qcow2-bitmap.c line 43
Quoted 9 lines
block/qcow2-bitmap.c new file +9
-0+#define BME_MAX_GRANULARITY_BITS 31 +#define BME_MIN_GRANULARITY_BITS 9 +#define BME_MAX_NAME_SIZE 1023 + +/* Bitmap directory entry flags */ +#define BME_RESERVED_FLAGS 0xffffffff + +/* bits [1, 8] U [56, 63] are reserved */ +#define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001fe
ull suffix is missing.
Quoted 20 lines
block/qcow2-bitmap.c new file +4
-0+ +typedef enum BitmapType { +BT_DIRTY_TRACKING_BITMAP =1 +}BitmapType ;block/qcow2.h +4
-0@@ -52,6 +52,10 @@* space for snapshot names and IDs */ #define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS) +/* Bitmap header extension constraints */ +#define QCOW_MAX_DIRTY_BITMAPS 65535 +#define QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE (1024 * QCOW_MAX_DIRTY_BITMAPS) +/* indicate that the refcount of the referenced cluster is exactly one. */ #define QCOW_OFLAG_COPIED (1ULL << 63)/* indicate that the cluster is compressed (they never have the copied flag) */
block/qcow2.h typedef struct QEMU_PACKED QCowSnapshotHeader { · line 149
Quoted 4 lines
block/qcow2.h +1
-0@@ -142,6 +146,22 @@ typedef struct QEMU_PACKED QCowSnapshotHeader {/* name follows */ } QCowSnapshotHeader; +/* Qcow2BitmapDirEntry is actually a bitmap directory entry */
This comment doesn't make a whole lot of sense now that this structure is indeed called what it "actually is". ;-)
Quoted 19 lines
block/qcow2.h +15
-0+typedef struct QEMU_PACKED Qcow2BitmapDirEntry { +/* header is 8 byte aligned */ +uint64_t bitmap_table_offset; + +uint32_t bitmap_table_size; +uint32_t flags; + +uint8_t type; +uint8_t granularity_bits; +uint16_t name_size; +uint32_t extra_data_size; +/* extra data follows */ +/* name follows */ +} Qcow2BitmapDirEntry; + typedef struct QEMU_PACKED QCowSnapshotExtraData {uint64_t vm_state_size_large;uint64_t disk_size;
block/qcow2.h typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array, · line 246
Quoted 5 lines
block/qcow2.h +2
-0@@ -222,6 +242,15 @@ typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,typedef void Qcow2SetRefcountFunc (void *refcount_array,uint64_t index,uint64_t value); +/* Be careful, Qcow2BitmapHeaderExt is not an extension of Qcow2BitmapDirEntry, it +* is Qcow2 header extension */
And this makes even less sense now.
(These comments don't stop me from giving an R-b, but I'm not so sure about the constants...)
Max
Quoted 11 lines
block/qcow2.h +7
-0+typedef struct Qcow2BitmapHeaderExt { +uint32_t nb_bitmaps; +uint32_t reserved32; +uint64_t bitmap_directory_size; +uint64_t bitmap_directory_offset; +} QEMU_PACKED Qcow2BitmapHeaderExt; + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size;
Re: [Qemu-devel] [PATCH 05/22] qcow2-bitmap: structs and consts
Quoted 90 lines · Max Reitz
Quoted 52 lines · Vladimir Sementsov-Ogievskiy
Create
block/qcow2-bitmap.c
Add data structures and constraints accordingly todocs/specs/qcow2.txtSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/Makefile.objs | 2 +- block/qcow2-bitmap.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+), 1 deletion(-) create mode 100644 block/qcow2-bitmap.cblock/Makefile.objs +1
-1@@ -1,5 +1,5 @@ block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o dmg.o -block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o +block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-bitmap.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.oblock/qcow2-bitmap.c new file +24
-0@@ -0,0 +1,47 @@ +/* +* Bitmaps for the QCOW version 2 format +* +* Copyright (c) 2014-2016 Vladimir Sementsov-Ogievskiy +* +* This file is derived from qcow2-snapshot.c, original copyright: +* Copyright (c) 2004-2006 Fabrice Bellard +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN block/qcow2-bitmap.c line 34
Quoted 10 lines
block/qcow2-bitmap.c new file +10
-0+* THE SOFTWARE. +*/ + +/* NOTICE: BME here means Bitmaps Extension and used as a namespace for +* _internal_ constants. Please do not use this _internal_ abbreviation for +* other needs and/or outside of this file. */ + +/* Bitmap directory entry constraints */ +#define BME_MAX_TABLE_SIZE 0x8000000 +#define BME_MAX_PHYS_SIZE 0x20000000/* 512 mb */ I suppose
BME_MAX_TABLE_SIZE(8M) is greater thanBME_MAX_PHYS_SIZE(512 MB) divided by the cluster size (>= 512; 512 MB /cluster_size<= 1 MB) because fully zero or one clusters do not require any physical space?Makes some sense, but I can see that this might make give some trouble when trying to serialize overly large bitmaps. But I guess that comes later in this series, so I'll wait for that point.
Another thing is that 512 MB is rather big. It gets worse: The bitmap may only require 512 MB on disk, but with a maximum table size of 8 MB, it can require up to 8M *
cluster_sizein memory (with just 64 MB of disk space!) by using the "read as all zeroes" or "read as all ones" flags. With the default cluster size of 64 kB, this would be 512 GB in RAM. That sounds bad to me.Well, it is probably fine as long as the bitmap is not auto-loaded... But we do have a flag for exactly that. So it seems to me that a manipulated image can easily consume huge amounts of RAM on the host.
So I think we also need some sane limitation on the in-RAM size of a bitmap (which is
BME_MAX_TABLE_SIZE*cluster_size, as far as I understand). The question of course is, what is sane? For a server system with no image manipulation possible from the outside, 1 GB may be completely fine. But imagine you download some qcow2 image to your laptop. Then, 1 GB may not be fine, actually.
Quoted 8 lines
Maybe it would make sense to use a runtime-adjustable limit here?
block/qcow2-bitmap.c line 40
Quoted 6 lines
block/qcow2-bitmap.c new file +6
-0+#define BME_MAX_GRANULARITY_BITS 31 +#define BME_MIN_GRANULARITY_BITS 9 +#define BME_MAX_NAME_SIZE 1023 + +/* Bitmap directory entry flags */ +#define BME_RESERVED_FLAGS 0xffffffff
Oh, and this should probably get a u suffix.
Max
Quoted 78 lines
block/qcow2-bitmap.c line 43
Quoted 3 lines
block/qcow2-bitmap.c new file +3
-0+ +/* bits [1, 8] U [56, 63] are reserved */ +#define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001feull suffix is missing.
Quoted 20 lines
block/qcow2-bitmap.c new file +4
-0+ +typedef enum BitmapType { +BT_DIRTY_TRACKING_BITMAP =1 +}BitmapType ;block/qcow2.h +4
-0@@ -52,6 +52,10 @@* space for snapshot names and IDs */ #define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS) +/* Bitmap header extension constraints */ +#define QCOW_MAX_DIRTY_BITMAPS 65535 +#define QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE (1024 * QCOW_MAX_DIRTY_BITMAPS) +/* indicate that the refcount of the referenced cluster is exactly one. */ #define QCOW_OFLAG_COPIED (1ULL << 63)/* indicate that the cluster is compressed (they never have the copied flag) */ block/qcow2.h typedef struct QEMU_PACKED QCowSnapshotHeader { · line 149
Quoted 4 lines
block/qcow2.h +1
-0@@ -142,6 +146,22 @@ typedef struct QEMU_PACKED QCowSnapshotHeader {/* name follows */ } QCowSnapshotHeader; +/* Qcow2BitmapDirEntry is actually a bitmap directory entry */ This comment doesn't make a whole lot of sense now that this structure is indeed called what it "actually is". ;-)
Quoted 19 lines
block/qcow2.h +15
-0+typedef struct QEMU_PACKED Qcow2BitmapDirEntry { +/* header is 8 byte aligned */ +uint64_t bitmap_table_offset; + +uint32_t bitmap_table_size; +uint32_t flags; + +uint8_t type; +uint8_t granularity_bits; +uint16_t name_size; +uint32_t extra_data_size; +/* extra data follows */ +/* name follows */ +} Qcow2BitmapDirEntry; + typedef struct QEMU_PACKED QCowSnapshotExtraData {uint64_t vm_state_size_large;uint64_t disk_size;block/qcow2.h typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array, · line 246
Quoted 5 lines
block/qcow2.h +2
-0@@ -222,6 +242,15 @@ typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,typedef void Qcow2SetRefcountFunc (void *refcount_array,uint64_t index,uint64_t value); +/* Be careful, Qcow2BitmapHeaderExt is not an extension of Qcow2BitmapDirEntry, it +* is Qcow2 header extension */ And this makes even less sense now.
(These comments don't stop me from giving an R-b, but I'm not so sure about the constants...)
Max
Quoted 11 lines
block/qcow2.h +7
-0+typedef struct Qcow2BitmapHeaderExt { +uint32_t nb_bitmaps; +uint32_t reserved32; +uint64_t bitmap_directory_size; +uint64_t bitmap_directory_offset; +} QEMU_PACKED Qcow2BitmapHeaderExt; + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size;
Re: [Qemu-devel] [PATCH 05/22] qcow2-bitmap: structs and consts
In reply to Max ReitzQuoted 71 lines · Max Reitz
Quoted 52 lines · Vladimir Sementsov-Ogievskiy
Create
block/qcow2-bitmap.c
Add data structures and constraints accordingly todocs/specs/qcow2.txtSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/Makefile.objs | 2 +- block/qcow2-bitmap.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+), 1 deletion(-)create mode 100644
block/qcow2-bitmap.cblock/Makefile.objs +1
-1@@ -1,5 +1,5 @@ block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o dmg.o -block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o +block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-bitmap.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.oblock/qcow2-bitmap.c new file +24
-0@@ -0,0 +1,47 @@ +/* +* Bitmaps for the QCOW version 2 format +* +* Copyright (c) 2014-2016 Vladimir Sementsov-Ogievskiy +* +* This file is derived from qcow2-snapshot.c, original copyright: +* Copyright (c) 2004-2006 Fabrice Bellard +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN block/qcow2-bitmap.c line 34
Quoted 10 lines
block/qcow2-bitmap.c new file +10
-0+* THE SOFTWARE. +*/ + +/* NOTICE: BME here means Bitmaps Extension and used as a namespace for +* _internal_ constants. Please do not use this _internal_ abbreviation for +* other needs and/or outside of this file. */ + +/* Bitmap directory entry constraints */ +#define BME_MAX_TABLE_SIZE 0x8000000 +#define BME_MAX_PHYS_SIZE 0x20000000/* 512 mb */ I suppose
BME_MAX_TABLE_SIZE(8M) is greater thanBME_MAX_PHYS_SIZE(512 MB) divided by the cluster size (>= 512; 512 MB /cluster_size<= 1 MB) because fully zero or one clusters do not require any physical space?Makes some sense, but I can see that this might make give some trouble when trying to serialize overly large bitmaps. But I guess that comes later in this series, so I'll wait for that point.
Quoted 6 lines
Another thing is that 512 MB is rather big. It gets worse: The bitmap may only require 512 MB on disk, but with a maximum table size of 8 MB, it can require up to 8M *
cluster_sizein memory (with just 64 MB of disk space!) by using the "read as all zeroes" or "read as all ones" flags. With the default cluster size of 64 kB, this would be 512 GB in RAM. That sounds bad to me.
This will not be so much if we introduce sparse bitmaps in qemu. But for current realization you are right, this should be restricted more carefully and noted in spec as 'Note: Qemu currently supports only...'
Quoted 94 lines
Well, it is probably fine as long as the bitmap is not auto-loaded... But we do have a flag for exactly that. So it seems to me that a manipulated image can easily consume huge amounts of RAM on the host.
So I think we also need some sane limitation on the in-RAM size of a bitmap (which is
BME_MAX_TABLE_SIZE*cluster_size, as far as I understand). The question of course is, what is sane? For a server system with no image manipulation possible from the outside, 1 GB may be completely fine. But imagine you download some qcow2 image to your laptop. Then, 1 GB may not be fine, actually.Maybe it would make sense to use a runtime-adjustable limit here?
block/qcow2-bitmap.c line 43
Quoted 9 lines
block/qcow2-bitmap.c new file +9
-0+#define BME_MAX_GRANULARITY_BITS 31 +#define BME_MIN_GRANULARITY_BITS 9 +#define BME_MAX_NAME_SIZE 1023 + +/* Bitmap directory entry flags */ +#define BME_RESERVED_FLAGS 0xffffffff + +/* bits [1, 8] U [56, 63] are reserved */ +#define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001feull suffix is missing.
Quoted 20 lines
block/qcow2-bitmap.c new file +4
-0+ +typedef enum BitmapType { +BT_DIRTY_TRACKING_BITMAP =1 +}BitmapType ;block/qcow2.h +4
-0@@ -52,6 +52,10 @@* space for snapshot names and IDs */ #define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS) +/* Bitmap header extension constraints */ +#define QCOW_MAX_DIRTY_BITMAPS 65535 +#define QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE (1024 * QCOW_MAX_DIRTY_BITMAPS) +/* indicate that the refcount of the referenced cluster is exactly one. */ #define QCOW_OFLAG_COPIED (1ULL << 63)/* indicate that the cluster is compressed (they never have the copied flag) */ block/qcow2.h typedef struct QEMU_PACKED QCowSnapshotHeader { · line 149
Quoted 4 lines
block/qcow2.h +1
-0@@ -142,6 +146,22 @@ typedef struct QEMU_PACKED QCowSnapshotHeader {/* name follows */ } QCowSnapshotHeader; +/* Qcow2BitmapDirEntry is actually a bitmap directory entry */ This comment doesn't make a whole lot of sense now that this structure is indeed called what it "actually is". ;-)
Quoted 19 lines
block/qcow2.h +15
-0+typedef struct QEMU_PACKED Qcow2BitmapDirEntry { +/* header is 8 byte aligned */ +uint64_t bitmap_table_offset; + +uint32_t bitmap_table_size; +uint32_t flags; + +uint8_t type; +uint8_t granularity_bits; +uint16_t name_size; +uint32_t extra_data_size; +/* extra data follows */ +/* name follows */ +} Qcow2BitmapDirEntry; + typedef struct QEMU_PACKED QCowSnapshotExtraData {uint64_t vm_state_size_large;uint64_t disk_size;block/qcow2.h typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array, · line 246
Quoted 5 lines
block/qcow2.h +2
-0@@ -222,6 +242,15 @@ typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,typedef void Qcow2SetRefcountFunc (void *refcount_array,uint64_t index,uint64_t value); +/* Be careful, Qcow2BitmapHeaderExt is not an extension of Qcow2BitmapDirEntry, it +* is Qcow2 header extension */ And this makes even less sense now.
(These comments don't stop me from giving an R-b, but I'm not so sure about the constants...)
Max
Quoted 11 lines
block/qcow2.h +7
-0+typedef struct Qcow2BitmapHeaderExt { +uint32_t nb_bitmaps; +uint32_t reserved32; +uint64_t bitmap_directory_size; +uint64_t bitmap_directory_offset; +} QEMU_PACKED Qcow2BitmapHeaderExt; + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size;
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 05/22] qcow2-bitmap: structs and consts
In reply to Max ReitzQuoted 82 lines · Max Reitz
Quoted 52 lines · Vladimir Sementsov-Ogievskiy
Create
block/qcow2-bitmap.c
Add data structures and constraints accordingly todocs/specs/qcow2.txtSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/Makefile.objs | 2 +- block/qcow2-bitmap.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+), 1 deletion(-)create mode 100644
block/qcow2-bitmap.cblock/Makefile.objs +1
-1@@ -1,5 +1,5 @@ block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o dmg.o -block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o +block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-bitmap.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.oblock/qcow2-bitmap.c new file +24
-0@@ -0,0 +1,47 @@ +/* +* Bitmaps for the QCOW version 2 format +* +* Copyright (c) 2014-2016 Vladimir Sementsov-Ogievskiy +* +* This file is derived from qcow2-snapshot.c, original copyright: +* Copyright (c) 2004-2006 Fabrice Bellard +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN block/qcow2-bitmap.c line 34
Quoted 10 lines
block/qcow2-bitmap.c new file +10
-0+* THE SOFTWARE. +*/ + +/* NOTICE: BME here means Bitmaps Extension and used as a namespace for +* _internal_ constants. Please do not use this _internal_ abbreviation for +* other needs and/or outside of this file. */ + +/* Bitmap directory entry constraints */ +#define BME_MAX_TABLE_SIZE 0x8000000 +#define BME_MAX_PHYS_SIZE 0x20000000/* 512 mb */ I suppose
BME_MAX_TABLE_SIZE(8M) is greater thanBME_MAX_PHYS_SIZE(512 MB) divided by the cluster size (>= 512; 512 MB /cluster_size<= 1 MB) because fully zero or one clusters do not require any physical space?Makes some sense, but I can see that this might make give some trouble when trying to serialize overly large bitmaps. But I guess that comes later in this series, so I'll wait for that point.
Another thing is that 512 MB is rather big. It gets worse: The bitmap may only require 512 MB on disk, but with a maximum table size of 8 MB, it can require up to 8M *
cluster_sizein memory (with just 64 MB of disk space!) by using the "read as all zeroes" or "read as all ones" flags. With the default cluster size of 64 kB, this would be 512 GB in RAM. That sounds bad to me.Well, it is probably fine as long as the bitmap is not auto-loaded... But we do have a flag for exactly that. So it seems to me that a manipulated image can easily consume huge amounts of RAM on the host.
Quoted 8 lines
So I think we also need some sane limitation on the in-RAM size of a bitmap (which is
BME_MAX_TABLE_SIZE*cluster_size, as far as I understand). The question of course is, what is sane? For a server system with no image manipulation possible from the outside, 1 GB may be completely fine. But imagine you download some qcow2 image to your laptop. Then, 1 GB may not be fine, actually.Maybe it would make sense to use a runtime-adjustable limit here?
Actualy BME_MAX_PHYS_SIZE is this limit:
in check_constraints we have
uint64_t phys_bitmap_bytes =
(uint64_t)h->bitmap_table_size * s->cluster_size;
...
(phys_bitmap_bytes > BME_MAX_PHYS_SIZE) ||
Quoted 81 lines
block/qcow2-bitmap.c line 43
Quoted 10 lines
block/qcow2-bitmap.c new file +9
-0+#define BME_MAX_GRANULARITY_BITS 31 +#define BME_MIN_GRANULARITY_BITS 9 +#define BME_MAX_NAME_SIZE 1023 + +/* Bitmap directory entry flags */ +#define BME_RESERVED_FLAGS 0xffffffff + +/* bits [1, 8] U [56, 63] are reserved */ +#define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001feull suffix is missing.
Quoted 20 lines
block/qcow2-bitmap.c new file +4
-0+ +typedef enum BitmapType { +BT_DIRTY_TRACKING_BITMAP =1 +}BitmapType ;block/qcow2.h +4
-0@@ -52,6 +52,10 @@* space for snapshot names and IDs */ #define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS) +/* Bitmap header extension constraints */ +#define QCOW_MAX_DIRTY_BITMAPS 65535 +#define QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE (1024 * QCOW_MAX_DIRTY_BITMAPS) +/* indicate that the refcount of the referenced cluster is exactly one. */ #define QCOW_OFLAG_COPIED (1ULL << 63)/* indicate that the cluster is compressed (they never have the copied flag) */ block/qcow2.h typedef struct QEMU_PACKED QCowSnapshotHeader { · line 149
Quoted 4 lines
block/qcow2.h +1
-0@@ -142,6 +146,22 @@ typedef struct QEMU_PACKED QCowSnapshotHeader {/* name follows */ } QCowSnapshotHeader; +/* Qcow2BitmapDirEntry is actually a bitmap directory entry */ This comment doesn't make a whole lot of sense now that this structure is indeed called what it "actually is". ;-)
Quoted 19 lines
block/qcow2.h +15
-0+typedef struct QEMU_PACKED Qcow2BitmapDirEntry { +/* header is 8 byte aligned */ +uint64_t bitmap_table_offset; + +uint32_t bitmap_table_size; +uint32_t flags; + +uint8_t type; +uint8_t granularity_bits; +uint16_t name_size; +uint32_t extra_data_size; +/* extra data follows */ +/* name follows */ +} Qcow2BitmapDirEntry; + typedef struct QEMU_PACKED QCowSnapshotExtraData {uint64_t vm_state_size_large;uint64_t disk_size;block/qcow2.h typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array, · line 246
Quoted 5 lines
block/qcow2.h +2
-0@@ -222,6 +242,15 @@ typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,typedef void Qcow2SetRefcountFunc (void *refcount_array,uint64_t index,uint64_t value); +/* Be careful, Qcow2BitmapHeaderExt is not an extension of Qcow2BitmapDirEntry, it +* is Qcow2 header extension */ And this makes even less sense now.
(These comments don't stop me from giving an R-b, but I'm not so sure about the constants...)
Max
Quoted 11 lines
block/qcow2.h +7
-0+typedef struct Qcow2BitmapHeaderExt { +uint32_t nb_bitmaps; +uint32_t reserved32; +uint64_t bitmap_directory_size; +uint64_t bitmap_directory_offset; +} QEMU_PACKED Qcow2BitmapHeaderExt; + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size;
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 05/22] qcow2-bitmap: structs and consts
Quoted 113 lines · Vladimir Sementsov-Ogievskiy
Quoted 100 lines · Max Reitz
Quoted 76 lines · Vladimir Sementsov-Ogievskiy
Create
block/qcow2-bitmap.c
Add data structures and constraints accordingly todocs/specs/qcow2.txtSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/Makefile.objs | 2 +- block/qcow2-bitmap.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+), 1 deletion(-)create mode 100644
block/qcow2-bitmap.cblock/Makefile.objs +0
-0@@ -1,5 +1,5 @@ block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.ovvfat.o dmg.o
-block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
+block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-bitmap.o
block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o
qed-cluster.o
block-obj-y += qed-check.o block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.oblock/qcow2-bitmap.c new file +9
-0@@ -0,0 +1,47 @@ +/* +* Bitmaps for the QCOW version 2 format +* +* Copyright (c) 2014-2016 Vladimir Sementsov-Ogievskiy +* +* This file is derived from qcow2-snapshot.c, original copyright: +* Copyright (c) 2004-2006 Fabrice Bellard +* +* Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */block/qcow2-bitmap.c new file +3
-0+ +/* NOTICE: BME here means Bitmaps Extension and used as a namespace for +* _internal_ constants. Please do not use this _internal_ abbreviation for
+ * other needs and/or outside of this file. */block/qcow2-bitmap.c line 34
Quoted 4 lines
block/qcow2-bitmap.c new file +4
-0+ +/* Bitmap directory entry constraints */ +#define BME_MAX_TABLE_SIZE 0x8000000 +#define BME_MAX_PHYS_SIZE 0x20000000/* 512 mb */ I suppose
BME_MAX_TABLE_SIZE(8M) is greater thanBME_MAX_PHYS_SIZE(512 MB) divided by the cluster size (>= 512; 512 MB /cluster_size<= 1 MB) because fully zero or one clusters do not require any physical space?Makes some sense, but I can see that this might make give some trouble when trying to serialize overly large bitmaps. But I guess that comes later in this series, so I'll wait for that point.
Another thing is that 512 MB is rather big. It gets worse: The bitmap may only require 512 MB on disk, but with a maximum table size of 8 MB, it can require up to 8M *
cluster_sizein memory (with just 64 MB of disk space!) by using the "read as all zeroes" or "read as all ones" flags. With the default cluster size of 64 kB, this would be 512 GB in RAM. That sounds bad to me.Well, it is probably fine as long as the bitmap is not auto-loaded... But we do have a flag for exactly that. So it seems to me that a manipulated image can easily consume huge amounts of RAM on the host.
Quoted 8 lines
So I think we also need some sane limitation on the in-RAM size of a bitmap (which is
BME_MAX_TABLE_SIZE*cluster_size, as far as I understand). The question of course is, what is sane? For a server system with no image manipulation possible from the outside, 1 GB may be completely fine. But imagine you download some qcow2 image to your laptop. Then, 1 GB may not be fine, actually.Maybe it would make sense to use a runtime-adjustable limit here?
Actualy
BME_MAX_PHYS_SIZEis this limit:
incheck_constraintswe have
Quoted 6 lines
uint64_tphys_bitmap_bytes=(uint64_t)h->bitmap_table_size * s->cluster_size;...
(
phys_bitmap_bytes>BME_MAX_PHYS_SIZE) ||
OK, so BME_MAX_PHYS_SIZE is actually supposed to be the limit of the size of the bitmaps in RAM? And I suppose it is going to be calculated differently in the future once qemu has sparse bitmap support?
My fault, then, I thought BME_MAX_PHYS_SIZE was supposed to be the limit of the size on disk. OK, makes sense then, but the question whether a runtime-adjustable limit would make sense still remains. OTOH, this is something that can always be added later on.
Max
PATCH v7 6/22 qcow2: add dirty bitmaps extension
In reply to Vladimir Sementsov-OgievskiyAdd dirty bitmap extension as specified in docs/specs/qcow2.txt. For now, just mirror extension header into Qcow2 state and check constraints.
For now, disable image resize if it has bitmaps. It will be fixed later.
--- block/qcow2.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 +++ 2 files changed, 87 insertions(+)
block/qcow2.c +83 -0
@@ -63,6 +63,7 @@ typedef struct {#define QCOW2_EXT_MAGIC_END 0#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA#define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857 +#define QCOW2_EXT_MAGIC_DIRTY_BITMAPS 0x23852875static int qcow2_probe (const uint8_t *buf,int buf_size,const char *filename) {
@@ -92,6 +93,7 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,QCowExtension ext;uint64_t offset;int ret; +Qcow2BitmapHeaderExt bitmaps_ext;#ifdef DEBUG_EXT printf ("qcow2_read_extensions: start=%ld end=%ld\n" , start_offset, end_offset);
@@ -162,6 +164,62 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, }break ; +case QCOW2_EXT_MAGIC_DIRTY_BITMAPS : + ret =bdrv_pread (bs->file, offset, &bitmaps_ext, ext.len); +if (ret <0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Could not read ext header" ); +return ret; + } + +if (bitmaps_ext.reserved32 !=0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Reserved field is not zero." ); +return -EINVAL ; + } + +be32_to_cpus (&bitmaps_ext.nb_bitmaps); +be64_to_cpus (&bitmaps_ext.bitmap_directory_size); +be64_to_cpus (&bitmaps_ext.bitmap_directory_offset); + +if (bitmaps_ext.nb_bitmaps >QCOW_MAX_DIRTY_BITMAPS ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"too many dirty bitmaps" ); +return -EINVAL ; + } + +if (bitmaps_ext.nb_bitmaps ==0 ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"found bitmaps extension with zero bitmaps" ); +return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_offset & (s->cluster_size -1 )) { +error_setg (errp,"ERROR: bitmaps_ext: " +"wrong dirty bitmap directory offset" ); +return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_size > +QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"too large dirty bitmap directory" ); +return -EINVAL ; + } + + s->nb_bitmaps = bitmaps_ext.nb_bitmaps; + s->bitmap_directory_offset = + bitmaps_ext.bitmap_directory_offset; + s->bitmap_directory_size = + bitmaps_ext.bitmap_directory_size; + +#ifdef DEBUG_EXT +printf ("Qcow2: Got dirty bitmaps extension:" +" offset=%" PRIu64" nb_bitmaps=%" PRIu32"\n" , + s->bitmap_directory_offset, s->nb_bitmaps); +#endif +break ; +default :/* unknown magic - save it in case we need to rewrite the header */ {
@@ -1939,6 +1997,24 @@ int qcow2_update_header(BlockDriverState *bs) buflen -= ret; } +if (s->nb_bitmaps >0 ) { +Qcow2BitmapHeaderExt bitmaps_header = { + .nb_bitmaps =cpu_to_be32 (s->nb_bitmaps), + .bitmap_directory_size = +cpu_to_be64 (s->bitmap_directory_size), + .bitmap_directory_offset = +cpu_to_be64 (s->bitmap_directory_offset) + }; + ret =header_ext_add (buf,QCOW2_EXT_MAGIC_DIRTY_BITMAPS , + &bitmaps_header,sizeof (bitmaps_header), + buflen); +if (ret <0 ) { + goto fail; + } + buf += ret; + buflen -= ret; + } +/* Keep unknown header extensions */ QLIST_FOREACH(uext, &s->unknown_header_ext, next) { ret =header_ext_add (buf, uext->magic, uext->data, uext->len, buflen);
@@ -2509,6 +2585,13 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset)return -ENOTSUP ; } +/* cannot proceed if image has bitmaps */ +if (s->nb_bitmaps) { +/* FIXME */ +error_report ("Can't resize an image which has bitmaps" ); +return -ENOTSUP ; + } +/* shrinking is currently not supported */ if (offset < bs->total_sectors *512 ) {error_report ("qcow2 doesn't support shrinking images yet" );
block/qcow2.h +4 -0
@@ -292,6 +292,10 @@ typedef struct BDRVQcow2State {unsigned int nb_snapshots;QCowSnapshot *snapshots; +uint64_t bitmap_directory_offset; +uint64_t bitmap_directory_size; +uint32_t nb_bitmaps; +int flags;int qcow_version;bool use_lazy_refcounts;
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 06/22] qcow2: add dirty bitmaps extension
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Add dirty bitmap extension as specified in
docs/specs/qcow2.txt. For now, just mirror extension header into Qcow2 state and check constraints.For now, disable image resize if it has bitmaps. It will be fixed later.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 +++ 2 files changed, 87 insertions(+)block/qcow2.c +0
-0
[...]
block/qcow2.c static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, · line 168
Quoted 6 lines
block/qcow2.c +2
-0@@ -162,6 +164,62 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, }break ; +case QCOW2_EXT_MAGIC_DIRTY_BITMAPS : + ret =bdrv_pread (bs->file, offset, &bitmaps_ext, ext.len);
Overflows with ext.len > sizeof(bitmaps_ext).
(ext.len < sizeof(bitmaps_ext) is also wrong, but less dramatically so.)
block/qcow2.c static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, · line 177
Quoted 9 lines
block/qcow2.c +9
-0+if (ret <0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Could not read ext header" ); +return ret; + } + +if (bitmaps_ext.reserved32 !=0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Reserved field is not zero." );
Please drop the full stop at the end.
Quoted 12 lines
block/qcow2.c +12
-0+return -EINVAL ; + } + +be32_to_cpus (&bitmaps_ext.nb_bitmaps); +be64_to_cpus (&bitmaps_ext.bitmap_directory_size); +be64_to_cpus (&bitmaps_ext.bitmap_directory_offset); + +if (bitmaps_ext.nb_bitmaps >QCOW_MAX_DIRTY_BITMAPS ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"too many dirty bitmaps" ); +return -EINVAL ; + }
block/qcow2.c static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, · line 199
Quoted 10 lines
block/qcow2.c +10
-0+ +if (bitmaps_ext.nb_bitmaps ==0 ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"found bitmaps extension with zero bitmaps" ); +return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_offset & (s->cluster_size -1 )) { +error_setg (errp,"ERROR: bitmaps_ext: " +"wrong dirty bitmap directory offset" );
s/wrong/invalid/
Quoted 26 lines
block/qcow2.c +23
-0+return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_size > +QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"too large dirty bitmap directory" ); +return -EINVAL ; + } + + s->nb_bitmaps = bitmaps_ext.nb_bitmaps; + s->bitmap_directory_offset = + bitmaps_ext.bitmap_directory_offset; + s->bitmap_directory_size = + bitmaps_ext.bitmap_directory_size; + +#ifdef DEBUG_EXT +printf ("Qcow2: Got dirty bitmaps extension:" +" offset=%" PRIu64" nb_bitmaps=%" PRIu32"\n" , + s->bitmap_directory_offset, s->nb_bitmaps); +#endif +break ; +default :/* unknown magic - save it in case we need to rewrite the header */ {
[...]
block/qcow2.c static int qcow2_truncate(BlockDriverState *bs, int64_t offset) · line 2590
Quoted 7 lines
block/qcow2.c +3
-0@@ -2509,6 +2585,13 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset)return -ENOTSUP ; } +/* cannot proceed if image has bitmaps */ +if (s->nb_bitmaps) { +/* FIXME */
I'd call it a TODO, but that's probably a matter of taste.
block/qcow2.c static int qcow2_truncate(BlockDriverState *bs, int64_t offset) · line 2597
Quoted 7 lines
block/qcow2.c +4
-0+error_report ("Can't resize an image which has bitmaps" ); +return -ENOTSUP ; + } +/* shrinking is currently not supported */ if (offset < bs->total_sectors *512 ) {error_report ("qcow2 doesn't support shrinking images yet" );
Max
Re: [Qemu-devel] [PATCH 06/22] qcow2: add dirty bitmaps extension
Quoted 38 lines · Max Reitz
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Add dirty bitmap extension as specified in
docs/specs/qcow2.txt. For now, just mirror extension header into Qcow2 state and check constraints.For now, disable image resize if it has bitmaps. It will be fixed later.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 +++ 2 files changed, 87 insertions(+)block/qcow2.c +0
-0[...]
block/qcow2.c static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, · line 168
Quoted 6 lines
block/qcow2.c +2
-0@@ -162,6 +164,62 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, }break ; +case QCOW2_EXT_MAGIC_DIRTY_BITMAPS : + ret =bdrv_pread (bs->file, offset, &bitmaps_ext, ext.len);Overflows with ext.len >
sizeof(bitmaps_ext).(ext.len <
sizeof(bitmaps_ext)is also wrong, but less dramatically so.)block/qcow2.c static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, · line 177
Quoted 9 lines
block/qcow2.c +9
-0+if (ret <0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Could not read ext header" ); +return ret; + } + +if (bitmaps_ext.reserved32 !=0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Reserved field is not zero." );
Quoted 1 line
Please drop the full stop at the end.
what do you mean? goto to fail: here? or not stop at all, just print error?
Quoted 71 lines
Quoted 13 lines
block/qcow2.c +12
-0+return -EINVAL ; + } + +be32_to_cpus (&bitmaps_ext.nb_bitmaps); +be64_to_cpus (&bitmaps_ext.bitmap_directory_size); +be64_to_cpus (&bitmaps_ext.bitmap_directory_offset); + +if (bitmaps_ext.nb_bitmaps >QCOW_MAX_DIRTY_BITMAPS ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"too many dirty bitmaps" ); +return -EINVAL ; + }block/qcow2.c static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, · line 199
Quoted 10 lines
block/qcow2.c +10
-0+ +if (bitmaps_ext.nb_bitmaps ==0 ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"found bitmaps extension with zero bitmaps" ); +return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_offset & (s->cluster_size -1 )) { +error_setg (errp,"ERROR: bitmaps_ext: " +"wrong dirty bitmap directory offset" );s/wrong/invalid/
Quoted 26 lines
block/qcow2.c +23
-0+return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_size > +QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +error_setg (errp,"ERROR: bitmaps_ext: " +"too large dirty bitmap directory" ); +return -EINVAL ; + } + + s->nb_bitmaps = bitmaps_ext.nb_bitmaps; + s->bitmap_directory_offset = + bitmaps_ext.bitmap_directory_offset; + s->bitmap_directory_size = + bitmaps_ext.bitmap_directory_size; + +#ifdef DEBUG_EXT +printf ("Qcow2: Got dirty bitmaps extension:" +" offset=%" PRIu64" nb_bitmaps=%" PRIu32"\n" , + s->bitmap_directory_offset, s->nb_bitmaps); +#endif +break ; +default :/* unknown magic - save it in case we need to rewrite the header */ {[...]
block/qcow2.c static int qcow2_truncate(BlockDriverState *bs, int64_t offset) · line 2590
Quoted 7 lines
block/qcow2.c +3
-0@@ -2509,6 +2585,13 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset)return -ENOTSUP ; } +/* cannot proceed if image has bitmaps */ +if (s->nb_bitmaps) { +/* FIXME */ I'd call it a TODO, but that's probably a matter of taste.
block/qcow2.c static int qcow2_truncate(BlockDriverState *bs, int64_t offset) · line 2597
Quoted 7 lines
block/qcow2.c +4
-0+error_report ("Can't resize an image which has bitmaps" ); +return -ENOTSUP ; + } +/* shrinking is currently not supported */ if (offset < bs->total_sectors *512 ) {error_report ("qcow2 doesn't support shrinking images yet" );Max
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 06/22] qcow2: add dirty bitmaps extension
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 39 lines · Max Reitz
Quoted 17 lines · Vladimir Sementsov-Ogievskiy
Add dirty bitmap extension as specified in
docs/specs/qcow2.txt. For now, just mirror extension header into Qcow2 state and check constraints.For now, disable image resize if it has bitmaps. It will be fixed later.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 +++ 2 files changed, 87 insertions(+)block/qcow2.c +0
-0[...]
block/qcow2.c static int · line 168
Quoted 6 lines
block/qcow2.c +1
-0@@ -162,6 +164,62 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, } break; + case QCOW2_EXT_MAGIC_DIRTY_BITMAPS: + ret = bdrv_pread(bs->file, offset, &bitmaps_ext, ext.len);Overflows with ext.len >
sizeof(bitmaps_ext).(ext.len <
sizeof(bitmaps_ext)is also wrong, but less dramatically so.)Quoted 9 lines
block/qcow2.c +9
-0+if (ret <0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Could not read ext header" ); +return ret; + } + +if (bitmaps_ext.reserved32 !=0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Reserved field is not zero." );
Quoted 3 lines
Quoted 1 line
Please drop the full stop at the end.
what do you mean? goto to fail: here? or not stop at all, just print error?
The "." at the end of the message. :-)
(https://en.wikipedia.org/wiki/Full_stop)
Max
Re: [Qemu-devel] [PATCH 06/22] qcow2: add dirty bitmaps extension
Quoted 41 lines · Max Reitz
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 39 lines · Max Reitz
Quoted 17 lines · Vladimir Sementsov-Ogievskiy
Add dirty bitmap extension as specified in
docs/specs/qcow2.txt. For now, just mirror extension header into Qcow2 state and check constraints.For now, disable image resize if it has bitmaps. It will be fixed later.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 +++ 2 files changed, 87 insertions(+)block/qcow2.c +0
-0[...]
block/qcow2.c static int · line 168
Quoted 6 lines
block/qcow2.c +1
-0@@ -162,6 +164,62 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, } break; + case QCOW2_EXT_MAGIC_DIRTY_BITMAPS: + ret = bdrv_pread(bs->file, offset, &bitmaps_ext, ext.len);Overflows with ext.len >
sizeof(bitmaps_ext).(ext.len <
sizeof(bitmaps_ext)is also wrong, but less dramatically so.)Quoted 9 lines
block/qcow2.c +9
-0+if (ret <0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Could not read ext header" ); +return ret; + } + +if (bitmaps_ext.reserved32 !=0 ) { +error_setg_errno (errp, -ret,"ERROR: bitmaps_ext: " +"Reserved field is not zero." );
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
Please drop the full stop at the end.
what do you mean? goto to fail: here? or not stop at all, just print error?
The "." at the end of the message. :-)
(https://en.wikipedia.org/wiki/Full_stop)
Max
aha, cool. didn't know this. )
Signature · 2 lines
Best regards, Vladimir
PATCH v7 7/22 qcow2-bitmap: introduce auto-loading bitmaps
In reply to Vladimir Sementsov-OgievskiyAuto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are loaded at image open and becomes BdrvDirtyBitmap's for corresponding drive. These bitmaps are deleted from Qcow2 image after loading to avoid conflicts.
Extra data in bitmaps is not supported for now.
--- block/qcow2-bitmap.c | 518 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/qcow2.c | 5 + block/qcow2.h | 3 + 3 files changed, 525 insertions(+), 1 deletion(-)
block/qcow2-bitmap.c +517 -1
@@ -25,6 +25,12 @@* THE SOFTWARE. */ +#include "qemu/osdep.h" +#include "qapi/error.h" + +#include "block/block_int.h" +#include "block/qcow2.h" +/* NOTICE: BME here means Bitmaps Extension and used as a namespace for * _internal_ constants. Please do not use this _internal_ abbreviation for * other needs and/or outside of this file. */
@@ -37,11 +43,521 @@#define BME_MAX_NAME_SIZE 1023/* Bitmap directory entry flags */ -#define BME_RESERVED_FLAGS 0xffffffff +#define BME_RESERVED_FLAGS 0xfffffffd +#define BME_FLAG_AUTO (1U << 1)/* bits [1, 8] U [56, 63] are reserved */ #define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001fe +#define for_each_bitmap_dir_entry_unsafe (entry, dir, size) \ + for (entry = (Qcow2BitmapDirEntry *)(dir); \ + entry < (Qcow2BitmapDirEntry *)((uint8_t *)(dir) + size); \ + entry = next_dir_entry(entry)) + +#define for_each_bitmap_dir_entry (entry, dir, size) \ + for (entry = (Qcow2BitmapDirEntry *)(dir); \ + assert(check_dir_iter(entry, (uint8_t *)(dir) + size)), \ + entry < (Qcow2BitmapDirEntry *)((uint8_t *)(dir) + size); \ + entry = next_dir_entry(entry)) +typedef enum BitmapType {BT_DIRTY_TRACKING_BITMAP =1 }BitmapType ; + +static inline bool can_write (BlockDriverState *bs) +{ +return !bdrv_is_read_only (bs) && !(bdrv_get_flags (bs) &BDRV_O_INACTIVE ); +} + +static inline void bitmap_dir_entry_to_cpu (Qcow2BitmapDirEntry *entry) +{ +be64_to_cpus (&entry->bitmap_table_offset); +be32_to_cpus (&entry->bitmap_table_size); +be32_to_cpus (&entry->flags); +be16_to_cpus (&entry->name_size); +be32_to_cpus (&entry->extra_data_size); +} + +static inline void bitmap_dir_entry_to_be (Qcow2BitmapDirEntry *entry) +{ +cpu_to_be64s (&entry->bitmap_table_offset); +cpu_to_be32s (&entry->bitmap_table_size); +cpu_to_be32s (&entry->flags); +cpu_to_be16s (&entry->name_size); +cpu_to_be32s (&entry->extra_data_size); +} + +static inline void bitmap_table_to_cpu (uint64_t *bitmap_table,size_t size) +{ +size_t i; + +for (i =0 ; i < size; ++i) { +be64_to_cpus (&bitmap_table[i]); + } +} + +static inline int calc_dir_entry_size (size_t name_size,size_t extra_data_size) +{ +return align_offset (sizeof (Qcow2BitmapDirEntry) + + name_size + extra_data_size,8 ); +} + +static inline int dir_entry_size (Qcow2BitmapDirEntry *entry) +{ +return calc_dir_entry_size (entry->name_size, entry->extra_data_size); +} + +static inline const char *dir_entry_name_notcstr (Qcow2BitmapDirEntry *entry) +{ +return (const char *)(entry +1 ) + entry->extra_data_size; +} + +static inline int copy_dir_entry (void *dest,Qcow2BitmapDirEntry *entry) +{ +int sz =dir_entry_size (entry); +memcpy (dest, entry, sz); +return sz; +} + +static inline Qcow2BitmapDirEntry *next_dir_entry (Qcow2BitmapDirEntry *entry) +{ +return (Qcow2BitmapDirEntry *)((uint8_t *)entry +dir_entry_size (entry)); +} + +static inline bool check_dir_iter (Qcow2BitmapDirEntry *it,void *directory_end) +{ +return ((void *)it == directory_end) || + (((void *)(it +1 ) <= directory_end) && + ((void *)next_dir_entry (it) <= directory_end)); +} + +static inline void bitmap_directory_to_be (uint8_t *dir,size_t size) +{ +uint8_t *end = dir + size; +while (dir < end) { +Qcow2BitmapDirEntry *e = (Qcow2BitmapDirEntry *)dir; + dir +=dir_entry_size (e); + +bitmap_dir_entry_to_be (e); + } +} + +static void clear_bitmap_table (BlockDriverState *bs,uint64_t *bitmap_table, +uint32_t bitmap_table_size) +{ +BDRVQcow2State *s = bs->opaque; +int cl_size = s->cluster_size; +int i; + +for (i =0 ; i < bitmap_table_size; ++i) { +uint64_t addr = bitmap_table[i]; +if (addr <=1 ) { +continue ; + } + +qcow2_free_clusters (bs, addr, cl_size,QCOW2_DISCARD_ALWAYS ); + bitmap_table[i] =0 ; + } +} + +static int bitmap_table_load (BlockDriverState *bs,Qcow2BitmapDirEntry *entry, +uint64_t **table) +{ +int ret; + + *table =g_try_new (uint64_t, entry->bitmap_table_size); +if (*table ==NULL ) { +return -ENOMEM ; + } + + ret =bdrv_pread (bs->file, entry->bitmap_table_offset, + *table, entry->bitmap_table_size *sizeof (uint64_t )); +if (ret <0 ) { +g_free (*table); + *table =NULL ; +return ret; + } + +bitmap_table_to_cpu (*table, entry->bitmap_table_size); + +return 0 ; +} + +static void do_free_bitmap_clusters (BlockDriverState *bs, +uint64_t table_offset, +uint32_t table_size, +uint64_t *bitmap_table) +{ +clear_bitmap_table (bs, bitmap_table, table_size); + +qcow2_free_clusters (bs, table_offset, table_size *sizeof (uint64_t ), +QCOW2_DISCARD_ALWAYS ); +} + +static int free_bitmap_clusters (BlockDriverState *bs,Qcow2BitmapDirEntry *entry) +{ +int ret; +uint64_t *bitmap_table; + + ret =bitmap_table_load (bs, entry, &bitmap_table); +if (ret <0 || bitmap_table ==NULL ) { +return ret; + } + +do_free_bitmap_clusters (bs, entry->bitmap_table_offset, + entry->bitmap_table_size, bitmap_table); + +return 0 ; +} + +/* dirty sectors in cluster is a number of sectors in the image, corresponding +* to one cluster of bitmap data */ +static uint64_t dirty_sectors_in_cluster (const BDRVQcow2State *s, +const BdrvDirtyBitmap *bitmap) +{ +uint32_t sector_granularity = +bdrv_dirty_bitmap_granularity (bitmap) >>BDRV_SECTOR_BITS ; + +return (uint64_t )sector_granularity * (s->cluster_size <<3 ); +} + +static int load_bitmap_data (BlockDriverState *bs, +const uint64_t *dirty_bitmap_table, +uint32_t dirty_bitmap_table_size, +BdrvDirtyBitmap *bitmap) +{ +int ret =0 ; +BDRVQcow2State *s = bs->opaque; +uint64_t sector, dsc; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +int cl_size = s->cluster_size; +uint8_t *buf =NULL ; +uint32_t i, tab_size = +size_to_clusters (s, +bdrv_dirty_bitmap_serialization_size (bitmap,0 , bm_size)); + +if (tab_size != dirty_bitmap_table_size) { +return -EINVAL ; + } + +bdrv_clear_dirty_bitmap (bitmap,NULL ); + + buf =g_malloc0 (cl_size); + dsc =dirty_sectors_in_cluster (s, bitmap); +for (i =0 , sector =0 ; i < tab_size; ++i, sector += dsc) { +uint64_t end =MIN (bm_size, sector + dsc); +uint64_t offset = dirty_bitmap_table[i]; + +if (offset ==1 ) { +bdrv_dirty_bitmap_deserialize_ones (bitmap, sector, end, false); + }else if (offset >1 ) { + ret =bdrv_pread (bs->file, offset, buf, cl_size); +if (ret <0 ) { + goto finish; + } +bdrv_dirty_bitmap_deserialize_part (bitmap, buf, sector, end, false); + } + } + ret =0 ; + +bdrv_dirty_bitmap_deserialize_finish (bitmap); + +finish: +g_free (buf); + +return ret; +} + +static BdrvDirtyBitmap *load_bitmap (BlockDriverState *bs, +Qcow2BitmapDirEntry *entry,Error **errp) +{ +int ret; +uint64_t *bitmap_table =NULL ; +uint32_t granularity; +BdrvDirtyBitmap *bitmap =NULL ; +char *name =g_strndup (dir_entry_name_notcstr (entry), entry->name_size); + + ret =bitmap_table_load (bs, entry, &bitmap_table); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Could not read bitmap_table table from image" ); + goto fail; + } + + granularity =1U << entry->granularity_bits; + bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); +if (bitmap ==NULL ) { + goto fail; + } + + ret =load_bitmap_data (bs, bitmap_table, entry->bitmap_table_size, + bitmap); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Could not read bitmap from image" ); + goto fail; + } + +g_free (name); +g_free (bitmap_table); +return bitmap; + +fail: +g_free (name); +g_free (bitmap_table); +if (bitmap !=NULL ) { +bdrv_release_dirty_bitmap (bs, bitmap); + } + +return NULL ; +} + +/* directory_read +* Read bitmaps directory from bs by @offset and @size. Convert it to cpu +* format from BE. +*/ +static uint8_t *directory_read (BlockDriverState *bs, +uint64_t offset,uint64_t size,Error **errp) +{ +int ret; +uint8_t *dir, *dir_end; +Qcow2BitmapDirEntry *e; + + dir =g_try_malloc (size); +if (dir ==NULL ) { +error_setg (errp,"Can't allocate space for bitmap directory." ); +return NULL ; + } + dir_end = dir + size; + + ret =bdrv_pread (bs->file, offset, dir, size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Can't read bitmap directory." ); + goto fail; + } + +/* loop is safe because next entry offset is calculated after conversion to +* cpu format */ +for_each_bitmap_dir_entry_unsafe (e, dir, size) { +if ((uint8_t *)(e +1 ) > dir_end) { + goto broken_dir; + } + +bitmap_dir_entry_to_cpu (e); + +if ((uint8_t *)next_dir_entry (e) > dir_end) { + goto broken_dir; + } + +if (e->extra_data_size !=0 ) { +error_setg (errp,"Can't load bitmap '%.*s' from '%s':" +"extra data not supported." , e->name_size, +dir_entry_name_notcstr (e), +bdrv_get_device_or_node_name (bs)); + goto fail; + } + } + +assert ((uint8_t *)e == dir_end); + +return dir; + +broken_dir: +error_setg (errp,"Broken bitmap directory." ); + +fail: +g_free (dir); + +return NULL ; +} + +static int update_header_sync (BlockDriverState *bs) +{ +int ret; + + ret =qcow2_update_header (bs); +if (ret <0 ) { +return ret; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { +return ret; + } + +return 0 ; +} + +/* write bitmap directory from the state to new allocated clusters */ +static int64_t directory_write (BlockDriverState *bs,const uint8_t *dir, +size_t size) +{ +int ret =0 ; +uint8_t *dir_be =NULL ; +int64_t dir_offset =0 ; + + dir_be =g_try_malloc (size); +if (dir_be ==NULL ) { +return -ENOMEM ; + } +memcpy (dir_be, dir, size); +bitmap_directory_to_be (dir_be, size); + +/* Allocate space for the new bitmap directory */ + dir_offset =qcow2_alloc_clusters (bs, size); +if (dir_offset <0 ) { + ret = dir_offset; + goto out; + } + +/* The bitmap directory position has not yet been updated, so these +* clusters must indeed be completely free */ + ret =qcow2_pre_write_overlap_check (bs,0 , dir_offset, size); +if (ret <0 ) { + goto out; + } + + ret =bdrv_pwrite (bs->file, dir_offset, dir_be, size); +if (ret <0 ) { + goto out; + } + +out: +g_free (dir_be); + +if (ret <0 ) { +if (dir_offset >0 ) { +qcow2_free_clusters (bs, dir_offset, size,QCOW2_DISCARD_ALWAYS ); + } + +return ret; + } + +return dir_offset; +} + +static int directory_update (BlockDriverState *bs,uint8_t *new_dir, +size_t new_size,uint32_t new_nb_bitmaps) +{ +BDRVQcow2State *s = bs->opaque; +int ret; +int64_t new_offset =0 ; +uint64_t old_offset = s->bitmap_directory_offset; +uint64_t old_size = s->bitmap_directory_size; +uint32_t old_nb_bitmaps = s->nb_bitmaps; +uint64_t old_autocl = s->autoclear_features; + +if (new_size >QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +return -EINVAL ; + } + +if ((new_size ==0 ) != (new_nb_bitmaps ==0 )) { +return -EINVAL ; + } + +if (new_nb_bitmaps >0 ) { + new_offset =directory_write (bs, new_dir, new_size); +if (new_offset <0 ) { +return new_offset; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { + goto fail; + } + } + s->bitmap_directory_offset = new_offset; + s->bitmap_directory_size = new_size; + s->nb_bitmaps = new_nb_bitmaps; + + ret =update_header_sync (bs); +if (ret <0 ) { + goto fail; + } + +if (old_size) { +qcow2_free_clusters (bs, old_offset, old_size,QCOW2_DISCARD_ALWAYS ); + } + +return 0 ; + +fail: +if (new_offset >0 ) { +qcow2_free_clusters (bs, new_offset, new_size,QCOW2_DISCARD_ALWAYS ); + s->bitmap_directory_offset = old_offset; + s->bitmap_directory_size = old_size; + s->nb_bitmaps = old_nb_bitmaps; + s->autoclear_features = old_autocl; + } + +return ret; +} + +int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint8_t *dir, *new_dir, *new_pos; +uint64_t dir_size; +Qcow2BitmapDirEntry *e; +uint32_t new_nb_bitmaps =0 ; + +if (s->nb_bitmaps ==0 ) { +/* No bitmaps - nothing to do */ +return 0 ; + } + + new_pos = new_dir =g_try_malloc (s->bitmap_directory_size); +if (new_dir ==NULL ) { +error_setg (errp,"Can't allocate space for bitmap directory." ); +return -ENOMEM ; + } + + dir_size = s->bitmap_directory_size; + dir =directory_read (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); +if (dir ==NULL ) { + ret = -EINVAL ; + goto out; + } + +for_each_bitmap_dir_entry (e, dir, dir_size) { +uint32_t fl = e->flags; + +if (fl &BME_FLAG_AUTO ) { +BdrvDirtyBitmap *bitmap =load_bitmap (bs, e, errp); +if (bitmap ==NULL ) { + ret = -EINVAL ; + goto out; + } + }else { +/* leave bitmap in the image */ + new_pos +=copy_dir_entry (new_pos, e); + new_nb_bitmaps++; + } + } + +if (!can_write (bs)) { + goto out; + } + + ret =directory_update (bs, new_dir, new_pos - new_dir, new_nb_bitmaps); +if (ret <0 ) { +error_setg (errp,"Can't update bitmap directory." ); + goto out; + } + +/* to be consistent, free bitmap only after successfull directory update */ +for_each_bitmap_dir_entry (e, dir, dir_size) { +uint32_t fl = e->flags; + +if (fl &BME_FLAG_AUTO ) { +free_bitmap_clusters (bs, e); + } + } + +out: +g_free (dir); +g_free (new_dir); + +return ret; +}
block/qcow2.c +5 -0
@@ -213,6 +213,11 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, s->bitmap_directory_size = bitmaps_ext.bitmap_directory_size; + ret =qcow2_read_bitmaps (bs, errp); +if (ret <0 ) { +return ret; + } +#ifdef DEBUG_EXT printf("Qcow2: Got dirty bitmaps extension:" " offset=%" PRIu64 " nb_bitmaps=%" PRIu32 "\n",
block/qcow2.h +3 -0
@@ -603,6 +603,9 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs,void qcow2_free_snapshots (BlockDriverState *bs);int qcow2_read_snapshots (BlockDriverState *bs); +/* qcow2-bitmap.c functions */ +int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp); +/* qcow2-cache.c functions */ Qcow2Cache *qcow2_cache_create (BlockDriverState *bs,int num_tables);int qcow2_cache_destroy (BlockDriverState * bs,Qcow2Cache *c);
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are
*with the AUTO flag set
Quoted 1 line
loaded at image open and becomes BdrvDirtyBitmap's for corresponding
"loaded when the image is opened and become BdrvDirtyBitmaps for the corresponding drive."
Quoted 1 line
drive. These bitmaps are deleted from Qcow2 image after loading to avoid
*from the Qcow2 image
Quoted 16 lines
conflicts.
Extra data in bitmaps is not supported for now.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 518 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/qcow2.c | 5 + block/qcow2.h | 3 + 3 files changed, 525 insertions(+), 1 deletion(-)block/qcow2-bitmap.c +0
-0@@ -25,6 +25,12 @@
[...]
block/qcow2-bitmap.c line 154
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+static void clear_bitmap_table (BlockDriverState *bs,uint64_t *bitmap_table, +uint32_t bitmap_table_size) +{ +BDRVQcow2State *s = bs->opaque; +int cl_size = s->cluster_size; +int i; + +for (i =0 ; i < bitmap_table_size; ++i) { +uint64_t addr = bitmap_table[i]; +if (addr <=1 ) {
I'd rather add a BME_TABLE_ENTRY_OFFSET_MASK
(0x00fffffffffffe00ull) and then use:
uint64_t addr = bitmap_table[i] & BME_TABLE_ENTRY_OFFSET_MASK; if (!addr) {
block/qcow2-bitmap.c line 158
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+continue ; + } + +qcow2_free_clusters (bs, addr, cl_size,QCOW2_DISCARD_ALWAYS );
I think this should rather be QCOW2_DISCARD_OTHER; or you introduce a new QCOW2_DISCARD_* flag.
(The same applies to all the other QCOW2_DISCARD_ALWAYS users here.)
Also, I don't really see the point of introducing cl_size just for this place; it doesn't even save you from a line wrap.
block/qcow2-bitmap.c line 169
Quoted 11 lines
block/qcow2-bitmap.c +11
-0+ bitmap_table[i] =0 ; + } +} + +static int bitmap_table_load (BlockDriverState *bs,Qcow2BitmapDirEntry *entry, +uint64_t **table) +{ +int ret; + + *table =g_try_new (uint64_t, entry->bitmap_table_size); +if (*table ==NULL ) {
Not that g_try_new() will return NULL if you try to allocate a buffer with size 0.
Normally, entry->bitmap_table_size shouldn't be 0, but I don't think you're catching that case anywhere.
block/qcow2-bitmap.c line 172
Quoted 3 lines
block/qcow2-bitmap.c +3
-0+return -ENOMEM ; + } +
I wouldn't mind an
assert(entry->bitmap_table_size <= UINT32_MAX / sizeof(uint64_t));
here because of the following multiplication (which is extended to 64 bit on 64 bit machines, but stays in 32 bit on 32 bit machines).
(Of course,
assert(entry->bitmap_table_size <= BME_MAX_TABLE_SIZE);
would be fine, too.)
Quoted 21 lines
block/qcow2-bitmap.c +21
-0+ ret =bdrv_pread (bs->file, entry->bitmap_table_offset, + *table, entry->bitmap_table_size *sizeof (uint64_t )); +if (ret <0 ) { +g_free (*table); + *table =NULL ; +return ret; + } + +bitmap_table_to_cpu (*table, entry->bitmap_table_size); + +return 0 ; +} + +static void do_free_bitmap_clusters (BlockDriverState *bs, +uint64_t table_offset, +uint32_t table_size, +uint64_t *bitmap_table) +{ +clear_bitmap_table (bs, bitmap_table, table_size); + + qcow2_free_clusters(bs, table_offset, table_size * sizeof(uint64_t),
block/qcow2-bitmap.c line 203
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ QCOW2_DISCARD_ALWAYS); +} + +static int free_bitmap_clusters (BlockDriverState *bs,Qcow2BitmapDirEntry *entry) +{ +int ret; +uint64_t *bitmap_table; + + ret =bitmap_table_load (bs, entry, &bitmap_table); +if (ret <0 || bitmap_table ==NULL ) {
That should be:
if (ret < 0) {
assert(bitmap_table == NULL);Or the other way around:
if (bitmap_table == NULL) {
assert(ret < 0);Otherwise, it looks like bitmap_table may be NULL with ret being 0, and the next statement would make this function return success even though it actually failed.
block/qcow2-bitmap.c line 210
Quoted 7 lines
block/qcow2-bitmap.c +7
-0+return ret; + } + +do_free_bitmap_clusters (bs, entry->bitmap_table_offset, + entry->bitmap_table_size, bitmap_table); + +return 0 ;
You're leaking bitmap_table here.
block/qcow2-bitmap.c line 214
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+} + +/* dirty sectors in cluster is a number of sectors in the image, corresponding +* to one cluster of bitmap data */
How about:
This function returns the number of sectors covered by a single cluster of dirty bitmap data.
Quoted 9 lines
block/qcow2-bitmap.c +9
-0+static uint64_t dirty_sectors_in_cluster (const BDRVQcow2State *s, +const BdrvDirtyBitmap *bitmap) +{ +uint32_t sector_granularity = +bdrv_dirty_bitmap_granularity (bitmap) >>BDRV_SECTOR_BITS ; + +return (uint64_t )sector_granularity * (s->cluster_size <<3 ); +} +
block/qcow2-bitmap.c line 233
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+static int load_bitmap_data (BlockDriverState *bs, +const uint64_t *dirty_bitmap_table, +uint32_t dirty_bitmap_table_size, +BdrvDirtyBitmap *bitmap) +{ +int ret =0 ; +BDRVQcow2State *s = bs->opaque; +uint64_t sector, dsc; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +int cl_size = s->cluster_size;
Once more, I don't really see the reason to put this into a dedicated variable (I feel like it makes the code harder to read because you have to keep in mind what cl_size is).
block/qcow2-bitmap.c line 241
Quoted 8 lines
block/qcow2-bitmap.c +8
-0+uint8_t *buf =NULL ; +uint32_t i, tab_size = +size_to_clusters (s, +bdrv_dirty_bitmap_serialization_size (bitmap,0 , bm_size)); + +if (tab_size != dirty_bitmap_table_size) { +return -EINVAL ; + }
You should also check that tab_size does not exceed BME_MAX_TABLE_SIZE.
And maybe you should check that the physical size of the bitmap data does not exceed BME_MAX_PHYS_SIZE, but I'm not so sure why you have that constant at all because it's basically superseded by the existence of BME_MAX_TABLE_SIZE.
block/qcow2-bitmap.c line 245
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+ +bdrv_clear_dirty_bitmap (bitmap,NULL ); + + buf =g_malloc0 (cl_size);
g_malloc() would suffice as well, since you'll be overwriting all of it anyway.
block/qcow2-bitmap.c line 253
Quoted 8 lines
block/qcow2-bitmap.c +8
-0+ dsc =dirty_sectors_in_cluster (s, bitmap); +for (i =0 , sector =0 ; i < tab_size; ++i, sector += dsc) { +uint64_t end =MIN (bm_size, sector + dsc); +uint64_t offset = dirty_bitmap_table[i]; + +if (offset ==1 ) { +bdrv_dirty_bitmap_deserialize_ones (bitmap, sector, end, false); + }else if (offset >1 ) {
As before, I'd organize this differently, e.g. like:
uint64_t table_entry = dirty_bitmap_table[i]; uint64_t offset = table_entry & BME_TABLE_ENTRY_OFFSET_MASK;
if (table_entry & BME_TABLE_ENTRY_RESERVED_MASK) {
ret = -EINVAL;
goto finish;
}if (!offset) {
if (table_entry & 1) {
bdrv_dirty_bitmap_deserialize_ones(bitmap, sector, end, false);
} else {
/* No need to deserialize zeroes because the dirty bitmap is
* already cleared */
}
} else {
ret = bdrv_pread(...);
...
}It's more code, yes, but I find it easier to read because it reflects the specification better.
Quoted 15 lines
block/qcow2-bitmap.c +15
-0+ ret =bdrv_pread (bs->file, offset, buf, cl_size); +if (ret <0 ) { + goto finish; + } +bdrv_dirty_bitmap_deserialize_part (bitmap, buf, sector, end, false); + } + } + ret =0 ; + +bdrv_dirty_bitmap_deserialize_finish (bitmap); + +finish: +g_free (buf); + +return ret;
block/qcow2-bitmap.c line 278
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+} + +static BdrvDirtyBitmap *load_bitmap (BlockDriverState *bs, +Qcow2BitmapDirEntry *entry,Error **errp) +{ +int ret; +uint64_t *bitmap_table =NULL ; +uint32_t granularity; +BdrvDirtyBitmap *bitmap =NULL ; +char *name =g_strndup (dir_entry_name_notcstr (entry), entry->name_size);
Maybe you should limit entry->name_size to BME_MAX_NAME_SIZE.
block/qcow2-bitmap.c line 287
Quoted 9 lines
block/qcow2-bitmap.c +9
-0+ + ret =bitmap_table_load (bs, entry, &bitmap_table); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Could not read bitmap_table table from image" ); + goto fail; + } + + granularity =1U << entry->granularity_bits;
You should be making sure that entry->granularity_bits does not exceed [BME_MIN_GRANULARITY_BITS, BME_MAX_GRANULARITY_BITS] before this point.
Quoted 26 lines
block/qcow2-bitmap.c +26
-0+ bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); +if (bitmap ==NULL ) { + goto fail; + } + + ret =load_bitmap_data (bs, bitmap_table, entry->bitmap_table_size, + bitmap); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Could not read bitmap from image" ); + goto fail; + } + +g_free (name); +g_free (bitmap_table); +return bitmap; + +fail: +g_free (name); +g_free (bitmap_table); +if (bitmap !=NULL ) { +bdrv_release_dirty_bitmap (bs, bitmap); + } + +return NULL ; +} +
block/qcow2-bitmap.c line 323
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+/* directory_read +* Read bitmaps directory from bs by @offset and @size. Convert it to cpu +* format from BE. +*/ +static uint8_t *directory_read (BlockDriverState *bs, +uint64_t offset,uint64_t size,Error **errp) +{ +int ret; +uint8_t *dir, *dir_end; +Qcow2BitmapDirEntry *e;
I think you should return an error if size >QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE.
block/qcow2-bitmap.c line 327
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+ + dir =g_try_malloc (size); +if (dir ==NULL ) { +error_setg (errp,"Can't allocate space for bitmap directory." );
Please drop the full stop at the end of the message.
(And I personally don't like contractions ("can't") in user-facing messages too much, but that is just my taste.)
block/qcow2-bitmap.c line 334
Quoted 7 lines
block/qcow2-bitmap.c +7
-0+return NULL ; + } + dir_end = dir + size; + + ret =bdrv_pread (bs->file, offset, dir, size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Can't read bitmap directory." );
Again, no full stop at the end of error messages, please.
block/qcow2-bitmap.c line 338
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+ goto fail; + } + +/* loop is safe because next entry offset is calculated after conversion to
Should it be s/safe/unsafe/?
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+* cpu format */ +for_each_bitmap_dir_entry_unsafe (e, dir, size) { +if ((uint8_t *)(e +1 ) > dir_end) { + goto broken_dir; + }
block/qcow2-bitmap.c line 353
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ +bitmap_dir_entry_to_cpu (e); + +if ((uint8_t *)next_dir_entry (e) > dir_end) { + goto broken_dir; + } + +if (e->extra_data_size !=0 ) { + error_setg(errp, "Can't load bitmap '%.*s' from '%s':" + "extra data not supported.", e->name_size,
Full stop again.
Also, I'm not quite sure why you give the device/node name here. You don't do that anywhere else and I think if we want to emit the information where something failed, it should be added at some previous point in the call chain.
block/qcow2-bitmap.c line 358
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+dir_entry_name_notcstr (e), + bdrv_get_device_or_node_name(bs)); + goto fail; + } + }
Maybe you should count the number of dirty bitmaps and return an error if it exceeds QCOW_MAX_DIRTY_BITMAPS.
block/qcow2-bitmap.c line 360
Quoted 2 lines
block/qcow2-bitmap.c +2
-0+ +assert ((uint8_t *)e == dir_end);
I think this should rather go to broken_dir if they're not equal.
block/qcow2-bitmap.c line 365
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+ +return dir; + +broken_dir: +error_setg (errp,"Broken bitmap directory." );
And again.
Quoted 76 lines
block/qcow2-bitmap.c +76
-0+ +fail: +g_free (dir); + +return NULL ; +} + +static int update_header_sync (BlockDriverState *bs) +{ +int ret; + + ret =qcow2_update_header (bs); +if (ret <0 ) { +return ret; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { +return ret; + } + +return 0 ; +} + +/* write bitmap directory from the state to new allocated clusters */ +static int64_t directory_write (BlockDriverState *bs,const uint8_t *dir, +size_t size) +{ +int ret =0 ; +uint8_t *dir_be =NULL ; +int64_t dir_offset =0 ; + + dir_be =g_try_malloc (size); +if (dir_be ==NULL ) { +return -ENOMEM ; + } +memcpy (dir_be, dir, size); +bitmap_directory_to_be (dir_be, size); + +/* Allocate space for the new bitmap directory */ + dir_offset =qcow2_alloc_clusters (bs, size); +if (dir_offset <0 ) { + ret = dir_offset; + goto out; + } + +/* The bitmap directory position has not yet been updated, so these +* clusters must indeed be completely free */ + ret =qcow2_pre_write_overlap_check (bs,0 , dir_offset, size); +if (ret <0 ) { + goto out; + } + + ret =bdrv_pwrite (bs->file, dir_offset, dir_be, size); +if (ret <0 ) { + goto out; + } + +out: +g_free (dir_be); + +if (ret <0 ) { +if (dir_offset >0 ) { +qcow2_free_clusters (bs, dir_offset, size,QCOW2_DISCARD_ALWAYS ); + } + +return ret; + } + +return dir_offset; +} + +static int directory_update (BlockDriverState *bs,uint8_t *new_dir, +size_t new_size,uint32_t new_nb_bitmaps) +{ +BDRVQcow2State *s = bs->opaque;
block/qcow2-bitmap.c line 451
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+int ret; +int64_t new_offset =0 ; +uint64_t old_offset = s->bitmap_directory_offset; +uint64_t old_size = s->bitmap_directory_size; +uint32_t old_nb_bitmaps = s->nb_bitmaps; +uint64_t old_autocl = s->autoclear_features; + +if (new_size >QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +return -EINVAL ; + }
Also, you should probably check new_nb_bitmaps against QCOW_MAX_DIRTY_BITMAPS, but maybe you'll be adding that check somewhere else in a future patch. In that case, an assertion wouldn't be too bad here, though.
Quoted 28 lines
block/qcow2-bitmap.c +28
-0+ +if ((new_size ==0 ) != (new_nb_bitmaps ==0 )) { +return -EINVAL ; + } + +if (new_nb_bitmaps >0 ) { + new_offset =directory_write (bs, new_dir, new_size); +if (new_offset <0 ) { +return new_offset; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { + goto fail; + } + } + s->bitmap_directory_offset = new_offset; + s->bitmap_directory_size = new_size; + s->nb_bitmaps = new_nb_bitmaps; + + ret =update_header_sync (bs); +if (ret <0 ) { + goto fail; + } + +if (old_size) { +qcow2_free_clusters (bs, old_offset, old_size,QCOW2_DISCARD_ALWAYS ); + }
block/qcow2-bitmap.c line 489
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ +return 0 ; + +fail: +if (new_offset >0 ) { +qcow2_free_clusters (bs, new_offset, new_size,QCOW2_DISCARD_ALWAYS ); + s->bitmap_directory_offset = old_offset; + s->bitmap_directory_size = old_size; + s->nb_bitmaps = old_nb_bitmaps; + s->autoclear_features = old_autocl;
Why are you restoring the autoclear features? From a quick glance I can't see any code path that changes this field here, and if there is one, it probably has a good reason to do so.
block/qcow2-bitmap.c line 500
Quoted 11 lines
block/qcow2-bitmap.c +11
-0+ } + +return ret; +} + +int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint8_t *dir, *new_dir, *new_pos; +uint64_t dir_size;
I suggest initializing this variable here (to s->bitmap_directory_size)...
block/qcow2-bitmap.c line 509
Quoted 9 lines
block/qcow2-bitmap.c +9
-0+Qcow2BitmapDirEntry *e; +uint32_t new_nb_bitmaps =0 ; + +if (s->nb_bitmaps ==0 ) { +/* No bitmaps - nothing to do */ +return 0 ; + } + + new_pos = new_dir =g_try_malloc (s->bitmap_directory_size);
...and then using it here...
block/qcow2-bitmap.c line 517
Quoted 8 lines
block/qcow2-bitmap.c +8
-0+if (new_dir ==NULL ) { +error_setg (errp,"Can't allocate space for bitmap directory." ); +return -ENOMEM ; + } + + dir_size = s->bitmap_directory_size; + dir =directory_read (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp);
...and here.
Alternatively, of course, you can just drop the variable and use s->bitmap_directory_size everywhere.
Quoted 12 lines
block/qcow2-bitmap.c +12
-0+if (dir ==NULL ) { + ret = -EINVAL ; + goto out; + } + +for_each_bitmap_dir_entry (e, dir, dir_size) { +uint32_t fl = e->flags; + +if (fl &BME_FLAG_AUTO ) { +BdrvDirtyBitmap *bitmap =load_bitmap (bs, e, errp); +if (bitmap ==NULL ) { + ret = -EINVAL ;
block/qcow2-bitmap.c line 539
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ goto out; + } + }else { +/* leave bitmap in the image */ + new_pos +=copy_dir_entry (new_pos, e); + new_nb_bitmaps++; + } + } + +if (!can_write (bs)) {
You should be setting ret here (gcc complains about that).
block/qcow2-bitmap.c line 545
Quoted 6 lines
block/qcow2-bitmap.c +6
-0+ goto out; + } + + ret =directory_update (bs, new_dir, new_pos - new_dir, new_nb_bitmaps); +if (ret <0 ) { +error_setg (errp,"Can't update bitmap directory." );
And once more, please no full stops at the end of error messages.
block/qcow2-bitmap.c line 549
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+ goto out; + } + +/* to be consistent, free bitmap only after successfull directory update */
*successful
Quoted 19 lines
block/qcow2-bitmap.c +14
-0+for_each_bitmap_dir_entry (e, dir, dir_size) { +uint32_t fl = e->flags; + +if (fl &BME_FLAG_AUTO ) { +free_bitmap_clusters (bs, e); + } + } + +out: +g_free (dir); +g_free (new_dir); + +return ret; +}block/qcow2.c +0
-0
block/qcow2.c static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, · line 220
Quoted 8 lines
block/qcow2.c +5
-0@@ -213,6 +213,11 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, s->bitmap_directory_size = bitmaps_ext.bitmap_directory_size; + ret =qcow2_read_bitmaps (bs, errp); +if (ret <0 ) { +return ret; + } +
I think I'd put this directly into qcow2_open(), just like qcow2_read_snapshots(); but that's an optional suggestion.
Max
Quoted 18 lines
block/qcow2.c +0
-0#ifdef DEBUG_EXT printf("Qcow2: Got dirty bitmaps extension:" " offset=%" PRIu64 " nb_bitmaps=%" PRIu32 "\n",block/qcow2.h +3
-0@@ -603,6 +603,9 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs,void qcow2_free_snapshots (BlockDriverState *bs);int qcow2_read_snapshots (BlockDriverState *bs); +/* qcow2-bitmap.c functions */ +int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp); +/* qcow2-cache.c functions */ Qcow2Cache *qcow2_cache_create (BlockDriverState *bs,int num_tables);int qcow2_cache_destroy (BlockDriverState * bs,Qcow2Cache *c);
Re: [Qemu-devel] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps
Quoted 2 lines · Max Reitz
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are
...
Quoted 5 lines
block/qcow2-bitmap.c +4
-0+ goto fail; + } + +/* loop is safe because next entry offset is calculated after conversion to Should it be s/safe/unsafe/?
loop is safe. _unsafe is related to absence of assert in a loop, as it loops through BE data. Bad wording, I'll change it somehow..
Quoted 18 lines
Quoted 6 lines
block/qcow2-bitmap.c +5
-0+* cpu format */ +for_each_bitmap_dir_entry_unsafe (e, dir, size) { +if ((uint8_t *)(e +1 ) > dir_end) { + goto broken_dir; + }block/qcow2-bitmap.c line 353
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ +bitmap_dir_entry_to_cpu (e); + +if ((uint8_t *)next_dir_entry (e) > dir_end) { + goto broken_dir; + } + +if (e->extra_data_size !=0 ) { + error_setg(errp, "Can't load bitmap '%.*s' from '%s':" + "extra data not supported.", e->name_size,Full stop again.
Quoted 4 lines
Also, I'm not quite sure why you give the device/node name here. You don't do that anywhere else and I think if we want to emit the information where something failed, it should be added at some previous point in the call chain.
For example, how? As I understand, I can emit it only by error_setg, so actually it would be better to add node and bitmap name to all error_setg in the code.. or create helper function for it.
Quoted 6 lines
Quoted 6 lines
block/qcow2-bitmap.c +5
-0+dir_entry_name_notcstr (e), + bdrv_get_device_or_node_name(bs)); + goto fail; + } + }
...
Quoted 21 lines
Quoted 21 lines
block/qcow2-bitmap.c +21
-0+if (ret <0 ) { + goto fail; + } + } + s->bitmap_directory_offset = new_offset; + s->bitmap_directory_size = new_size; + s->nb_bitmaps = new_nb_bitmaps; + + ret =update_header_sync (bs); +if (ret <0 ) { + goto fail; + } + +if (old_size) { +qcow2_free_clusters (bs, old_offset, old_size,QCOW2_DISCARD_ALWAYS ); + } + +return 0 ; + +fail: +if (new_offset >0 ) {
Quoted 8 lines
block/qcow2-bitmap.c line 489
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+qcow2_free_clusters (bs, new_offset, new_size,QCOW2_DISCARD_ALWAYS ); + s->bitmap_directory_offset = old_offset; + s->bitmap_directory_size = old_size; + s->nb_bitmaps = old_nb_bitmaps; + s->autoclear_features = old_autocl;Why are you restoring the autoclear features? From a quick glance I can't see any code path that changes this field here, and if there is one, it probably has a good reason to do so.
hmm.. it's an artefact from future). should be moved to later patch.
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
Quoted 2 lines · Max Reitz
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are
...
Quoted 6 lines
block/qcow2-bitmap.c +4
-0+ goto fail; + } + +/* loop is safe because next entry offset is calculated after conversion to
Should it be s/safe/unsafe/?
Quoted 2 lines
loop is safe.
_unsafeis related to absence of assert in a loop, as it loops through BE data. Bad wording, I'll change it somehow..
Yes, I know that the loop is safe in the sense of the word "safe", but I meant that it's kind of confusing that the loop uses "for_each_bitmap_dir_entry_unsafe" and thus is "unsafe", too.
Another idea would be to write "This is actually safe" instead of "loop is safe".
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+* cpu format */ +for_each_bitmap_dir_entry_unsafe (e, dir, size) { +if ((uint8_t *)(e +1 ) > dir_end) { + goto broken_dir; + }block/qcow2-bitmap.c line 353
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ +bitmap_dir_entry_to_cpu (e); + +if ((uint8_t *)next_dir_entry (e) > dir_end) { + goto broken_dir; + } + +if (e->extra_data_size !=0 ) { + error_setg(errp, "Can't load bitmap '%.*s' from '%s':" + "extra data not supported.", e->name_size,Full stop again.
Quoted 8 lines
Quoted 4 lines
Also, I'm not quite sure why you give the device/node name here. You don't do that anywhere else and I think if we want to emit the information where something failed, it should be added at some previous point in the call chain.
For example, how? As I understand, I can emit it only by
error_setg, so actually it would be better to add node and bitmap name to allerror_setgin the code.. or create helper function for it.
error_prepend() would be the function.
This code will be invoked by any code that is opening an image. There are of course a couple of places where that can be the case: For external tools such as qemu-img, it's normally pretty clear which image is meant (and it additionally uses error_reportf_err() to give you more information).
For -drive, every error message will at least be prepended by the corresponding -drive parameter, so that will help a bit.
blockdev-add, unfortunately, doesn't do anything like this. But the user can of course choose to construct the BDS graph node by node and thus always know which node an error originates from.
Anyway, if you want to add this information to every error message, you should probably do so in bdrv_open_inherit(). The issue I'd take with this is that most users probably won't set the node name themselves (either they don't at all or it's some management tool that does), so reporting the node name doesn't actually help them at all (and management tools are not supposed to parse error messages, so it won't help in that case either).
Max
Re: [Qemu-devel] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps
Quoted 12 lines · Max Reitz
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
Quoted 2 lines · Max Reitz
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are
...
Quoted 6 lines
block/qcow2-bitmap.c +4
-0+ goto fail; + } + +/* loop is safe because next entry offset is calculated after conversion to
Should it be s/safe/unsafe/?
Quoted 8 lines
Quoted 2 lines
loop is safe.
_unsafeis related to absence of assert in a loop, as it loops through BE data. Bad wording, I'll change it somehow..Yes, I know that the loop is safe in the sense of the word "safe", but I meant that it's kind of confusing that the loop uses "
for_each_bitmap_dir_entry_unsafe" and thus is "unsafe", too.Another idea would be to write "This is actually safe" instead of "loop is safe".
Finally I've decided to introduce normal list of normal structures like in snapshots..
Quoted 48 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
block/qcow2-bitmap.c +5
-0+* cpu format */ +for_each_bitmap_dir_entry_unsafe (e, dir, size) { +if ((uint8_t *)(e +1 ) > dir_end) { + goto broken_dir; + }block/qcow2-bitmap.c line 353
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ +bitmap_dir_entry_to_cpu (e); + +if ((uint8_t *)next_dir_entry (e) > dir_end) { + goto broken_dir; + } + +if (e->extra_data_size !=0 ) { + error_setg(errp, "Can't load bitmap '%.*s' from '%s':" + "extra data not supported.", e->name_size,Full stop again.
Quoted 7 lines
Quoted 4 lines
Also, I'm not quite sure why you give the device/node name here. You don't do that anywhere else and I think if we want to emit the information where something failed, it should be added at some previous point in the call chain.
For example, how? As I understand, I can emit it only by
error_setg, so actually it would be better to add node and bitmap name to allerror_setgin the code.. or create helper function for it.
error_prepend()would be the function.This code will be invoked by any code that is opening an image. There are of course a couple of places where that can be the case: For external tools such as qemu-img, it's normally pretty clear which image is meant (and it additionally uses
error_reportf_err()to give you more information).For -drive, every error message will at least be prepended by the corresponding -drive parameter, so that will help a bit.
blockdev-add, unfortunately, doesn't do anything like this. But the user can of course choose to construct the BDS graph node by node and thus always know which node an error originates from.
Anyway, if you want to add this information to every error message, you should probably do so in
bdrv_open_inherit(). The issue I'd take with this is that most users probably won't set the node name themselves (either they don't at all or it's some management tool that does), so reporting the node name doesn't actually help them at all (and management tools are not supposed to parse error messages, so it won't help in that case either).
Quoted 2 lines
Max
Thanks for explanations, and for the whole review, it's great! Sorry for my laziness and for spelling( O! I've discovered vim spell, so one problem less, I hope.
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps
In reply to Max ReitzQuoted 2 lines · Max Reitz
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are
[...]
Quoted 16 lines
block/qcow2.c +5
-0@@ -213,6 +213,11 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, s->bitmap_directory_size = bitmaps_ext.bitmap_directory_size; + ret =qcow2_read_bitmaps (bs, errp); +if (ret <0 ) { +return ret; + } +I think I'd put this directly into
qcow2_open(), just likeqcow2_read_snapshots(); but that's an optional suggestion.
Quoted 3 lines
Max
Snapshots are not header extension.. so it is not the case. Here qcow2_read_bitmaps looks like part of header extension loading, and header extension fields describe other parts of the extension.. I think this is a good point, isn't it?
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 2 lines · Max Reitz
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are
[...]
Quoted 16 lines
block/qcow2.c +4
-0@@ -213,6 +213,11 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, s->bitmap_directory_size = bitmaps_ext.bitmap_directory_size; + ret =qcow2_read_bitmaps (bs, errp); +if (ret <0 ) { +return ret; + } +I think I'd put this directly into
qcow2_open(), just likeqcow2_read_snapshots(); but that's an optional suggestion.
Quoted 8 lines
Quoted 3 lines
Max
Snapshots are not header extension.. so it is not the case. Here
qcow2_read_bitmapslooks like part of header extension loading, and header extension fields describe other parts of the extension.. I think this is a good point, isn't it?
I said it's optional, so it's optional. :-)
I personally feel like a header extension is just the bit of data in the qcow2 header. The bitmaps itself are managed by that extension, but not part of the extension itself. Therefore, I wouldn't load it here but in the main function qcow2_open().
However, this is a personal opinion and thus it's an optional suggestion (as I said), so if you disagree (which you apparently do) then don't let it bother you. :-)
Max
Re: [Qemu-devel] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps
In reply to Vladimir Sementsov-OgievskiyQuoted 18 lines · Vladimir Sementsov-Ogievskiy
Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are loaded at image open and becomes BdrvDirtyBitmap's for corresponding drive. These bitmaps are deleted from Qcow2 image after loading to avoid conflicts.
Extra data in bitmaps is not supported for now.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 518 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/qcow2.c | 5 + block/qcow2.h | 3 + 3 files changed, 525 insertions(+), 1 deletion(-)block/qcow2-bitmap.c +0
-0
[...]
Quoted 42 lines
block/qcow2-bitmap.c +42
-0+static int directory_update (BlockDriverState *bs,uint8_t *new_dir, +size_t new_size,uint32_t new_nb_bitmaps) +{ +BDRVQcow2State *s = bs->opaque; +int ret; +int64_t new_offset =0 ; +uint64_t old_offset = s->bitmap_directory_offset; +uint64_t old_size = s->bitmap_directory_size; +uint32_t old_nb_bitmaps = s->nb_bitmaps; +uint64_t old_autocl = s->autoclear_features; + +if (new_size >QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +return -EINVAL ; + } + +if ((new_size ==0 ) != (new_nb_bitmaps ==0 )) { +return -EINVAL ; + } + +if (new_nb_bitmaps >0 ) { + new_offset =directory_write (bs, new_dir, new_size); +if (new_offset <0 ) { +return new_offset; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { + goto fail; + } + } + s->bitmap_directory_offset = new_offset; + s->bitmap_directory_size = new_size; + s->nb_bitmaps = new_nb_bitmaps; + + ret =update_header_sync (bs); +if (ret <0 ) { + goto fail; + } + +if (old_size) { +qcow2_free_clusters (bs, old_offset, old_size,QCOW2_DISCARD_ALWAYS ); + }
block/qcow2-bitmap.c line 489
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ +return 0 ; + +fail: +if (new_offset >0 ) { +qcow2_free_clusters (bs, new_offset, new_size,QCOW2_DISCARD_ALWAYS ); + s->bitmap_directory_offset = old_offset; + s->bitmap_directory_size = old_size; + s->nb_bitmaps = old_nb_bitmaps; + s->autoclear_features = old_autocl;
What if bdrv_flush() in update_header_sync() failed? Then you cannot be sure what bitmap directory the header is actually pointing to. In that case, it would be wrong to assume it's still pointing to the old one and freeing the new one.
Max
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+ } + +return ret; +}
Re: [Qemu-devel] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps
07.10.2016 22:25, Max Reitz пишет:
Quoted 78 lines · Vladimir Sementsov-Ogievskiy
Quoted 18 lines
Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are loaded at image open and becomes BdrvDirtyBitmap's for corresponding drive. These bitmaps are deleted from Qcow2 image after loading to avoid conflicts.
Extra data in bitmaps is not supported for now.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 518 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/qcow2.c | 5 + block/qcow2.h | 3 + 3 files changed, 525 insertions(+), 1 deletion(-)block/qcow2-bitmap.c +0
-0[...]
Quoted 42 lines
block/qcow2-bitmap.c +42
-0+static int directory_update (BlockDriverState *bs,uint8_t *new_dir, +size_t new_size,uint32_t new_nb_bitmaps) +{ +BDRVQcow2State *s = bs->opaque; +int ret; +int64_t new_offset =0 ; +uint64_t old_offset = s->bitmap_directory_offset; +uint64_t old_size = s->bitmap_directory_size; +uint32_t old_nb_bitmaps = s->nb_bitmaps; +uint64_t old_autocl = s->autoclear_features; + +if (new_size >QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +return -EINVAL ; + } + +if ((new_size ==0 ) != (new_nb_bitmaps ==0 )) { +return -EINVAL ; + } + +if (new_nb_bitmaps >0 ) { + new_offset =directory_write (bs, new_dir, new_size); +if (new_offset <0 ) { +return new_offset; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { + goto fail; + } + } + s->bitmap_directory_offset = new_offset; + s->bitmap_directory_size = new_size; + s->nb_bitmaps = new_nb_bitmaps; + + ret =update_header_sync (bs); +if (ret <0 ) { + goto fail; + } + +if (old_size) { +qcow2_free_clusters (bs, old_offset, old_size,QCOW2_DISCARD_ALWAYS ); + }block/qcow2-bitmap.c line 489
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ +return 0 ; + +fail: +if (new_offset >0 ) { +qcow2_free_clusters (bs, new_offset, new_size,QCOW2_DISCARD_ALWAYS ); + s->bitmap_directory_offset = old_offset; + s->bitmap_directory_size = old_size; + s->nb_bitmaps = old_nb_bitmaps; + s->autoclear_features = old_autocl;What if
bdrv_flush()inupdate_header_sync()failed? Then you cannot be sure what bitmap directory the header is actually pointing to. In that case, it would be wrong to assume it's still pointing to the old one and freeing the new one.
Quoted 1 line
Max
Hmm.. Good point. Any suggestions?
I'm trying to achieve some kind of transaction - if function failed file is not changed.. But now I understand that because of flush ability to fail I can't achieve this..
Has write and flush any guaranties in case of fail? Would it be old-or-new state, or some mixed state is possible?
Quoted 5 lines
Quoted 5 lines
block/qcow2-bitmap.c +4
-0+ } + +return ret; +}
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps
Quoted 84 lines · Vladimir Sementsov-Ogievskiy
07.10.2016 22:25, Max Reitz пишет:
Quoted 81 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines
Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are loaded at image open and becomes BdrvDirtyBitmap's for corresponding drive. These bitmaps are deleted from Qcow2 image after loading to avoid conflicts.
Extra data in bitmaps is not supported for now.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 518 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/qcow2.c | 5 + block/qcow2.h | 3 + 3 files changed, 525 insertions(+), 1 deletion(-)block/qcow2-bitmap.c +0
-0[...]
Quoted 50 lines
block/qcow2-bitmap.c +41
-0+static int directory_update (BlockDriverState *bs,uint8_t *new_dir, +size_t new_size,uint32_t new_nb_bitmaps) +{ +BDRVQcow2State *s = bs->opaque; +int ret; +int64_t new_offset =0 ; +uint64_t old_offset = s->bitmap_directory_offset; +uint64_t old_size = s->bitmap_directory_size; +uint32_t old_nb_bitmaps = s->nb_bitmaps; +uint64_t old_autocl = s->autoclear_features; + +if (new_size >QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE ) { +return -EINVAL ; + } + +if ((new_size ==0 ) != (new_nb_bitmaps ==0 )) { +return -EINVAL ; + } + +if (new_nb_bitmaps >0 ) { + new_offset =directory_write (bs, new_dir, new_size); +if (new_offset <0 ) { +return new_offset; + } + + ret =bdrv_flush (bs); +if (ret <0 ) { + goto fail; + } + } + s->bitmap_directory_offset = new_offset; + s->bitmap_directory_size = new_size; + s->nb_bitmaps = new_nb_bitmaps; + + ret =update_header_sync (bs); +if (ret <0 ) { + goto fail; + } + +if (old_size) { + qcow2_free_clusters(bs, old_offset, old_size,
QCOW2_DISCARD_ALWAYS);block/qcow2-bitmap.c +7
-0+ } + +return 0 ; + +fail: +if (new_offset >0 ) { + qcow2_free_clusters(bs, new_offset, new_size,
QCOW2_DISCARD_ALWAYS);block/qcow2-bitmap.c line 489
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+ s->bitmap_directory_offset = old_offset; + s->bitmap_directory_size = old_size; + s->nb_bitmaps = old_nb_bitmaps; + s->autoclear_features = old_autocl;What if
bdrv_flush()inupdate_header_sync()failed? Then you cannot be sure what bitmap directory the header is actually pointing to. In that case, it would be wrong to assume it's still pointing to the old one and freeing the new one.Quoted 1 line
Max
Quoted 8 lines
Hmm.. Good point. Any suggestions?
I'm trying to achieve some kind of transaction - if function failed file is not changed.. But now I understand that because of flush ability to fail I can't achieve this..
Has write and flush any guaranties in case of fail? Would it be old-or-new state, or some mixed state is possible?
Anything is possible, I think. However, if flushing the data fails, it still means writing it was successful -- you just don't know whether it has been written to the disk successfully.
The most correct way out would be to signal image corruption, but that isn't what we should do, I think.
If the write operation was successful but flushing was not, I'd always assume the data hopefully gets flushed later and continue as if it had been written to disk successfully. Therefore, I think it would be best for update_header_sync() to just ignore bdrv_flush()'s return value.
Max
PATCH v7 8/22 block/dirty-bitmap: add autoload field to BdrvDirtyBitmap
In reply to Vladimir Sementsov-OgievskiyMirror AUTO flag from Qcow2 bitmap in BdrvDirtyBitmap. This will be needed in future, to save this flag back to Qcow2 for persistent bitmaps.
--- block/dirty-bitmap.c | 15 +++++++++++++++ block/qcow2-bitmap.c | 2 ++ include/block/dirty-bitmap.h | 2 ++ 3 files changed, 19 insertions(+)
block/dirty-bitmap.c +15 -0
@@ -44,6 +44,7 @@ struct BdrvDirtyBitmap {int64_t size;/* Size of the bitmap (Number of sectors) */ bool disabled;/* Bitmap is read-only */ int active_iterators;/* How many iterators are active */ +bool autoload;/* bitmap must be autoloaded on image opening */ QLIST_ENTRY (BdrvDirtyBitmap) list; };
@@ -70,6 +71,8 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)assert (!bdrv_dirty_bitmap_frozen (bitmap));g_free (bitmap->name); bitmap->name =NULL ; + + bitmap->autoload = false; } BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
@@ -238,6 +241,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs, bitmap->name =NULL ; successor->name = name; bitmap->successor =NULL ; + successor->autoload = bitmap->autoload; + bitmap->autoload = false;bdrv_release_dirty_bitmap (bs, bitmap);return successor;
@@ -540,3 +545,13 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap) {return hbitmap_count (bitmap->meta); } + +void bdrv_dirty_bitmap_set_autoload (BdrvDirtyBitmap *bitmap,bool autoload) +{ + bitmap->autoload = autoload; +} + +bool bdrv_dirty_bitmap_get_autoload (const BdrvDirtyBitmap *bitmap) +{ +return bitmap->autoload; +}
block/qcow2-bitmap.c +2 -0
@@ -529,6 +529,8 @@ int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp) ret = -EINVAL ; goto out; } + +bdrv_dirty_bitmap_set_autoload (bitmap, true); }else {/* leave bitmap in the image */ new_pos +=copy_dir_entry (new_pos, e);
include/block/dirty-bitmap.h +2 -0
@@ -75,4 +75,6 @@ void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap, bool finish);void bdrv_dirty_bitmap_deserialize_finish (BdrvDirtyBitmap *bitmap); +void bdrv_dirty_bitmap_set_autoload (BdrvDirtyBitmap *bitmap,bool autoload); +bool bdrv_dirty_bitmap_get_autoload (const BdrvDirtyBitmap *bitmap);#endif
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 08/22] block/dirty-bitmap: add autoload field to BdrvDirtyBitmap
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Mirror AUTO flag from Qcow2 bitmap in BdrvDirtyBitmap. This will be needed in future, to save this flag back to Qcow2 for persistent bitmaps.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/dirty-bitmap.c | 15 +++++++++++++++ block/qcow2-bitmap.c | 2 ++ include/block/dirty-bitmap.h | 2 ++ 3 files changed, 19 insertions(+)block/dirty-bitmap.c +0
-0
block/dirty-bitmap.c struct BdrvDirtyBitmap { · line 47
Quoted 4 lines
block/dirty-bitmap.c +1
-0@@ -44,6 +44,7 @@ struct BdrvDirtyBitmap {int64_t size;/* Size of the bitmap (Number of sectors) */ bool disabled;/* Bitmap is read-only */ int active_iterators;/* How many iterators are active */ +bool autoload;/* bitmap must be autoloaded on image opening */
I think a note that this is for persistent dirty bitmaps would be useful, e.g.:
"/* For persistent bitmaps: Bitmap must be autoloaded [...] */"
Up to you, though, so however you decide:
PATCH v7 9/22 block: introduce persistent dirty bitmaps
In reply to Vladimir Sementsov-OgievskiyNew field BdrvDirtyBitmap.persistent means, that bitmap should be saved on bdrv_close, using format driver. Format driver should maintain bitmap storing.
--- block.c | 30 ++++++++++++++++++++++++++++++ block/dirty-bitmap.c | 27 +++++++++++++++++++++++++++ block/qcow2-bitmap.c | 1 + include/block/block.h | 2 ++ include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 6 ++++++ 6 files changed, 68 insertions(+)
block.c +30 -0
@@ -2196,6 +2196,7 @@ void bdrv_reopen_abort(BDRVReopenState *reopen_state)static void bdrv_close (BlockDriverState *bs) {BdrvAioNotifier *ban, *ban_next; +Error *local_err =NULL ;assert (!bs->job);assert (!bs->refcnt);
@@ -2204,6 +2205,10 @@ static void bdrv_close(BlockDriverState *bs)bdrv_flush (bs);bdrv_drain (bs);/* in case flush left pending I/O */ +bdrv_store_persistent_bitmaps (bs, &local_err); +if (local_err !=NULL ) { +error_report_err (local_err); + }bdrv_release_named_dirty_bitmaps (bs);assert (QLIST_EMPTY (&bs->dirty_bitmaps));
@@ -3969,3 +3974,28 @@ void bdrv_del_child(BlockDriverState *parent_bs, BdrvChild *child, Error **errp) parent_bs->drv->bdrv_del_child (parent_bs, child, errp); } + +void bdrv_store_persistent_bitmaps (BlockDriverState *bs,Error **errp) +{ +BlockDriver *drv = bs->drv; + +if (!bdrv_has_persistent_bitmaps (bs)) { +return ; + } + +if (!drv) { +error_setg_errno (errp,ENOMEDIUM , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return ; + } + +if (!drv->bdrv_store_persistent_bitmaps) { +error_setg_errno (errp,ENOTSUP , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return ; + } + + drv->bdrv_store_persistent_bitmaps (bs, errp); +}
block/dirty-bitmap.c +27 -0
@@ -44,6 +44,7 @@ struct BdrvDirtyBitmap {int64_t size;/* Size of the bitmap (Number of sectors) */ bool disabled;/* Bitmap is read-only */ int active_iterators;/* How many iterators are active */ +bool persistent;/* bitmap must be saved to owner disk image */ bool autoload;/* bitmap must be autoloaded on image opening */ QLIST_ENTRY (BdrvDirtyBitmap) list; };
@@ -72,6 +73,7 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)g_free (bitmap->name); bitmap->name =NULL ; + bitmap->persistent = false; bitmap->autoload = false; }
@@ -241,6 +243,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs, bitmap->name =NULL ; successor->name = name; bitmap->successor =NULL ; + successor->persistent = bitmap->persistent; + bitmap->persistent = false; successor->autoload = bitmap->autoload; bitmap->autoload = false;bdrv_release_dirty_bitmap (bs, bitmap);
@@ -555,3 +559,26 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) {return bitmap->autoload; } + +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap, +bool persistent) +{ + bitmap->persistent = persistent; +} + +bool bdrv_dirty_bitmap_get_persistance (BdrvDirtyBitmap *bitmap) +{ +return bitmap->persistent; +} + +bool bdrv_has_persistent_bitmaps (BlockDriverState *bs) +{ +BdrvDirtyBitmap *bm; + QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) { +if (bm->persistent) { +return true; + } + } + +return false; +}
block/qcow2-bitmap.c +1 -0
@@ -530,6 +530,7 @@ int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp) goto out; } +bdrv_dirty_bitmap_set_persistance (bitmap, true);bdrv_dirty_bitmap_set_autoload (bitmap, true); }else {/* leave bitmap in the image */
include/block/block.h +2 -0
@@ -532,4 +532,6 @@ void bdrv_add_child(BlockDriverState *parent, BlockDriverState *child, Error **errp);void bdrv_del_child (BlockDriverState *parent,BdrvChild *child,Error **errp); +void bdrv_store_persistent_bitmaps (BlockDriverState *bs,Error **errp); +#endif
include/block/block_int.h +2 -0
@@ -320,6 +320,8 @@ struct BlockDriver {void (*bdrv_del_child)(BlockDriverState *parent,BdrvChild *child,Error **errp); +void (*bdrv_store_persistent_bitmaps)(BlockDriverState *bs,Error **errp); +QLIST_ENTRY (BlockDriver) list; };
include/block/dirty-bitmap.h +6 -0
@@ -77,4 +77,10 @@ void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);void bdrv_dirty_bitmap_set_autoload (BdrvDirtyBitmap *bitmap,bool autoload);bool bdrv_dirty_bitmap_get_autoload (const BdrvDirtyBitmap *bitmap); +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap, +bool persistent); +bool bdrv_dirty_bitmap_get_persistance (BdrvDirtyBitmap *bitmap); + +bool bdrv_has_persistent_bitmaps (BlockDriverState *bs); +#endif
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 09/22] block: introduce persistent dirty bitmaps
Quoted 27 lines · Vladimir Sementsov-Ogievskiy
New field BdrvDirtyBitmap.persistent means, that bitmap should be saved on
bdrv_close, using format driver. Format driver should maintain bitmap storing.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block.c | 30 ++++++++++++++++++++++++++++++ block/dirty-bitmap.c | 27 +++++++++++++++++++++++++++ block/qcow2-bitmap.c | 1 + include/block/block.h | 2 ++ include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 6 ++++++ 6 files changed, 68 insertions(+)block.c +1
-0@@ -2196,6 +2196,7 @@ void bdrv_reopen_abort(BDRVReopenState *reopen_state)static void bdrv_close (BlockDriverState *bs) {BdrvAioNotifier *ban, *ban_next; +Error *local_err =NULL ;assert (!bs->job);assert (!bs->refcnt);
block.c static void bdrv_close(BlockDriverState *bs) · line 2211
Quoted 7 lines
block.c +4
-0@@ -2204,6 +2205,10 @@ static void bdrv_close(BlockDriverState *bs)bdrv_flush (bs);bdrv_drain (bs);/* in case flush left pending I/O */ +bdrv_store_persistent_bitmaps (bs, &local_err); +if (local_err !=NULL ) { +error_report_err (local_err); + }
That seems pretty wrong to me. If the persistent bitmaps cannot be stored, the node should not be closed to avoid loss of data.
block.c static void bdrv_close(BlockDriverState *bs) · line 2212
Quoted 1 line
block.c +0
-0bdrv_release_named_dirty_bitmaps (bs);
Especially since the next function will just drop all the dirty bitmaps.
I see the issue that bdrv_close() is only called by bdrv_delete() which in turn is only called by bdrv_unref(); and how are you supposed to react to bdrv_unref() failing?
So I'm not sure how this issue should be addressed, but this is most certainly not ideal. You should not just drop supposedly persistent dirty bitmaps if they cannot be saved.
We really should to have some way to keep the bitmap around if it cannot be saved, but I don't know how to do that either.
In any case, we should make sure that the node supports saving persistent dirty bitmaps, because having persistent dirty bitmaps at a node that does not support them is something we can and must prevent beforehand.
But I don't know how to handle failure if writing the dirty bitmap fails. I guess one could argue that it's the same as bdrv_flush() failing and thus can be handled in the same way, i.e. ignore it. I'm not happy with that, but I'd accept it if there's no other way.
Quoted 61 lines
block.c +25
-0assert (QLIST_EMPTY (&bs->dirty_bitmaps));@@ -3969,3 +3974,28 @@ void bdrv_del_child(BlockDriverState *parent_bs, BdrvChild *child, Error **errp) parent_bs->drv->bdrv_del_child (parent_bs, child, errp); } + +void bdrv_store_persistent_bitmaps (BlockDriverState *bs,Error **errp) +{ +BlockDriver *drv = bs->drv; + +if (!bdrv_has_persistent_bitmaps (bs)) { +return ; + } + +if (!drv) { +error_setg_errno (errp,ENOMEDIUM , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return ; + } + +if (!drv->bdrv_store_persistent_bitmaps) { +error_setg_errno (errp,ENOTSUP , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return ; + } + + drv->bdrv_store_persistent_bitmaps (bs, errp); +}block/dirty-bitmap.c +4
-0@@ -44,6 +44,7 @@ struct BdrvDirtyBitmap {int64_t size;/* Size of the bitmap (Number of sectors) */ bool disabled;/* Bitmap is read-only */ int active_iterators;/* How many iterators are active */ +bool persistent;/* bitmap must be saved to owner disk image */ bool autoload;/* bitmap must be autoloaded on image opening */ QLIST_ENTRY (BdrvDirtyBitmap) list; };@@ -72,6 +73,7 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)g_free (bitmap->name); bitmap->name =NULL ; + bitmap->persistent = false; bitmap->autoload = false; }@@ -241,6 +243,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs, bitmap->name =NULL ; successor->name = name; bitmap->successor =NULL ; + successor->persistent = bitmap->persistent; + bitmap->persistent = false; successor->autoload = bitmap->autoload; bitmap->autoload = false;bdrv_release_dirty_bitmap (bs, bitmap);
block/dirty-bitmap.c bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) · line 564
Quoted 6 lines
block/dirty-bitmap.c +3
-0@@ -555,3 +559,26 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) {return bitmap->autoload; } + +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap, +bool persistent)
The second parameter should be aligned to the opening parenthesis.
Max
Quoted 20 lines
block/dirty-bitmap.c +20
-0+{ + bitmap->persistent = persistent; +} + +bool bdrv_dirty_bitmap_get_persistance (BdrvDirtyBitmap *bitmap) +{ +return bitmap->persistent; +} + +bool bdrv_has_persistent_bitmaps (BlockDriverState *bs) +{ +BdrvDirtyBitmap *bm; + QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) { +if (bm->persistent) { +return true; + } + } + +return false; +}
Re: [Qemu-devel] [PATCH 09/22] block: introduce persistent dirty bitmaps
Quoted 57 lines · Max Reitz
Quoted 27 lines · Vladimir Sementsov-Ogievskiy
New field BdrvDirtyBitmap.persistent means, that bitmap should be saved on
bdrv_close, using format driver. Format driver should maintain bitmap storing.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block.c | 30 ++++++++++++++++++++++++++++++ block/dirty-bitmap.c | 27 +++++++++++++++++++++++++++ block/qcow2-bitmap.c | 1 + include/block/block.h | 2 ++ include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 6 ++++++ 6 files changed, 68 insertions(+)block.c +1
-0@@ -2196,6 +2196,7 @@ void bdrv_reopen_abort(BDRVReopenState *reopen_state)static void bdrv_close (BlockDriverState *bs) {BdrvAioNotifier *ban, *ban_next; +Error *local_err =NULL ;assert (!bs->job);assert (!bs->refcnt);block.c static void bdrv_close(BlockDriverState *bs) · line 2211
Quoted 7 lines
block.c +4
-0@@ -2204,6 +2205,10 @@ static void bdrv_close(BlockDriverState *bs)bdrv_flush (bs);bdrv_drain (bs);/* in case flush left pending I/O */ +bdrv_store_persistent_bitmaps (bs, &local_err); +if (local_err !=NULL ) { +error_report_err (local_err); + }That seems pretty wrong to me. If the persistent bitmaps cannot be stored, the node should not be closed to avoid loss of data.
block.c static void bdrv_close(BlockDriverState *bs) · line 2212
Quoted 1 line
block.c +0
-0bdrv_release_named_dirty_bitmaps (bs);Especially since the next function will just drop all the dirty bitmaps.
I see the issue that
bdrv_close()is only called bybdrv_delete()which in turn is only called bybdrv_unref(); and how are you supposed to react tobdrv_unref()failing?So I'm not sure how this issue should be addressed, but this is most certainly not ideal. You should not just drop supposedly persistent dirty bitmaps if they cannot be saved.
We really should to have some way to keep the bitmap around if it cannot be saved, but I don't know how to do that either.
In any case, we should make sure that the node supports saving persistent dirty bitmaps, because having persistent dirty bitmaps at a node that does not support them is something we can and must prevent beforehand.
Quoted 4 lines
But I don't know how to handle failure if writing the dirty bitmap fails. I guess one could argue that it's the same as
bdrv_flush()failing and thus can be handled in the same way, i.e. ignore it. I'm not happy with that, but I'd accept it if there's no other way.
For now, the only usage of these bitmaps is incremental backup and bitmaps are not critical data. If we lost them we will just do full backup. If there will be some critical persistent bdrv dirty bitmaps in future, we can introduce a callback BdrvDirtyBitmap.store_failed for them, which will somehow handle that case.. Detach bitmap from bs and save it in memory, add qmp commands to raw-dump them, etc.. I
Quoted 92 lines
Quoted 62 lines
block.c +25
-0assert (QLIST_EMPTY (&bs->dirty_bitmaps));@@ -3969,3 +3974,28 @@ void bdrv_del_child(BlockDriverState *parent_bs, BdrvChild *child, Error **errp) parent_bs->drv->bdrv_del_child (parent_bs, child, errp); } + +void bdrv_store_persistent_bitmaps (BlockDriverState *bs,Error **errp) +{ +BlockDriver *drv = bs->drv; + +if (!bdrv_has_persistent_bitmaps (bs)) { +return ; + } + +if (!drv) { +error_setg_errno (errp,ENOMEDIUM , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return ; + } + +if (!drv->bdrv_store_persistent_bitmaps) { +error_setg_errno (errp,ENOTSUP , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return ; + } + + drv->bdrv_store_persistent_bitmaps (bs, errp); +}block/dirty-bitmap.c +4
-0@@ -44,6 +44,7 @@ struct BdrvDirtyBitmap {int64_t size;/* Size of the bitmap (Number of sectors) */ bool disabled;/* Bitmap is read-only */ int active_iterators;/* How many iterators are active */ +bool persistent;/* bitmap must be saved to owner disk image */ bool autoload;/* bitmap must be autoloaded on image opening */ QLIST_ENTRY (BdrvDirtyBitmap) list; };@@ -72,6 +73,7 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)g_free (bitmap->name); bitmap->name =NULL ; + bitmap->persistent = false; bitmap->autoload = false; }@@ -241,6 +243,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs, bitmap->name =NULL ; successor->name = name; bitmap->successor =NULL ; + successor->persistent = bitmap->persistent; + bitmap->persistent = false; successor->autoload = bitmap->autoload; bitmap->autoload = false;bdrv_release_dirty_bitmap (bs, bitmap);block/dirty-bitmap.c bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) · line 564
Quoted 6 lines
block/dirty-bitmap.c +3
-0@@ -555,3 +559,26 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) {return bitmap->autoload; } + +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap, +bool persistent)The second parameter should be aligned to the opening parenthesis.
Max
Quoted 20 lines
block/dirty-bitmap.c +20
-0+{ + bitmap->persistent = persistent; +} + +bool bdrv_dirty_bitmap_get_persistance (BdrvDirtyBitmap *bitmap) +{ +return bitmap->persistent; +} + +bool bdrv_has_persistent_bitmaps (BlockDriverState *bs) +{ +BdrvDirtyBitmap *bm; + QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) { +if (bm->persistent) { +return true; + } + } + +return false; +}
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 09/22] block: introduce persistent dirty bitmaps
Quoted 62 lines · Vladimir Sementsov-Ogievskiy
Quoted 56 lines · Max Reitz
Quoted 27 lines · Vladimir Sementsov-Ogievskiy
New field BdrvDirtyBitmap.persistent means, that bitmap should be saved on
bdrv_close, using format driver. Format driver should maintain bitmap storing.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block.c | 30 ++++++++++++++++++++++++++++++ block/dirty-bitmap.c | 27 +++++++++++++++++++++++++++ block/qcow2-bitmap.c | 1 + include/block/block.h | 2 ++ include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 6 ++++++ 6 files changed, 68 insertions(+)block.c +1
-0@@ -2196,6 +2196,7 @@ void bdrv_reopen_abort(BDRVReopenState *reopen_state)static void bdrv_close (BlockDriverState *bs) {BdrvAioNotifier *ban, *ban_next; +Error *local_err =NULL ;assert (!bs->job);assert (!bs->refcnt);block.c static void bdrv_close(BlockDriverState *bs) · line 2210
Quoted 6 lines
block.c +3
-0@@ -2204,6 +2205,10 @@ static void bdrv_close(BlockDriverState *bs)bdrv_flush (bs);bdrv_drain (bs);/* in case flush left pending I/O */ +bdrv_store_persistent_bitmaps (bs, &local_err); +if (local_err !=NULL ) { +error_report_err (local_err); + }That seems pretty wrong to me. If the persistent bitmaps cannot be stored, the node should not be closed to avoid loss of data.
block.c static void bdrv_close(BlockDriverState *bs) · line 2212
Quoted 1 line
block.c +0
-0bdrv_release_named_dirty_bitmaps (bs);Especially since the next function will just drop all the dirty bitmaps.
I see the issue that
bdrv_close()is only called bybdrv_delete()which in turn is only called bybdrv_unref(); and how are you supposed to react tobdrv_unref()failing?So I'm not sure how this issue should be addressed, but this is most certainly not ideal. You should not just drop supposedly persistent dirty bitmaps if they cannot be saved.
We really should to have some way to keep the bitmap around if it cannot be saved, but I don't know how to do that either.
In any case, we should make sure that the node supports saving persistent dirty bitmaps, because having persistent dirty bitmaps at a node that does not support them is something we can and must prevent beforehand.
Quoted 4 lines
But I don't know how to handle failure if writing the dirty bitmap fails. I guess one could argue that it's the same as
bdrv_flush()failing and thus can be handled in the same way, i.e. ignore it. I'm not happy with that, but I'd accept it if there's no other way.
Quoted 6 lines
For now, the only usage of these bitmaps is incremental backup and bitmaps are not critical data. If we lost them we will just do full backup. If there will be some critical persistent bdrv dirty bitmaps in future, we can introduce a callback BdrvDirtyBitmap.store_failed for them, which will somehow handle that case.. Detach bitmap from bs and save it in memory, add qmp commands to raw-dump them, etc.. I
Yes, fine with me. Still, we should make an effort to detect the case that some block driver will not be able to store a certain persistent bitmap attached to one of its nodes as early as possible, ideally already when the bitmap is created.
Max
Re: [Qemu-devel] [PATCH 09/22] block: introduce persistent dirty bitmaps
In reply to Vladimir Sementsov-OgievskiyQuoted 13 lines · Vladimir Sementsov-Ogievskiy
New field BdrvDirtyBitmap.persistent means, that bitmap should be saved on
bdrv_close, using format driver. Format driver should maintain bitmap storing.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block.c | 30 ++++++++++++++++++++++++++++++ block/dirty-bitmap.c | 27 +++++++++++++++++++++++++++ block/qcow2-bitmap.c | 1 + include/block/block.h | 2 ++ include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 6 ++++++ 6 files changed, 68 insertions(+)
[...]
Quoted 4 lines
block/dirty-bitmap.c +0
-0
[...]
block/dirty-bitmap.c bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) · line 566
Quoted 9 lines
block/dirty-bitmap.c +5
-0@@ -555,3 +559,26 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) {return bitmap->autoload; } + +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap, +bool persistent) +{ + bitmap->persistent = persistent;
After some thinking, I think this function should be more complex: It should check whether the node the bitmap is attached to actually can handle persistent bitmaps and whether it would actually support storing *this* bitmap.
For instance, a qcow2 node would not support writing overly large bitmaps (limited by BME_MAX_TABLE_SIZE and BME_MAX_PHYS_SIZE) or bitmaps with overly large granularities (BME_MAX_GRANULARITY_BITS) or bitmaps whose name is already occupied by some bitmap that is already stored in the file but has not been loaded.
Checking this here will trivially prevent users from creating such bitmaps and will also preempt detection of such failures during bdrv_close() when they cannot be handled gracefully.
Max
Quoted 1 line
block/dirty-bitmap.c +1
-0+}
Re: [Qemu-devel] [PATCH 09/22] block: introduce persistent dirty bitmaps
Quoted 42 lines · Max Reitz
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
New field BdrvDirtyBitmap.persistent means, that bitmap should be saved on
bdrv_close, using format driver. Format driver should maintain bitmap storing.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block.c | 30 ++++++++++++++++++++++++++++++ block/dirty-bitmap.c | 27 +++++++++++++++++++++++++++ block/qcow2-bitmap.c | 1 + include/block/block.h | 2 ++ include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 6 ++++++ 6 files changed, 68 insertions(+)[...]
Quoted 4 lines
block/dirty-bitmap.c +0
-0[...]
block/dirty-bitmap.c bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) · line 566
Quoted 9 lines
block/dirty-bitmap.c +5
-0@@ -555,3 +559,26 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) {return bitmap->autoload; } + +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap, +bool persistent) +{ + bitmap->persistent = persistent;After some thinking, I think this function should be more complex: It should check whether the node the bitmap is attached to actually can handle persistent bitmaps and whether it would actually support storing *this* bitmap.
For instance, a qcow2 node would not support writing overly large bitmaps (limited by
BME_MAX_TABLE_SIZEandBME_MAX_PHYS_SIZE) or bitmaps with overly large granularities (BME_MAX_GRANULARITY_BITS) or bitmaps whose name is already occupied by some bitmap that is already stored in the file but has not been loaded.
Quoted 5 lines
Checking this here will trivially prevent users from creating such bitmaps and will also preempt detection of such failures during
bdrv_close()when they cannot be handled gracefully.Max
Good point, but I can't do it exactly as you say, because I call this function from qcow2_read_bitmaps, for just created bitmap and it should not be checked and of course it's name is occupied..
Quoted 2 lines
Quoted 2 lines
block/dirty-bitmap.c +1
-0+}
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 09/22] block: introduce persistent dirty bitmaps
Quoted 44 lines · Vladimir Sementsov-Ogievskiy
Quoted 43 lines · Max Reitz
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
New field BdrvDirtyBitmap.persistent means, that bitmap should be saved on
bdrv_close, using format driver. Format driver should maintain bitmap storing.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block.c | 30 ++++++++++++++++++++++++++++++ block/dirty-bitmap.c | 27 +++++++++++++++++++++++++++ block/qcow2-bitmap.c | 1 + include/block/block.h | 2 ++ include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 6 ++++++ 6 files changed, 68 insertions(+)[...]
Quoted 4 lines
block/dirty-bitmap.c +0
-0[...]
block/dirty-bitmap.c bool bdrv_dirty_bitmap_get_autoload(const · line 566
Quoted 10 lines
block/dirty-bitmap.c +5
-0@@ -555,3 +559,26 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) {return bitmap->autoload; } + +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap, +bool persistent) +{ + bitmap->persistent = persistent;After some thinking, I think this function should be more complex: It should check whether the node the bitmap is attached to actually can handle persistent bitmaps and whether it would actually support storing *this* bitmap.
For instance, a qcow2 node would not support writing overly large bitmaps (limited by
BME_MAX_TABLE_SIZEandBME_MAX_PHYS_SIZE) or bitmaps with overly large granularities (BME_MAX_GRANULARITY_BITS) or bitmaps whose name is already occupied by some bitmap that is already stored in the file but has not been loaded.
Quoted 7 lines
Quoted 5 lines
Checking this here will trivially prevent users from creating such bitmaps and will also preempt detection of such failures during
bdrv_close()when they cannot be handled gracefully.Max
Good point, but I can't do it exactly as you say, because I call this function from
qcow2_read_bitmaps, for just created bitmap and it should not be checked and of course it's name is occupied..
So, I'll add an additional checking function, to call it from qmp_block_dirty_bitmap_add, if persistent parameter is set to true.
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
+}
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 09/22] block: introduce persistent dirty bitmaps
Quoted 50 lines · Vladimir Sementsov-Ogievskiy
Quoted 49 lines · Vladimir Sementsov-Ogievskiy
Quoted 48 lines · Max Reitz
Quoted 14 lines · Vladimir Sementsov-Ogievskiy
New field BdrvDirtyBitmap.persistent means, that bitmap should be saved on
bdrv_close, using format driver. Format driver should maintain bitmap
storing.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block.c | 30 ++++++++++++++++++++++++++++++ block/dirty-bitmap.c | 27 +++++++++++++++++++++++++++ block/qcow2-bitmap.c | 1 + include/block/block.h | 2 ++ include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 6 ++++++ 6 files changed, 68 insertions(+)[...]
Quoted 4 lines
block/dirty-bitmap.c +0
-0[...]
block/dirty-bitmap.c bool bdrv_dirty_bitmap_get_autoload(const · line 566
Quoted 10 lines
block/dirty-bitmap.c +5
-0@@ -555,3 +559,26 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) {return bitmap->autoload; } + +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap, +bool persistent) +{ + bitmap->persistent = persistent;After some thinking, I think this function should be more complex: It should check whether the node the bitmap is attached to actually can handle persistent bitmaps and whether it would actually support storing *this* bitmap.
For instance, a qcow2 node would not support writing overly large bitmaps (limited by
BME_MAX_TABLE_SIZEandBME_MAX_PHYS_SIZE) or bitmaps with overly large granularities (BME_MAX_GRANULARITY_BITS) or bitmaps whose name is already occupied by some bitmap that is already stored in the file but has not been loaded.Checking this here will trivially prevent users from creating such bitmaps and will also preempt detection of such failures during
bdrv_close()when they cannot be handled gracefully.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Max
Good point, but I can't do it exactly as you say, because I call this function from
qcow2_read_bitmaps, for just created bitmap and it should not be checked and of course it's name is occupied..So, I'll add an additional checking function, to call it from
qmp_block_dirty_bitmap_add, if persistent parameter is set to true.
That would work just as well, yes. Thanks!
Max
PATCH v7 10/22 block/dirty-bitmap: add bdrv_dirty_bitmap_next()
In reply to Vladimir Sementsov-Ogievskiy--- block/dirty-bitmap.c | 7 +++++++ include/block/dirty-bitmap.h | 3 +++ 2 files changed, 10 insertions(+)
block/dirty-bitmap.c +7 -0
@@ -582,3 +582,10 @@ bool bdrv_has_persistent_bitmaps(BlockDriverState *bs)return false; } + +BdrvDirtyBitmap *bdrv_dirty_bitmap_next (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap) +{ +return bitmap ==NULL ?QLIST_FIRST (&bs->dirty_bitmaps) : +QLIST_NEXT (bitmap, list); +}
include/block/dirty-bitmap.h +3 -0
@@ -81,6 +81,9 @@ void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap *bitmap, bool persistent);bool bdrv_dirty_bitmap_get_persistance (BdrvDirtyBitmap *bitmap); +BdrvDirtyBitmap *bdrv_dirty_bitmap_next (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap); +bool bdrv_has_persistent_bitmaps (BlockDriverState *bs);#endif
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 10/22] block/dirty-bitmap: add bdrv_dirty_bitmap_next()
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/dirty-bitmap.c | 7 +++++++ include/block/dirty-bitmap.h | 3 +++ 2 files changed, 10 insertions(+)
PATCH v7 11/22 qcow2-bitmap: add qcow2_store_persistent_bitmaps()
In reply to Vladimir Sementsov-OgievskiyRealize block bitmap stroing interface, to allow qcow2 images store persistent bitmaps.
--- block/qcow2-bitmap.c | 241 +++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 2 + block/qcow2.h | 2 + 3 files changed, 245 insertions(+)
block/qcow2-bitmap.c +241 -0
@@ -27,6 +27,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "qemu/cutils.h" #include "block/block_int.h" #include "block/qcow2.h"
@@ -96,6 +97,15 @@ static inline void bitmap_table_to_cpu(uint64_t *bitmap_table, size_t size) } } +static inline void bitmap_table_to_be (uint64_t *bitmap_table,size_t size) +{ +size_t i; + +for (i =0 ; i < size; ++i) { +cpu_to_be64s (&bitmap_table[i]); + } +} +static inline int calc_dir_entry_size (size_t name_size,size_t extra_data_size) { return align_offset(sizeof(Qcow2BitmapDirEntry) +
@@ -564,3 +574,234 @@ out:return ret; } + +/* store_bitmap_data() +* Store bitmap to image, filling bitamp table accordingly. +*/ +static int store_bitmap_data (BlockDriverState *bs,BdrvDirtyBitmap *bitmap, +uint64_t *bitmap_table,uint32_t bitmap_table_size) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint64_t sector, dsc; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +int cl_size = s->cluster_size; +uint8_t *buf =NULL ; +uint32_t tb_size = +size_to_clusters (s, +bdrv_dirty_bitmap_serialization_size (bitmap,0 , bm_size)); + +BdrvDirtyBitmapIter *dbi; + +if (tb_size != bitmap_table_size) { +return -EINVAL ; + } + +memset (bitmap_table,0 , bitmap_table_size *sizeof (bitmap_table[0 ])); + + dbi =bdrv_dirty_iter_new (bitmap,0 ); + buf =g_malloc (cl_size); + dsc =dirty_sectors_in_cluster (s, bitmap); + +while ((sector =bdrv_dirty_iter_next (dbi)) !=-1 ) { +uint64_t cluster = sector / dsc; + sector = cluster * dsc; +uint64_t end =MIN (bm_size, sector + dsc); +uint64_t write_size = +bdrv_dirty_bitmap_serialization_size (bitmap, sector, end - sector); + +int64_t off =qcow2_alloc_clusters (bs, cl_size); +if (off <0 ) { + ret = off; + goto finish; + } + bitmap_table[cluster] = off; + +bdrv_dirty_bitmap_serialize_part (bitmap, buf, sector, end); +if (write_size < cl_size) { +memset (buf + write_size,0 , cl_size - write_size); + } + + ret =bdrv_pwrite (bs->file, off, buf, cl_size); +if (ret <0 ) { + goto finish; + } + +if (end >= bm_size) { +break ; + } + +bdrv_set_dirty_iter (dbi, end); + } + ret =0 ;/* writes */ + +finish: +if (ret <0 ) { +clear_bitmap_table (bs, bitmap_table, bitmap_table_size); + } +g_free (buf); +bdrv_dirty_iter_free (dbi); + +return ret; +} + +/* store_bitmap() +* Store bitmap to qcow2 and set bitmap_table. bitmap_table itself is not +* stored to qcow2. +*/ +static int store_bitmap (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap, +Qcow2BitmapDirEntry *entry) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +const char *bm_name =bdrv_dirty_bitmap_name (bitmap); + +uint64_t *tb; +int64_t tb_offset; +uint32_t tb_size = +size_to_clusters (s, +bdrv_dirty_bitmap_serialization_size (bitmap,0 , bm_size)); + + tb =g_try_new (uint64_t, tb_size); +if (tb ==NULL ) { +return -ENOMEM ; + } + + ret =store_bitmap_data (bs, bitmap, tb, tb_size); +if (ret <0 ) { +g_free (tb); +return ret; + } + + tb_offset =qcow2_alloc_clusters (bs, tb_size *sizeof (tb[0 ])); +if (tb_offset <0 ) { + ret = tb_offset; + goto fail; + } + +bitmap_table_to_be (tb, tb_size); + ret =bdrv_pwrite (bs->file, tb_offset, tb, tb_size *sizeof (tb[0 ])); +if (ret <0 ) { + goto fail; + } + +g_free (tb); + + entry->bitmap_table_offset = tb_offset; + entry->bitmap_table_size = tb_size; + entry->flags =bdrv_dirty_bitmap_granularity (bitmap) ?BME_FLAG_AUTO :0 ; + entry->type =BT_DIRTY_TRACKING_BITMAP ; + entry->granularity_bits =ctz32 (bdrv_dirty_bitmap_granularity (bitmap)); + entry->name_size =strlen (bm_name); + entry->extra_data_size =0 ; +memcpy (entry +1 , bm_name, entry->name_size); + +return 0 ; + +fail: +clear_bitmap_table (bs, tb, tb_size); + +if (tb_offset >0 ) { +qcow2_free_clusters (bs, tb_offset, tb_size,QCOW2_DISCARD_ALWAYS ); + } + +g_free (tb); + +return ret; +} + +static Qcow2BitmapDirEntry *find_bitmap_by_name (uint8_t *bitmap_directory, +size_t size,const char *name) +{ +Qcow2BitmapDirEntry *e; + +for_each_bitmap_dir_entry (e, bitmap_directory, size) { +if (strncmp ((char *)(e +1 ), name, e->name_size) ==0 ) { +return e; + } + } + +return NULL ; +} + +void qcow2_store_persistent_bitmaps (BlockDriverState *bs,Error **errp) +{ +BdrvDirtyBitmap *bm; +BDRVQcow2State *s = bs->opaque; +uint32_t new_nb_bitmaps = s->nb_bitmaps; +uint64_t new_dir_size = s->bitmap_directory_size; +uint8_t *dir =NULL , *new_dir =NULL ; +int ret; +Qcow2BitmapDirEntry *new_pos; + +if (s->nb_bitmaps >0 ) { + dir =directory_read (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); +if (dir ==NULL ) { + goto out; + } + } + +for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm !=NULL ; + bm =bdrv_dirty_bitmap_next (bs, bm)) { +const char *name =bdrv_dirty_bitmap_name (bm); + +if (!bdrv_dirty_bitmap_get_persistance (bm)) { +continue ; + } + +if (s->nb_bitmaps >0 && +find_bitmap_by_name (dir, s->bitmap_directory_size, name)) { +error_setg (errp, +"Can't store bitmap '%s' to '%s', as it already exists" , + name,bdrv_get_device_or_node_name (bs)); + goto out; + } + + new_nb_bitmaps++; + new_dir_size +=calc_dir_entry_size (strlen (name),0 ); + } + +if (s->nb_bitmaps == new_nb_bitmaps) { +/* No new bitmaps - nothing to do */ + goto out; + } + + new_dir =g_try_malloc0 (new_dir_size); +if (new_dir ==NULL ) { +error_setg (errp,"Can't allocate space for bitmap directory." ); + goto out; + } + +memcpy (new_dir, dir, s->bitmap_directory_size); + new_pos = (Qcow2BitmapDirEntry *)(new_dir + s->bitmap_directory_size); + +for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm !=NULL ; + bm =bdrv_dirty_bitmap_next (bs, bm)) { +if (!bdrv_dirty_bitmap_get_persistance (bm)) { +continue ; + } + + ret =store_bitmap (bs, bm, new_pos); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Can't store bitmap '%s' to '%s'" , +bdrv_dirty_bitmap_name (bm), +bdrv_get_device_or_node_name (bs)); + goto out; + } + new_pos =next_dir_entry (new_pos); + } + + ret =directory_update (bs, new_dir, new_dir_size, new_nb_bitmaps); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Can't update bitmap directory in '%s'" , +bdrv_get_device_or_node_name (bs)); + goto out; + } + +out: +g_free (new_dir); +g_free (dir); +}
block/qcow2.c +2 -0
@@ -3493,6 +3493,8 @@ BlockDriver bdrv_qcow2 = { .bdrv_detach_aio_context = qcow2_detach_aio_context, .bdrv_attach_aio_context = qcow2_attach_aio_context, + + .bdrv_store_persistent_bitmaps = qcow2_store_persistent_bitmaps, };static void bdrv_qcow2_init (void )
block/qcow2.h +2 -0
@@ -627,4 +627,6 @@ int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset, void **table);void qcow2_cache_put (BlockDriverState *bs,Qcow2Cache *c,void **table); +void qcow2_store_persistent_bitmaps (BlockDriverState *bs,Error **errp); +#endif
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 11/22] qcow2-bitmap: add qcow2_store_persistent_bitmaps()
Quoted 39 lines · Vladimir Sementsov-Ogievskiy
Realize block bitmap stroing interface, to allow qcow2 images store persistent bitmaps.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 241 +++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 2 + block/qcow2.h | 2 + 3 files changed, 245 insertions(+)block/qcow2-bitmap.c +10
-0@@ -27,6 +27,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "qemu/cutils.h" #include "block/block_int.h" #include "block/qcow2.h" @@ -96,6 +97,15 @@ static inline void bitmap_table_to_cpu(uint64_t *bitmap_table, size_t size) } } +static inline void bitmap_table_to_be (uint64_t *bitmap_table,size_t size) +{ +size_t i; + +for (i =0 ; i < size; ++i) { +cpu_to_be64s (&bitmap_table[i]); + } +} +static inline int calc_dir_entry_size (size_t name_size,size_t extra_data_size) { return align_offset(sizeof(Qcow2BitmapDirEntry) +
block/qcow2-bitmap.c out: · line 579
Quoted 6 lines
block/qcow2-bitmap.c +3
-0@@ -564,3 +574,234 @@ out:return ret; } + +/* store_bitmap_data() +* Store bitmap to image, filling bitamp table accordingly.
s/bitamp/bitmap/
block/qcow2-bitmap.c out: · line 588
Quoted 9 lines
block/qcow2-bitmap.c +9
-0+*/ +static int store_bitmap_data (BlockDriverState *bs,BdrvDirtyBitmap *bitmap, +uint64_t *bitmap_table,uint32_t bitmap_table_size) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint64_t sector, dsc; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +int cl_size = s->cluster_size;
Once more, I don't think this variable is necessary, and I feel like it makes reading the code more difficult for no gain.
block/qcow2-bitmap.c out: · line 592
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+uint8_t *buf =NULL ; +uint32_t tb_size = +size_to_clusters (s, +bdrv_dirty_bitmap_serialization_size (bitmap,0 , bm_size));
Should be uint64_t or you might accidentally get a truncation here. I'm not sure how you would get such huge bitmaps but it's trivial to use uint64_t.
block/qcow2-bitmap.c out: · line 600
Quoted 8 lines
block/qcow2-bitmap.c +8
-0+ +BdrvDirtyBitmapIter *dbi; + +if (tb_size != bitmap_table_size) { +return -EINVAL ; + } + +memset (bitmap_table,0 , bitmap_table_size *sizeof (bitmap_table[0 ]));
On 32 bit machines, this multiplication can overflow. There should at least be an assertion to prevent this. Other than that, of course, there needs to be some place which limits bitmap_table_size to some sane value and emits a real error if it exceeds that value.
block/qcow2-bitmap.c out: · line 606
Quoted 6 lines
block/qcow2-bitmap.c +6
-0+ + dbi =bdrv_dirty_iter_new (bitmap,0 ); + buf =g_malloc (cl_size); + dsc =dirty_sectors_in_cluster (s, bitmap); + +while ((sector =bdrv_dirty_iter_next (dbi)) !=-1 ) {
sector should be int64_t instead of uint64_t, then.
block/qcow2-bitmap.c out: · line 608
Quoted 2 lines
block/qcow2-bitmap.c +2
-0+uint64_t cluster = sector / dsc; + sector = cluster * dsc;
Our coding style does not allow interleaving declarations and non-declarations.
block/qcow2-bitmap.c out: · line 620
Quoted 12 lines
block/qcow2-bitmap.c +12
-0+uint64_t end =MIN (bm_size, sector + dsc); +uint64_t write_size = +bdrv_dirty_bitmap_serialization_size (bitmap, sector, end - sector); + +int64_t off =qcow2_alloc_clusters (bs, cl_size); +if (off <0 ) { + ret = off; + goto finish; + } + bitmap_table[cluster] = off; + +bdrv_dirty_bitmap_serialize_part (bitmap, buf, sector, end);
s/end/end - sector/?
block/qcow2-bitmap.c out: · line 624
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+if (write_size < cl_size) { +memset (buf + write_size,0 , cl_size - write_size); + } +
I guess there should be a metadata overlap check here.
block/qcow2-bitmap.c out: · line 636
Quoted 12 lines
block/qcow2-bitmap.c +12
-0+ ret =bdrv_pwrite (bs->file, off, buf, cl_size); +if (ret <0 ) { + goto finish; + } + +if (end >= bm_size) { +break ; + } + +bdrv_set_dirty_iter (dbi, end); + } + ret =0 ;/* writes */
What is that comment supposed to mean?
block/qcow2-bitmap.c out: · line 645
Quoted 9 lines
block/qcow2-bitmap.c +9
-0+ +finish: +if (ret <0 ) { +clear_bitmap_table (bs, bitmap_table, bitmap_table_size); + } +g_free (buf); +bdrv_dirty_iter_free (dbi); + +return ret;
In case you decide to keep BME_MAX_PHYS_SIZE, this function should check somewhere that the physical size of the bitmap does not exceed that value.
block/qcow2-bitmap.c out: · line 650
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+} + +/* store_bitmap() +* Store bitmap to qcow2 and set bitmap_table. bitmap_table itself is not +* stored to qcow2.
First of all, there is no parameter called "bitmap_table", and second, yes, the bitmap table is written to the qcow2 file.
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+*/ +static int store_bitmap (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap, +Qcow2BitmapDirEntry *entry) +{
block/qcow2-bitmap.c out: · line 665
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+int ret; +BDRVQcow2State *s = bs->opaque; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +const char *bm_name =bdrv_dirty_bitmap_name (bitmap); + +uint64_t *tb; +int64_t tb_offset; +uint32_t tb_size = +size_to_clusters (s, +bdrv_dirty_bitmap_serialization_size (bitmap,0 , bm_size));
As above, this variable should be of type uint64_t.
Also, you have to check that it does not exceed BME_MAX_TABLE_SIZE.
block/qcow2-bitmap.c out: · line 678
Quoted 13 lines
block/qcow2-bitmap.c +13
-0+ + tb =g_try_new (uint64_t, tb_size); +if (tb ==NULL ) { +return -ENOMEM ; + } + + ret =store_bitmap_data (bs, bitmap, tb, tb_size); +if (ret <0 ) { +g_free (tb); +return ret; + } + + tb_offset =qcow2_alloc_clusters (bs, tb_size *sizeof (tb[0 ]));
If you don't limit tb_size, then this multiplication can overflow on 32 bit machines.
block/qcow2-bitmap.c out: · line 683
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+if (tb_offset <0 ) { + ret = tb_offset; + goto fail; + } +
There should be a metadata overlap check here.
block/qcow2-bitmap.c out: · line 694
Quoted 11 lines
block/qcow2-bitmap.c +11
-0+bitmap_table_to_be (tb, tb_size); + ret =bdrv_pwrite (bs->file, tb_offset, tb, tb_size *sizeof (tb[0 ])); +if (ret <0 ) { + goto fail; + } + +g_free (tb); + + entry->bitmap_table_offset = tb_offset; + entry->bitmap_table_size = tb_size; + entry->flags =bdrv_dirty_bitmap_granularity (bitmap) ?BME_FLAG_AUTO :0 ;
s/granularity/get_autoload/
block/qcow2-bitmap.c out: · line 696
Quoted 2 lines
block/qcow2-bitmap.c +2
-0+ entry->type =BT_DIRTY_TRACKING_BITMAP ; + entry->granularity_bits =ctz32 (bdrv_dirty_bitmap_granularity (bitmap));
You should probably check somewhere that the resulting value for entry->granularity_bits is in the BME_{MIN,MAX}_GRANULARITY_BITS range.
block/qcow2-bitmap.c out: · line 697
Quoted 1 line
block/qcow2-bitmap.c +1
-0+ entry->name_size =strlen (bm_name);
And that this length does not exceed BME_MAX_NAME_SIZE.
block/qcow2-bitmap.c out: · line 707
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ entry->extra_data_size =0 ; +memcpy (entry +1 , bm_name, entry->name_size); + +return 0 ; + +fail: +clear_bitmap_table (bs, tb, tb_size); + +if (tb_offset >0 ) { +qcow2_free_clusters (bs, tb_offset, tb_size,QCOW2_DISCARD_ALWAYS );
As before, I'd vote for QCOW2_DISCARD_OTHER.
Quoted 89 lines
block/qcow2-bitmap.c +89
-0+ } + +g_free (tb); + +return ret; +} + +static Qcow2BitmapDirEntry *find_bitmap_by_name (uint8_t *bitmap_directory, +size_t size,const char *name) +{ +Qcow2BitmapDirEntry *e; + +for_each_bitmap_dir_entry (e, bitmap_directory, size) { +if (strncmp ((char *)(e +1 ), name, e->name_size) ==0 ) { +return e; + } + } + +return NULL ; +} + +void qcow2_store_persistent_bitmaps (BlockDriverState *bs,Error **errp) +{ +BdrvDirtyBitmap *bm; +BDRVQcow2State *s = bs->opaque; +uint32_t new_nb_bitmaps = s->nb_bitmaps; +uint64_t new_dir_size = s->bitmap_directory_size; +uint8_t *dir =NULL , *new_dir =NULL ; +int ret; +Qcow2BitmapDirEntry *new_pos; + +if (s->nb_bitmaps >0 ) { + dir =directory_read (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); +if (dir ==NULL ) { + goto out; + } + } + +for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm !=NULL ; + bm =bdrv_dirty_bitmap_next (bs, bm)) { +const char *name =bdrv_dirty_bitmap_name (bm); + +if (!bdrv_dirty_bitmap_get_persistance (bm)) { +continue ; + } + +if (s->nb_bitmaps >0 && +find_bitmap_by_name (dir, s->bitmap_directory_size, name)) { +error_setg (errp, +"Can't store bitmap '%s' to '%s', as it already exists" , + name,bdrv_get_device_or_node_name (bs)); + goto out; + } + + new_nb_bitmaps++; + new_dir_size +=calc_dir_entry_size (strlen (name),0 ); + } + +if (s->nb_bitmaps == new_nb_bitmaps) { +/* No new bitmaps - nothing to do */ + goto out; + } + + new_dir =g_try_malloc0 (new_dir_size); +if (new_dir ==NULL ) { +error_setg (errp,"Can't allocate space for bitmap directory." ); + goto out; + } + +memcpy (new_dir, dir, s->bitmap_directory_size); + new_pos = (Qcow2BitmapDirEntry *)(new_dir + s->bitmap_directory_size); + +for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm !=NULL ; + bm =bdrv_dirty_bitmap_next (bs, bm)) { +if (!bdrv_dirty_bitmap_get_persistance (bm)) { +continue ; + } + + ret =store_bitmap (bs, bm, new_pos); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Can't store bitmap '%s' to '%s'" , +bdrv_dirty_bitmap_name (bm), +bdrv_get_device_or_node_name (bs)); + goto out; + } + new_pos =next_dir_entry (new_pos); + } +
block/qcow2-bitmap.c out: · line 806
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ ret =directory_update (bs, new_dir, new_dir_size, new_nb_bitmaps); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Can't update bitmap directory in '%s'" , +bdrv_get_device_or_node_name (bs)); + goto out; + } + +out: +g_free (new_dir); +g_free (dir);
This error path leaks all the bitmaps that have been written successfully (if any). I guess this is more or less fine if directory_update() failed (because you can't really tell the state of the image header after directory_update(), so better be safe) but it's not so fine if just some store_bitmap() failed.
Max
Quoted 26 lines
block/qcow2-bitmap.c +1
-0+}block/qcow2.c +2
-0@@ -3493,6 +3493,8 @@ BlockDriver bdrv_qcow2 = { .bdrv_detach_aio_context = qcow2_detach_aio_context, .bdrv_attach_aio_context = qcow2_attach_aio_context, + + .bdrv_store_persistent_bitmaps = qcow2_store_persistent_bitmaps, };static void bdrv_qcow2_init (void )block/qcow2.h +2
-0@@ -627,4 +627,6 @@ int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset, void **table);void qcow2_cache_put (BlockDriverState *bs,Qcow2Cache *c,void **table); +void qcow2_store_persistent_bitmaps (BlockDriverState *bs,Error **errp); +#endif
Re: [Qemu-devel] [PATCH 11/22] qcow2-bitmap: add qcow2_store_persistent_bitmaps()
Quoted 2 lines · Max Reitz
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Realize block bitmap stroing interface, to allow qcow2 images store
[snip]
Quoted 5 lines
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+uint64_t end =MIN (bm_size, sector + dsc); +uint64_t write_size = +bdrv_dirty_bitmap_serialization_size (bitmap, sector, end - sector); + +int64_t off =qcow2_alloc_clusters (bs, cl_size);
Quoted 8 lines
block/qcow2-bitmap.c out: · line 620
Quoted 7 lines
block/qcow2-bitmap.c +7
-0+if (off <0 ) { + ret = off; + goto finish; + } + bitmap_table[cluster] = off; + +bdrv_dirty_bitmap_serialize_part (bitmap, buf, sector, end);s/end/end - sector/?
o_0 terrible mistake, thank you.
Quoted 6 lines
block/qcow2-bitmap.c out: · line 624
Quoted 5 lines
block/qcow2-bitmap.c +4
-0+if (write_size < cl_size) { +memset (buf + write_size,0 , cl_size - write_size); + } +I guess there should be a metadata overlap check here.
What is the general rule of checking it? Should I check it before all my extension related writes?
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 11/22] qcow2-bitmap: add qcow2_store_persistent_bitmaps()
Quoted 24 lines · Vladimir Sementsov-Ogievskiy
Quoted 2 lines · Max Reitz
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Realize block bitmap stroing interface, to allow qcow2 images store
[snip]
Quoted 6 lines
Quoted 6 lines
block/qcow2-bitmap.c +5
-1+uint64_t end =MIN (bm_size, sector + dsc); +uint64_t write_size = +bdrv_dirty_bitmap_serialization_size (bitmap, sector, end - sector); + + int64_t off = qcow2_alloc_clusters(bs, cl_size);Quoted 8 lines
block/qcow2-bitmap.c out: · line 620
Quoted 7 lines
block/qcow2-bitmap.c +7
-0+if (off <0 ) { + ret = off; + goto finish; + } + bitmap_table[cluster] = off; + +bdrv_dirty_bitmap_serialize_part (bitmap, buf, sector, end);s/end/end - sector/?
o_0terrible mistake, thank you.Quoted 1 line
Quoted 8 lines
Quoted 5 lines
block/qcow2-bitmap.c out: · line 624
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+if (write_size < cl_size) { +memset (buf + write_size,0 , cl_size - write_size); + } +I guess there should be a metadata overlap check here.
What is the general rule of checking it? Should I check it before all my extension related writes?
The general rule is supposed to be "One check before every write to bs->file".
Max
Re: [Qemu-devel] [PATCH 11/22] qcow2-bitmap: add qcow2_store_persistent_bitmaps()
In reply to Max ReitzQuoted 7 lines · Max Reitz
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Realize block bitmap stroing interface, to allow qcow2 images store persistent bitmaps.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 241 +++++++++++++++++++++++++++++++++++++++++++++++++++
[...]
Quoted 16 lines
block/qcow2-bitmap.c out: · line 636
Quoted 13 lines
block/qcow2-bitmap.c +12
-0+ ret =bdrv_pwrite (bs->file, off, buf, cl_size); +if (ret <0 ) { + goto finish; + } + +if (end >= bm_size) { +break ; + } + +bdrv_set_dirty_iter (dbi, end); + } + ret =0 ;/* writes */ What is that comment supposed to mean?
Now I think I can drop this assignment, as bdrv_aligned_preadv have 'return ret < 0 ? ret : 0;' in the end... Am I right? Can bdrv_pwrite and friends return positive value on success?
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 11/22] qcow2-bitmap: add qcow2_store_persistent_bitmaps()
Quoted 29 lines · Vladimir Sementsov-Ogievskiy
Quoted 8 lines · Max Reitz
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
Realize block bitmap stroing interface, to allow qcow2 images store persistent bitmaps.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 241+++++++++++++++++++++++++++++++++++++++++++++++++++
[...]
Quoted 16 lines
block/qcow2-bitmap.c out: · line 636
Quoted 13 lines
block/qcow2-bitmap.c +12
-0+ ret =bdrv_pwrite (bs->file, off, buf, cl_size); +if (ret <0 ) { + goto finish; + } + +if (end >= bm_size) { +break ; + } + +bdrv_set_dirty_iter (dbi, end); + } + ret =0 ;/* writes */ What is that comment supposed to mean?
Quoted 3 lines
Now I think I can drop this assignment, as
bdrv_aligned_preadvhave 'return ret < 0 ? ret : 0;' in the end... Am I right? Canbdrv_pwriteand friends return positive value on success?
I think so, but if they do return positive value they're supposed to be equal to the number of bytes written. In practice, though, any non-negative return value is treated the same (as success).
I was just wondering about the comment, though, not about the assignment itself.
Max
Re: [Qemu-devel] [PATCH 11/22] qcow2-bitmap: add qcow2_store_persistent_bitmaps()
Quoted 7 lines · Max Reitz
Quoted 3 lines
Now I think I can drop this assignment, as
bdrv_aligned_preadvhave 'return ret < 0 ? ret : 0;' in the end... Am I right? Canbdrv_pwriteand friends return positive value on success?I think so, but if they do return positive value they're supposed to be equal to the number of bytes written. In practice, though, any non-negative return value is treated the same (as success).
At one point in my conversion to byte access, I ran into test failures when trying to return number of bytes written; so commit 1a62d0a hard-codes a return of 0 on success instead. It will be a long and painful audit to find which callers expect which semantics, which I did not want to do.
Quoted 6 lines
I was just wondering about the comment, though, not about the assignment itself.
Max
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 11/22] qcow2-bitmap: add qcow2_store_persistent_bitmaps()
In reply to Vladimir Sementsov-OgievskiyQuoted 111 lines · Vladimir Sementsov-Ogievskiy
Realize block bitmap stroing interface, to allow qcow2 images store persistent bitmaps.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 241 +++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 2 + block/qcow2.h | 2 + 3 files changed, 245 insertions(+)block/qcow2-bitmap.c +60
-0@@ -27,6 +27,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "qemu/cutils.h" #include "block/block_int.h" #include "block/qcow2.h" @@ -96,6 +97,15 @@ static inline void bitmap_table_to_cpu(uint64_t *bitmap_table, size_t size) } } +static inline void bitmap_table_to_be (uint64_t *bitmap_table,size_t size) +{ +size_t i; + +for (i =0 ; i < size; ++i) { +cpu_to_be64s (&bitmap_table[i]); + } +} +static inline int calc_dir_entry_size (size_t name_size,size_t extra_data_size) { return align_offset(sizeof(Qcow2BitmapDirEntry) +@@ -564,3 +574,234 @@ out:return ret; } + +/* store_bitmap_data() +* Store bitmap to image, filling bitamp table accordingly. +*/ +static int store_bitmap_data (BlockDriverState *bs,BdrvDirtyBitmap *bitmap, +uint64_t *bitmap_table,uint32_t bitmap_table_size) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint64_t sector, dsc; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +int cl_size = s->cluster_size; +uint8_t *buf =NULL ; +uint32_t tb_size = +size_to_clusters (s, +bdrv_dirty_bitmap_serialization_size (bitmap,0 , bm_size)); + +BdrvDirtyBitmapIter *dbi; + +if (tb_size != bitmap_table_size) { +return -EINVAL ; + } + +memset (bitmap_table,0 , bitmap_table_size *sizeof (bitmap_table[0 ])); + + dbi =bdrv_dirty_iter_new (bitmap,0 ); + buf =g_malloc (cl_size); + dsc =dirty_sectors_in_cluster (s, bitmap); + +while ((sector =bdrv_dirty_iter_next (dbi)) !=-1 ) { +uint64_t cluster = sector / dsc; + sector = cluster * dsc; +uint64_t end =MIN (bm_size, sector + dsc); +uint64_t write_size = +bdrv_dirty_bitmap_serialization_size (bitmap, sector, end - sector); + +int64_t off =qcow2_alloc_clusters (bs, cl_size); +if (off <0 ) { + ret = off; + goto finish; + } + bitmap_table[cluster] = off; + +bdrv_dirty_bitmap_serialize_part (bitmap, buf, sector, end); +if (write_size < cl_size) { +memset (buf + write_size,0 , cl_size - write_size); + } + + ret =bdrv_pwrite (bs->file, off, buf, cl_size); +if (ret <0 ) {
block/qcow2-bitmap.c out: · line 636
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ goto finish; + } + +if (end >= bm_size) { +break ; + } + +bdrv_set_dirty_iter (dbi, end); + } + ret =0 ;/* writes */
What is that comment supposed to mean?
block/qcow2-bitmap.c out: · line 645
Quoted 9 lines
block/qcow2-bitmap.c +9
-0+ +finish: +if (ret <0 ) { +clear_bitmap_table (bs, bitmap_table, bitmap_table_size); + } +g_free (buf); +bdrv_dirty_iter_free (dbi); + +return ret;
In case you decide to keep BME_MAX_PHYS_SIZE, this function should check somewhere that the physical size of the bitmap does not exceed that value.
block/qcow2-bitmap.c out: · line 650
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+} + +/* store_bitmap() +* Store bitmap to qcow2 and set bitmap_table. bitmap_table itself is not +* stored to qcow2.
First of all, there is no parameter called "bitmap_table", and second, yes, the bitmap table is written to the qcow2 file.
Quoted 5 lines
+ */
block/qcow2-bitmap.c +4
-0+static int store_bitmap (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap, +Qcow2BitmapDirEntry *entry) +{
block/qcow2-bitmap.c out: · line 665
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+int ret; +BDRVQcow2State *s = bs->opaque; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +const char *bm_name =bdrv_dirty_bitmap_name (bitmap); + +uint64_t *tb; +int64_t tb_offset; +uint32_t tb_size = +size_to_clusters (s, +bdrv_dirty_bitmap_serialization_size (bitmap,0 , bm_size));
As above, this variable should be of type uint64_t.
Also, you have to check that it does not exceed BME_MAX_TABLE_SIZE.
block/qcow2-bitmap.c out: · line 678
Quoted 13 lines
block/qcow2-bitmap.c +13
-0+ + tb =g_try_new (uint64_t, tb_size); +if (tb ==NULL ) { +return -ENOMEM ; + } + + ret =store_bitmap_data (bs, bitmap, tb, tb_size); +if (ret <0 ) { +g_free (tb); +return ret; + } + + tb_offset =qcow2_alloc_clusters (bs, tb_size *sizeof (tb[0 ]));
If you don't limit tb_size, then this multiplication can overflow on 32 bit machines.
block/qcow2-bitmap.c out: · line 683
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+if (tb_offset <0 ) { + ret = tb_offset; + goto fail; + } +
There should be a metadata overlap check here.
block/qcow2-bitmap.c out: · line 694
Quoted 11 lines
block/qcow2-bitmap.c +11
-0+bitmap_table_to_be (tb, tb_size); + ret =bdrv_pwrite (bs->file, tb_offset, tb, tb_size *sizeof (tb[0 ])); +if (ret <0 ) { + goto fail; + } + +g_free (tb); + + entry->bitmap_table_offset = tb_offset; + entry->bitmap_table_size = tb_size; + entry->flags =bdrv_dirty_bitmap_granularity (bitmap) ?BME_FLAG_AUTO :0 ;
s/granularity/get_autoload/
block/qcow2-bitmap.c out: · line 696
Quoted 2 lines
block/qcow2-bitmap.c +2
-0+ entry->type =BT_DIRTY_TRACKING_BITMAP ; + entry->granularity_bits =ctz32 (bdrv_dirty_bitmap_granularity (bitmap));
You should probably check somewhere that the resulting value for entry->granularity_bits is in the BME_{MIN,MAX}_GRANULARITY_BITS range.
Quoted 1 line
+
entry->name_size=strlen(bm_name);
And that this length does not exceed BME_MAX_NAME_SIZE.
block/qcow2-bitmap.c out: · line 707
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ entry->extra_data_size =0 ; +memcpy (entry +1 , bm_name, entry->name_size); + +return 0 ; + +fail: +clear_bitmap_table (bs, tb, tb_size); + +if (tb_offset >0 ) { +qcow2_free_clusters (bs, tb_offset, tb_size,QCOW2_DISCARD_ALWAYS );
As before, I'd vote for QCOW2_DISCARD_OTHER.
Quoted 89 lines
block/qcow2-bitmap.c +89
-0+ } + +g_free (tb); + +return ret; +} + +static Qcow2BitmapDirEntry *find_bitmap_by_name (uint8_t *bitmap_directory, +size_t size,const char *name) +{ +Qcow2BitmapDirEntry *e; + +for_each_bitmap_dir_entry (e, bitmap_directory, size) { +if (strncmp ((char *)(e +1 ), name, e->name_size) ==0 ) { +return e; + } + } + +return NULL ; +} + +void qcow2_store_persistent_bitmaps (BlockDriverState *bs,Error **errp) +{ +BdrvDirtyBitmap *bm; +BDRVQcow2State *s = bs->opaque; +uint32_t new_nb_bitmaps = s->nb_bitmaps; +uint64_t new_dir_size = s->bitmap_directory_size; +uint8_t *dir =NULL , *new_dir =NULL ; +int ret; +Qcow2BitmapDirEntry *new_pos; + +if (s->nb_bitmaps >0 ) { + dir =directory_read (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); +if (dir ==NULL ) { + goto out; + } + } + +for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm !=NULL ; + bm =bdrv_dirty_bitmap_next (bs, bm)) { +const char *name =bdrv_dirty_bitmap_name (bm); + +if (!bdrv_dirty_bitmap_get_persistance (bm)) { +continue ; + } + +if (s->nb_bitmaps >0 && +find_bitmap_by_name (dir, s->bitmap_directory_size, name)) { +error_setg (errp, +"Can't store bitmap '%s' to '%s', as it already exists" , + name,bdrv_get_device_or_node_name (bs)); + goto out; + } + + new_nb_bitmaps++; + new_dir_size +=calc_dir_entry_size (strlen (name),0 ); + } + +if (s->nb_bitmaps == new_nb_bitmaps) { +/* No new bitmaps - nothing to do */ + goto out; + } + + new_dir =g_try_malloc0 (new_dir_size); +if (new_dir ==NULL ) { +error_setg (errp,"Can't allocate space for bitmap directory." ); + goto out; + } + +memcpy (new_dir, dir, s->bitmap_directory_size); + new_pos = (Qcow2BitmapDirEntry *)(new_dir + s->bitmap_directory_size); + +for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm !=NULL ; + bm =bdrv_dirty_bitmap_next (bs, bm)) { +if (!bdrv_dirty_bitmap_get_persistance (bm)) { +continue ; + } + + ret =store_bitmap (bs, bm, new_pos); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Can't store bitmap '%s' to '%s'" , +bdrv_dirty_bitmap_name (bm), +bdrv_get_device_or_node_name (bs)); + goto out; + } + new_pos =next_dir_entry (new_pos); + } +
block/qcow2-bitmap.c out: · line 806
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ ret =directory_update (bs, new_dir, new_dir_size, new_nb_bitmaps); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Can't update bitmap directory in '%s'" , +bdrv_get_device_or_node_name (bs)); + goto out; + } + +out: +g_free (new_dir); +g_free (dir);
This error path leaks all the bitmaps that have been written successfully (if any). I guess this is more or less fine if directory_update() failed (because you can't really tell the state of the image header after directory_update(), so better be safe) but it's not so fine if just some store_bitmap() failed.
Max
Quoted 26 lines
block/qcow2-bitmap.c +1
-0+}block/qcow2.c +2
-0@@ -3493,6 +3493,8 @@ BlockDriver bdrv_qcow2 = { .bdrv_detach_aio_context = qcow2_detach_aio_context, .bdrv_attach_aio_context = qcow2_attach_aio_context, + + .bdrv_store_persistent_bitmaps = qcow2_store_persistent_bitmaps, };static void bdrv_qcow2_init (void )block/qcow2.h +2
-0@@ -627,4 +627,6 @@ int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset, void **table);void qcow2_cache_put (BlockDriverState *bs,Qcow2Cache *c,void **table); +void qcow2_store_persistent_bitmaps (BlockDriverState *bs,Error **errp); +#endif
[Qemu-devel] DROP THIS Re: [PATCH 11/22] qcow2-bitmap: add qcow2_store_persistent_bitmaps()
Sorry, this was an accidental reply.
Quoted 391 lines · Vladimir Sementsov-Ogievskiy
Quoted 120 lines · Vladimir Sementsov-Ogievskiy
Realize block bitmap stroing interface, to allow qcow2 images store persistent bitmaps.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 241 +++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 2 + block/qcow2.h | 2 + 3 files changed, 245 insertions(+)block/qcow2-bitmap.c +10
-0@@ -27,6 +27,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "qemu/cutils.h" #include "block/block_int.h" #include "block/qcow2.h" @@ -96,6 +97,15 @@ static inline void bitmap_table_to_cpu(uint64_t *bitmap_table, size_t size) } } +static inline void bitmap_table_to_be (uint64_t *bitmap_table,size_t size) +{ +size_t i; + +for (i =0 ; i < size; ++i) { +cpu_to_be64s (&bitmap_table[i]); + } +} + static inline int calc_dir_entry_size(size_t name_size, size_t
extra_data_size)
{
return align_offset(sizeof(Qcow2BitmapDirEntry) +block/qcow2-bitmap.c +5
-0@@ -564,3 +574,234 @@ out:return ret; } + +/* store_bitmap_data() +* Store bitmap to image, filling bitamp table accordingly. +*/ +static int store_bitmap_data(BlockDriverState *bs, BdrvDirtyBitmap*bitmap,
+uint64_t*bitmap_table,uint32_tbitmap_table_size)block/qcow2-bitmap.c +10
-0+{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint64_t sector, dsc; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +int cl_size = s->cluster_size; +uint8_t *buf =NULL ; +uint32_t tb_size = + size_to_clusters(s, + bdrv_dirty_bitmap_serialization_size(bitmap, 0,
bm_size));block/qcow2-bitmap.c +8
-0+ +BdrvDirtyBitmapIter *dbi; + +if (tb_size != bitmap_table_size) { +return -EINVAL ; + } + + memset(bitmap_table, 0, bitmap_table_size *
sizeof(bitmap_table[0]));block/qcow2-bitmap.c +26
-1+ + dbi =bdrv_dirty_iter_new (bitmap,0 ); + buf =g_malloc (cl_size); + dsc =dirty_sectors_in_cluster (s, bitmap); + +while ((sector =bdrv_dirty_iter_next (dbi)) !=-1 ) { +uint64_t cluster = sector / dsc; + sector = cluster * dsc; +uint64_t end =MIN (bm_size, sector + dsc); +uint64_t write_size = +bdrv_dirty_bitmap_serialization_size (bitmap, sector, end - sector); + + int64_t off = qcow2_alloc_clusters(bs, cl_size); +if (off <0 ) { + ret = off; + goto finish; + } + bitmap_table[cluster] = off; + +bdrv_dirty_bitmap_serialize_part (bitmap, buf, sector, end); +if (write_size < cl_size) { +memset (buf + write_size,0 , cl_size - write_size); + } + + ret =bdrv_pwrite (bs->file, off, buf, cl_size); +if (ret <0 ) {block/qcow2-bitmap.c out: · line 636
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ goto finish; + } + +if (end >= bm_size) { +break ; + } + +bdrv_set_dirty_iter (dbi, end); + } + ret =0 ;/* writes */ What is that comment supposed to mean?
block/qcow2-bitmap.c out: · line 645
Quoted 9 lines
block/qcow2-bitmap.c +9
-0+ +finish: +if (ret <0 ) { +clear_bitmap_table (bs, bitmap_table, bitmap_table_size); + } +g_free (buf); +bdrv_dirty_iter_free (dbi); + +return ret;In case you decide to keep
BME_MAX_PHYS_SIZE, this function should check somewhere that the physical size of the bitmap does not exceed that value.Quoted 6 lines
block/qcow2-bitmap.c +4
-0+} + +/* store_bitmap() +* Store bitmap to qcow2 and set bitmap_table. bitmap_table itself is not
+ * stored to qcow2.First of all, there is no parameter called "
bitmap_table", and second, yes, the bitmap table is written to the qcow2 file.Quoted 15 lines
+ */
block/qcow2-bitmap.c +14
-0+static int store_bitmap (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap, +Qcow2BitmapDirEntry *entry) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +const char *bm_name =bdrv_dirty_bitmap_name (bitmap); + +uint64_t *tb; +int64_t tb_offset; +uint32_t tb_size = + size_to_clusters(s, + bdrv_dirty_bitmap_serialization_size(bitmap, 0,Quoted 1 line
bm_size));As above, this variable should be of type
uint64_t.Also, you have to check that it does not exceed
BME_MAX_TABLE_SIZE.block/qcow2-bitmap.c out: · line 678
Quoted 13 lines
block/qcow2-bitmap.c +13
-0+ + tb =g_try_new (uint64_t, tb_size); +if (tb ==NULL ) { +return -ENOMEM ; + } + + ret =store_bitmap_data (bs, bitmap, tb, tb_size); +if (ret <0 ) { +g_free (tb); +return ret; + } + + tb_offset =qcow2_alloc_clusters (bs, tb_size *sizeof (tb[0 ]));If you don't limit
tb_size, then this multiplication can overflow on 32 bit machines.block/qcow2-bitmap.c out: · line 683
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+if (tb_offset <0 ) { + ret = tb_offset; + goto fail; + } +There should be a metadata overlap check here.
Quoted 12 lines
block/qcow2-bitmap.c +2
-0+bitmap_table_to_be (tb, tb_size); + ret = bdrv_pwrite(bs->file, tb_offset, tb, tb_size *
sizeof(tb[0]));block/qcow2-bitmap.c +9
-0+if (ret <0 ) { + goto fail; + } + +g_free (tb); + + entry->bitmap_table_offset = tb_offset; + entry->bitmap_table_size = tb_size; + entry->flags = bdrv_dirty_bitmap_granularity(bitmap) ?Quoted 1 line
BME_FLAG_AUTO: 0;s/granularity/get_autoload/
Quoted 3 lines
block/qcow2-bitmap.c +2
-0+ entry->type =BT_DIRTY_TRACKING_BITMAP ; + entry->granularity_bits =ctz32(bdrv_dirty_bitmap_granularity(bitmap));
You should probably check somewhere that the resulting value for
entry->granularity_bitsis in the BME_{MIN,MAX}_GRANULARITY_BITS range.Quoted 1 line
+
entry->name_size=strlen(bm_name);And that this length does not exceed
BME_MAX_NAME_SIZE.Quoted 11 lines
block/qcow2-bitmap.c +10
-0+ entry->extra_data_size =0 ; +memcpy (entry +1 , bm_name, entry->name_size); + +return 0 ; + +fail: +clear_bitmap_table (bs, tb, tb_size); + +if (tb_offset >0 ) { + qcow2_free_clusters(bs, tb_offset, tb_size,
QCOW2_DISCARD_ALWAYS);As before, I'd vote for
QCOW2_DISCARD_OTHER.Quoted 99 lines
block/qcow2-bitmap.c +8
-0+ } + +g_free (tb); + +return ret; +} + +static Qcow2BitmapDirEntry *find_bitmap_by_name(uint8_t
*bitmap_directory,
+size_tsize, const
char *name)block/qcow2-bitmap.c +40
-0+{ + Qcow2BitmapDirEntry *e; + + for_each_bitmap_dir_entry(e, bitmap_directory, size) { + if (strncmp((char *)(e + 1), name, e->name_size) == 0) { + return e; + } + } + + return NULL; +} + +void qcow2_store_persistent_bitmaps(BlockDriverState *bs, Error **errp) +{ + BdrvDirtyBitmap *bm; + BDRVQcow2State *s = bs->opaque; + uint32_t new_nb_bitmaps = s->nb_bitmaps; + uint64_t new_dir_size = s->bitmap_directory_size; + uint8_t *dir = NULL, *new_dir = NULL; + int ret; + Qcow2BitmapDirEntry *new_pos; + + if (s->nb_bitmaps > 0) { + dir = directory_read(bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); + if (dir == NULL) { + goto out; + } + } + + for (bm = bdrv_dirty_bitmap_next(bs, NULL); bm != NULL; + bm = bdrv_dirty_bitmap_next(bs, bm)) { + const char *name = bdrv_dirty_bitmap_name(bm); + + if (!bdrv_dirty_bitmap_get_persistance(bm)) { + continue; + } + + if (s->nb_bitmaps > 0 && + find_bitmap_by_name(dir, s->bitmap_directory_size,name)) {
block/qcow2-bitmap.c +2
-0+ error_setg(errp, + "Can't store bitmap '%s' to '%s', as italready exists",
block/qcow2-bitmap.c +21
-0+ name, bdrv_get_device_or_node_name(bs)); + goto out; + } + + new_nb_bitmaps++; + new_dir_size +=calc_dir_entry_size (strlen (name),0 ); + } + +if (s->nb_bitmaps == new_nb_bitmaps) { +/* No new bitmaps - nothing to do */ + goto out; + } + + new_dir =g_try_malloc0 (new_dir_size); +if (new_dir ==NULL ) { +error_setg (errp,"Can't allocate space for bitmap directory." ); + goto out; + } + +memcpy (new_dir, dir, s->bitmap_directory_size); + new_pos = (Qcow2BitmapDirEntry *)(new_dir +
s->bitmap_directory_size);block/qcow2-bitmap.c +10
-0+ +for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm !=NULL ; + bm =bdrv_dirty_bitmap_next (bs, bm)) { +if (!bdrv_dirty_bitmap_get_persistance (bm)) { +continue ; + } + + ret =store_bitmap (bs, bm, new_pos); +if (ret <0 ) { + error_setg_errno(errp, -ret, "Can't store bitmap '%s' to'%s'",
block/qcow2-bitmap.c +10
-0+bdrv_dirty_bitmap_name (bm), + bdrv_get_device_or_node_name(bs)); + goto out; + } + new_pos =next_dir_entry (new_pos); + } + + ret =directory_update (bs, new_dir, new_dir_size, new_nb_bitmaps); +if (ret <0 ) { + error_setg_errno(errp, -ret, "Can't update bitmap directoryin '%s'",
block/qcow2-bitmap.c out: · line 806
Quoted 7 lines
block/qcow2-bitmap.c +7
-0+ bdrv_get_device_or_node_name(bs)); + goto out; + } + +out: +g_free (new_dir); +g_free (dir);This error path leaks all the bitmaps that have been written successfully (if any). I guess this is more or less fine if
directory_update()failed (because you can't really tell the state of the image header afterdirectory_update(), so better be safe) but it's not so fine if just somestore_bitmap()failed.Max
Quoted 29 lines
block/qcow2-bitmap.c +1
-0+}block/qcow2.c +2
-0@@ -3493,6 +3493,8 @@ BlockDriver bdrv_qcow2 = { .bdrv_detach_aio_context = qcow2_detach_aio_context, .bdrv_attach_aio_context = qcow2_attach_aio_context, + + .bdrv_store_persistent_bitmaps = qcow2_store_persistent_bitmaps, };static void bdrv_qcow2_init (void )block/qcow2.h +1
-0@@ -627,4 +627,6 @@ int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset, void **table);void qcow2_cache_put (BlockDriverState *bs,Qcow2Cache *c,void **table); +void qcow2_store_persistent_bitmaps(BlockDriverState *bs, Error**errp);
+
#endif
Signature · 2 lines
Best regards, Vladimir
PATCH v7 12/22 qcow2-bitmap: add IN_USE flag
In reply to Vladimir Sementsov-OgievskiyThis flag means that the bitmap is now in use by the software or was not successfully saved. In any way, with this flag set the bitmap data must be considered inconsistent and should not be loaded.
With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.
We also remove in-use bitmaps from the image on open.
--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
block/qcow2-bitmap.c +16 -1
@@ -28,6 +28,7 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "qemu/cutils.h" +#include "exec/log.h" #include "block/block_int.h" #include "block/qcow2.h"
@@ -44,7 +45,8 @@#define BME_MAX_NAME_SIZE 1023/* Bitmap directory entry flags */ -#define BME_RESERVED_FLAGS 0xfffffffd +#define BME_RESERVED_FLAGS 0xfffffffc +#define BME_FLAG_IN_USE 1#define BME_FLAG_AUTO (1U << 1)/* bits [1, 8] U [56, 63] are reserved */
@@ -287,6 +289,11 @@ static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs,BdrvDirtyBitmap *bitmap =NULL ;char *name =g_strndup (dir_entry_name_notcstr (entry), entry->name_size); +if (entry->flags &BME_FLAG_IN_USE ) { +error_setg (errp,"Bitmap '%s' is in use" , name); + goto fail; + } + ret =bitmap_table_load (bs, entry, &bitmap_table);if (ret <0 ) { error_setg_errno(errp, -ret,
@@ -533,6 +540,14 @@ int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp)for_each_bitmap_dir_entry (e, dir, dir_size) {uint32_t fl = e->flags; +if (fl &BME_FLAG_IN_USE ) { +qemu_log ("qcow2 warning: " +"removing in_use bitmap '%.*s' for image %s.\n" , + e->name_size, (char *)(e +1 ),bdrv_get_device_or_node_name (bs)); + +continue ; + } +if (fl &BME_FLAG_AUTO ) {BdrvDirtyBitmap *bitmap =load_bitmap (bs, e, errp);if (bitmap ==NULL ) {
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
This flag means that the bitmap is now in use by the software or was not successfully saved. In any way, with this flag set the bitmap data must be considered inconsistent and should not be loaded.
With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.
Quoted 6 lines
We also remove in-use bitmaps from the image on open.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
Don't you want to make use of this flag? It would appear useful to me if you just marked autoload bitmaps as in_use instead of deleting them from the image when it's opened and then overwrite them when the image is closed.
Quoted 13 lines
block/qcow2-bitmap.c +1
-0@@ -28,6 +28,7 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "qemu/cutils.h" +#include "exec/log.h" #include "block/block_int.h" #include "block/qcow2.h"
block/qcow2-bitmap.c line 49
Quoted 6 lines
block/qcow2-bitmap.c +2
-1@@ -44,7 +45,8 @@#define BME_MAX_NAME_SIZE 1023/* Bitmap directory entry flags */ -#define BME_RESERVED_FLAGS 0xfffffffd +#define BME_RESERVED_FLAGS 0xfffffffc +#define BME_FLAG_IN_USE 1
This should be (1U << 0) to be consistent with the definition of BME_FLAG_AUTO.
Quoted 16 lines
block/qcow2-bitmap.c +5
-0#define BME_FLAG_AUTO (1U << 1)/* bits [1, 8] U [56, 63] are reserved */ @@ -287,6 +289,11 @@ static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs,BdrvDirtyBitmap *bitmap =NULL ;char *name =g_strndup (dir_entry_name_notcstr (entry), entry->name_size); +if (entry->flags &BME_FLAG_IN_USE ) { +error_setg (errp,"Bitmap '%s' is in use" , name); + goto fail; + } + ret =bitmap_table_load (bs, entry, &bitmap_table);if (ret <0 ) { error_setg_errno(errp, -ret,
block/qcow2-bitmap.c int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp) · line 546
Quoted 7 lines
block/qcow2-bitmap.c +4
-0@@ -533,6 +540,14 @@ int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp)for_each_bitmap_dir_entry (e, dir, dir_size) {uint32_t fl = e->flags; +if (fl &BME_FLAG_IN_USE ) { +qemu_log ("qcow2 warning: " +"removing in_use bitmap '%.*s' for image %s.\n" , + e->name_size, (char *)(e +1 ),bdrv_get_device_or_node_name (bs));
I'm not sure whether qemu_log() is the right way to do this. I think fprintf() to stderr might actually be more appropriate. qemu_log() looks like it's mostly used for debugging purposes.
Also, this is not a warning. You are just doing it. You are not warning someone about a cliff when he's already fallen off.
But you can actually make it a warning: Just leave the bitmap in the image (maybe someone can do something with it?) and instead let qemu-img check clean it up. The warning should then just be "Warning: Ignoring in_use bitmap %.*s, use qemu-img check -r all to delete it".
Max
Quoted 8 lines
block/qcow2-bitmap.c +4
-0+ +continue ; + } +if (fl &BME_FLAG_AUTO ) {BdrvDirtyBitmap *bitmap =load_bitmap (bs, e, errp);if (bitmap ==NULL ) {
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
07.10.2016 22:44, Max Reitz пишет:
Quoted 12 lines · Vladimir Sementsov-Ogievskiy
Quoted 11 lines
This flag means that the bitmap is now in use by the software or was not successfully saved. In any way, with this flag set the bitmap data must be considered inconsistent and should not be loaded.
With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.
We also remove in-use bitmaps from the image on open.
Quoted 7 lines
Quoted 4 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Don't you want to make use of this flag? It would appear useful to me if you just marked autoload bitmaps as
in_useinstead of deleting them from the image when it's opened and then overwrite them when the image is closed.
And what is the use of it?
Any way, storing an invalid copy of online data is bad I think. Therefore I will have to free bitmap data clusters and zero bitmap table in file.
Quoted 69 lines
Quoted 14 lines
block/qcow2-bitmap.c +1
-0@@ -28,6 +28,7 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "qemu/cutils.h" +#include "exec/log.h" #include "block/block_int.h" #include "block/qcow2.h" block/qcow2-bitmap.c line 49
Quoted 6 lines
block/qcow2-bitmap.c +2
-1@@ -44,7 +45,8 @@#define BME_MAX_NAME_SIZE 1023/* Bitmap directory entry flags */ -#define BME_RESERVED_FLAGS 0xfffffffd +#define BME_RESERVED_FLAGS 0xfffffffc +#define BME_FLAG_IN_USE 1This should be (1U << 0) to be consistent with the definition of
BME_FLAG_AUTO.Quoted 16 lines
block/qcow2-bitmap.c +5
-0#define BME_FLAG_AUTO (1U << 1)/* bits [1, 8] U [56, 63] are reserved */ @@ -287,6 +289,11 @@ static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs,BdrvDirtyBitmap *bitmap =NULL ;char *name =g_strndup (dir_entry_name_notcstr (entry), entry->name_size); +if (entry->flags &BME_FLAG_IN_USE ) { +error_setg (errp,"Bitmap '%s' is in use" , name); + goto fail; + } + ret =bitmap_table_load (bs, entry, &bitmap_table);if (ret <0 ) { error_setg_errno(errp, -ret,block/qcow2-bitmap.c int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp) · line 546
Quoted 7 lines
block/qcow2-bitmap.c +4
-0@@ -533,6 +540,14 @@ int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp)for_each_bitmap_dir_entry (e, dir, dir_size) {uint32_t fl = e->flags; +if (fl &BME_FLAG_IN_USE ) { +qemu_log ("qcow2 warning: " +"removing in_use bitmap '%.*s' for image %s.\n" , + e->name_size, (char *)(e +1 ),bdrv_get_device_or_node_name (bs));I'm not sure whether
qemu_log()is the right way to do this. I thinkfprintf()to stderr might actually be more appropriate.qemu_log()looks like it's mostly used for debugging purposes.Also, this is not a warning. You are just doing it. You are not warning someone about a cliff when he's already fallen off.
But you can actually make it a warning: Just leave the bitmap in the image (maybe someone can do something with it?) and instead let qemu-img check clean it up. The warning should then just be "Warning: Ignoring
in_usebitmap %.*s, use qemu-img check -r all to delete it".Max
Quoted 8 lines
block/qcow2-bitmap.c +4
-0+ +continue ; + } +if (fl &BME_FLAG_AUTO ) {BdrvDirtyBitmap *bitmap =load_bitmap (bs, e, errp);if (bitmap ==NULL ) {
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 17 lines · Vladimir Sementsov-Ogievskiy
07.10.2016 22:44, Max Reitz пишет:
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 15 lines
This flag means that the bitmap is now in use by the software or was not successfully saved. In any way, with this flag set the bitmap data must be considered inconsistent and should not be loaded.
With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.
We also remove in-use bitmaps from the image on open.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
Quoted 6 lines
Quoted 4 lines
Don't you want to make use of this flag? It would appear useful to me if you just marked autoload bitmaps as
in_useinstead of deleting them from the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
Quoted 3 lines
Any way, storing an invalid copy of online data is bad I think. Therefore I will have to free bitmap data clusters and zero bitmap table in file.
Why can't you just set this flag to mark it invalid? In case qemu crashes, in one case you'll have some invalid bitmaps and in the other you won't have any of them at all. I don't think there's a difference from that perspective.
Max
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
21.10.2016 22:58, Max Reitz пишет:
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Quoted 17 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 15 lines
This flag means that the bitmap is now in use by the software or was not successfully saved. In any way, with this flag set the bitmap data must be considered inconsistent and should not be loaded.
With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.
We also remove in-use bitmaps from the image on open.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Don't you want to make use of this flag? It would appear useful to me if you just marked autoload bitmaps as
in_useinstead of deleting them from the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Quoted 11 lines
Quoted 4 lines
Any way, storing an invalid copy of online data is bad I think. Therefore I will have to free bitmap data clusters and zero bitmap table in file.
Why can't you just set this flag to mark it invalid? In case qemu crashes, in one case you'll have some invalid bitmaps and in the other you won't have any of them at all. I don't think there's a difference from that perspective.
Max
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 23 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines
This flag means that the bitmap is now in use by the software or was not successfully saved. In any way, with this flag set the bitmap data must be considered inconsistent and should not be loaded.
With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.
Quoted 6 lines
We also remove in-use bitmaps from the image on open.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Don't you want to make use of this flag? It would appear useful to me if you just marked autoload bitmaps as
in_useinstead of deleting them from the image when it's opened and then overwrite them when the image is closed.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of in_use bitmaps on close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).
Also, if I don't free them on open, I'll have to free them on remove of bdrv dirty bitmap..
Quoted 14 lines
Quoted 12 lines
Quoted 5 lines
Any way, storing an invalid copy of online data is bad I think. Therefore I will have to free bitmap data clusters and zero bitmap table in file.
Why can't you just set this flag to mark it invalid? In case qemu crashes, in one case you'll have some invalid bitmaps and in the other you won't have any of them at all. I don't think there's a difference from that perspective.
Max
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 34 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 33 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and
must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 7 lines
Quoted 6 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them from
the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
Quoted 8 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).
It's not a real problem, but it does take time, and I maintain that it's time it doesn't need to take because you can just use the in_use flag.
I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...
Quoted 2 lines
Also, if I don't free them on open, I'll have to free them on remove of bdrv dirty bitmap..
Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted by the user or when the image file is closed. Just setting the single in_use flag for all of the auto-loaded bitmaps will basically not take any time.
On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some of the existing clusters.
Max
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 67 lines · Max Reitz
Quoted 34 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 33 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and
must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 7 lines
Quoted 6 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them from
the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
Quoted 8 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).It's not a real problem, but it does take time, and I maintain that it's time it doesn't need to take because you can just use the
in_useflag.I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...
Quoted 2 lines
Also, if I don't free them on open, I'll have to free them on remove of bdrv dirty bitmap..
Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted by the user or when the image file is closed. Just setting the single
in_useflag for all of the auto-loaded bitmaps will basically not take any time.
Quoted 4 lines
On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some of the existing clusters.
By the way, dealing with removal of bitmaps when they are deleted by the user is also positive when it comes to migration or read-only disks that are later reopened R/W: Currently, as far as I can see, you just keep the auto-load bitmaps in the image if the image is part of an incoming migration or opened read-only, but then you continue under the assumption that they are removed from the image. That's not very good.
If you delete the bitmaps only when the user asks you to, then you can just return an error that the bitmap cannot be removed at this time because the image file is read-only or used in migration (and I don't think the latter can even happen when it's the user who has requested removal of the bitmap).
Max
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 68 lines · Max Reitz
Quoted 63 lines · Max Reitz
Quoted 33 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 32 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and
must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 7 lines
Quoted 6 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them from
the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
Quoted 7 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).It's not a real problem, but it does take time, and I maintain that it's time it doesn't need to take because you can just use the
in_useflag.I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...
Quoted 2 lines
Also, if I don't free them on open, I'll have to free them on remove of bdrv dirty bitmap..
Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted by the user or when the image file is closed. Just setting the single
in_useflag for all of the auto-loaded bitmaps will basically not take any time.Quoted 4 lines
On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some of the existing clusters.
Quoted 6 lines
By the way, dealing with removal of bitmaps when they are deleted by the user is also positive when it comes to migration or read-only disks that are later reopened R/W: Currently, as far as I can see, you just keep the auto-load bitmaps in the image if the image is part of an incoming migration or opened read-only, but then you continue under the assumption that they are removed from the image. That's not very good.
You are right, I need to handle reopening more carefully. In my way, I need to delete bitmaps when reopening R/W and in yours - set in_use bit.
Quoted 9 lines
If you delete the bitmaps only when the user asks you to, then you can just return an error that the bitmap cannot be removed at this time because the image file is read-only or used in migration (and I don't think the latter can even happen when it's the user who has requested removal of the bitmap).
Max
Ok, I'll rewrite this way, using 'in_use' bit and trying to avoid allocate/free overhead.
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 85 lines · Vladimir Sementsov-Ogievskiy
Quoted 68 lines · Max Reitz
Quoted 63 lines · Max Reitz
Quoted 33 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 32 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and
must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 7 lines
Quoted 6 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them from
the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
Quoted 7 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).It's not a real problem, but it does take time, and I maintain that it's time it doesn't need to take because you can just use the
in_useflag.I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...
Quoted 2 lines
Also, if I don't free them on open, I'll have to free them on remove of bdrv dirty bitmap..
Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted by the user or when the image file is closed. Just setting the single
in_useflag for all of the auto-loaded bitmaps will basically not take any time.Quoted 4 lines
On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some of the existing clusters.
Quoted 6 lines
By the way, dealing with removal of bitmaps when they are deleted by the user is also positive when it comes to migration or read-only disks that are later reopened R/W: Currently, as far as I can see, you just keep the auto-load bitmaps in the image if the image is part of an incoming migration or opened read-only, but then you continue under the assumption that they are removed from the image. That's not very good.
You are right, I need to handle reopening more carefully. In my way, I need to delete bitmaps when reopening R/W and in yours - set
in_usebit.Quoted 7 lines
If you delete the bitmaps only when the user asks you to, then you can just return an error that the bitmap cannot be removed at this time because the image file is read-only or used in migration (and I don't think the latter can even happen when it's the user who has requested removal of the bitmap).
Quoted 6 lines
Quoted 2 lines
Max
Ok, I'll rewrite this way, using '
in_use' bit and trying to avoid allocate/free overhead.
Trying to reuse clusters of in_use bitmaps (including contiguous allocations for bitmap tables) will complicate things a lot.. We can use extra clusters of one in_use bitmap to save another one, the same is for bitmap tables. Extra clusters of old bitmap table (in case of resize) can be used for saving other bitmap data, etc..
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 86 lines · Vladimir Sementsov-Ogievskiy
Quoted 85 lines · Vladimir Sementsov-Ogievskiy
Quoted 68 lines · Max Reitz
Quoted 63 lines · Max Reitz
Quoted 33 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 32 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and
must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 7 lines
Quoted 6 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them from
the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
Quoted 7 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).It's not a real problem, but it does take time, and I maintain that it's time it doesn't need to take because you can just use the
in_useflag.I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...
Quoted 2 lines
Also, if I don't free them on open, I'll have to free them on remove of bdrv dirty bitmap..
Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted by the user or when the image file is closed. Just setting the single
in_useflag for all of the auto-loaded bitmaps will basically not take any time.Quoted 4 lines
On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some of the existing clusters.
Quoted 6 lines
By the way, dealing with removal of bitmaps when they are deleted by the user is also positive when it comes to migration or read-only disks that are later reopened R/W: Currently, as far as I can see, you just keep the auto-load bitmaps in the image if the image is part of an incoming migration or opened read-only, but then you continue under the assumption that they are removed from the image. That's not very good.
You are right, I need to handle reopening more carefully. In my way, I need to delete bitmaps when reopening R/W and in yours - set
in_usebit.Quoted 7 lines
If you delete the bitmaps only when the user asks you to, then you can just return an error that the bitmap cannot be removed at this time because the image file is read-only or used in migration (and I don't think the latter can even happen when it's the user who has requested removal of the bitmap).
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
Max
Ok, I'll rewrite this way, using '
in_use' bit and trying to avoid allocate/free overhead.Trying to reuse clusters of
in_usebitmaps (including contiguous allocations for bitmap tables) will complicate things a lot.. We can use extra clusters of onein_usebitmap to save another one, the same is for bitmap tables. Extra clusters of old bitmap table (in case of resize) can be used for saving other bitmap data, etc..
A compromise strategy of reusing:
1. accumulate all data clusters of in_use bitmaps, which was loaded by this qcow2 instance (we will not touch other old in_use bitmaps) to free_clusters list
2. for each persistent BdrvDirtyBitmap:
If there is corresponding in_use bitmap in the image, and its
table size == needed table size (there was no resizes), then let's reuse bitmap table.
else, move old bitmap table clusters to free_clusters and
allocate new table.
3. for each persistent BdrvDirtyBitmap:
For bitmap data clusters, take them from free_clusters list, and
if it is empty - allocate new clusters.
4. free extra clusters from free_clusters list if any.
This strategy is not optimal, but not bad I thing. Is it ok for you? (I'm not sure that this all is not premature optimization, and may be true way is to just free all old staff and reallocate it, as I do.)
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 112 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Vladimir Sementsov-Ogievskiy
Quoted 85 lines · Vladimir Sementsov-Ogievskiy
Quoted 68 lines · Max Reitz
Quoted 63 lines · Max Reitz
Quoted 33 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 32 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and
must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 7 lines
Quoted 6 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them from
the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
Quoted 7 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).It's not a real problem, but it does take time, and I maintain that it's time it doesn't need to take because you can just use the
in_useflag.I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...
Quoted 2 lines
Also, if I don't free them on open, I'll have to free them on remove of bdrv dirty bitmap..
Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted by the user or when the image file is closed. Just setting the single
in_useflag for all of the auto-loaded bitmaps will basically not take any time.Quoted 4 lines
On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some of the existing clusters.
Quoted 6 lines
By the way, dealing with removal of bitmaps when they are deleted by the user is also positive when it comes to migration or read-only disks that are later reopened R/W: Currently, as far as I can see, you just keep the auto-load bitmaps in the image if the image is part of an incoming migration or opened read-only, but then you continue under the assumption that they are removed from the image. That's not very good.
You are right, I need to handle reopening more carefully. In my way, I need to delete bitmaps when reopening R/W and in yours - set
in_usebit.Quoted 7 lines
If you delete the bitmaps only when the user asks you to, then you can just return an error that the bitmap cannot be removed at this time because the image file is read-only or used in migration (and I don't think the latter can even happen when it's the user who has requested removal of the bitmap).
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
Max
Ok, I'll rewrite this way, using '
in_use' bit and trying to avoid allocate/free overhead.Trying to reuse clusters of
in_usebitmaps (including contiguous allocations for bitmap tables) will complicate things a lot.. We can use extra clusters of onein_usebitmap to save another one, the same is for bitmap tables. Extra clusters of old bitmap table (in case of resize) can be used for saving other bitmap data, etc..A compromise strategy of reusing:
1. accumulate all data clusters of
in_usebitmaps, which was loaded by this qcow2 instance (we will not touch other oldin_usebitmaps) tofree_clusterslist
2. for each persistent BdrvDirtyBitmap:
If there is correspondingin_usebitmap in the image, and its
table size == needed table size (there was no resizes), then let's reuse bitmap table.
else, move old bitmap table clusters tofree_clustersand
allocate new table.
3. for each persistent BdrvDirtyBitmap:
For bitmap data clusters, take them fromfree_clusterslist,
and if it is empty - allocate new clusters.
4. free extra clusters fromfree_clusterslist if any.
Quoted 3 lines
This strategy is not optimal, but not bad I thing. Is it ok for you? (I'm not sure that this all is not premature optimization, and may be true way is to just free all old staff and reallocate it, as I do.)
With something like this it is not trivial to maintain consistency in qcow2 image in case of fail on reusing (we can get double-linked clusters, or dead links in the image on disk).. Simple solution will be to remove all in_use bitmaps from header extension, or reallocate for them bitmap tables with all zeroes. So, finally:
target: we want to reuse clusters of in_use bitmaps, which was loaded by current qcow2 driver (let's call them our in_use bitmaps.
sequence:
1. save bitmap directory in memory
2. remove our in_use bitmaps from bitmap directory in disk. from this point, in case of fail we will finish with some leaking clusters and no corruptions.
3. free_clusters list = all data clusters of our in_use bitmaps
4. for each our in_use bitmap: If table size is not appropriate move bitmap table clusters to free_clusters
5. save bitmaps, using free_clusters list for data clusters, old existing bitmap tables and allocating new space when needed.
6. free extra clusters from free_clusters list if any
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 119 lines · Vladimir Sementsov-Ogievskiy
Quoted 112 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Vladimir Sementsov-Ogievskiy
Quoted 85 lines · Vladimir Sementsov-Ogievskiy
Quoted 68 lines · Max Reitz
Quoted 63 lines · Max Reitz
Quoted 33 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 32 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and
must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 7 lines
Quoted 6 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them from
the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
Quoted 7 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).It's not a real problem, but it does take time, and I maintain that it's time it doesn't need to take because you can just use the
in_useflag.I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...
Quoted 2 lines
Also, if I don't free them on open, I'll have to free them on remove of bdrv dirty bitmap..
Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted by the user or when the image file is closed. Just setting the single
in_useflag for all of the auto-loaded bitmaps will basically not take any time.Quoted 4 lines
On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some of the existing clusters.
Quoted 6 lines
By the way, dealing with removal of bitmaps when they are deleted by the user is also positive when it comes to migration or read-only disks that are later reopened R/W: Currently, as far as I can see, you just keep the auto-load bitmaps in the image if the image is part of an incoming migration or opened read-only, but then you continue under the assumption that they are removed from the image. That's not very good.
You are right, I need to handle reopening more carefully. In my way, I need to delete bitmaps when reopening R/W and in yours - set
in_usebit.Quoted 7 lines
If you delete the bitmaps only when the user asks you to, then you can just return an error that the bitmap cannot be removed at this time because the image file is read-only or used in migration (and I don't think the latter can even happen when it's the user who has requested removal of the bitmap).
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
Max
Ok, I'll rewrite this way, using '
in_use' bit and trying to avoid allocate/free overhead.Trying to reuse clusters of
in_usebitmaps (including contiguous allocations for bitmap tables) will complicate things a lot.. We can use extra clusters of onein_usebitmap to save another one, the same is for bitmap tables. Extra clusters of old bitmap table (in case of resize) can be used for saving other bitmap data, etc..A compromise strategy of reusing:
1. accumulate all data clusters of
in_usebitmaps, which was loaded by this qcow2 instance (we will not touch other oldin_usebitmaps) tofree_clusterslist
2. for each persistent BdrvDirtyBitmap:
If there is correspondingin_usebitmap in the image, and its
table size == needed table size (there was no resizes), then let's reuse bitmap table.
else, move old bitmap table clusters tofree_clustersand
allocate new table.
3. for each persistent BdrvDirtyBitmap:
For bitmap data clusters, take them fromfree_clusterslist,
and if it is empty - allocate new clusters.
4. free extra clusters fromfree_clusterslist if any.Quoted 3 lines
This strategy is not optimal, but not bad I thing. Is it ok for you? (I'm not sure that this all is not premature optimization, and may be true way is to just free all old staff and reallocate it, as I do.)
With something like this it is not trivial to maintain consistency in qcow2 image in case of fail on reusing (we can get double-linked clusters, or dead links in the image on disk).. Simple solution will be to remove all
in_usebitmaps from header extension, or reallocate for them bitmap tables with all zeroes. So, finally:
Quoted 7 lines
target: we want to reuse clusters of
in_usebitmaps, which was loaded by current qcow2 driver (let's call them ourin_usebitmaps.sequence:
1. save bitmap directory in memory
2. remove ourin_usebitmaps from bitmap directory in disk. from this point, in case of fail we will finish with some leaking clusters and no corruptions.
3.free_clusterslist = all data clusters of ourin_usebitmaps
4. for each ourin_usebitmap: If table size is not appropriate move bitmap table clusters tofree_clusters
better is to free it, to not deceiveqcow2 allocator.. Can I allocate several clusters by alloc(size), then assume that I have (size + cluster_size - 1)/cluster_size clusters and that I can free these clusters separately?
Quoted 4 lines
5. save bitmaps, using
free_clusterslist for data clusters, old existing bitmap tables and allocating new space when needed.
6. free extra clusters fromfree_clusterslist if any
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
In reply to Vladimir Sementsov-OgievskiyQuoted 108 lines · Vladimir Sementsov-Ogievskiy
Quoted 99 lines · Vladimir Sementsov-Ogievskiy
Quoted 81 lines · Max Reitz
Quoted 75 lines · Max Reitz
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 39 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 38 lines · Vladimir Sementsov-Ogievskiy
Quoted 33 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 24 lines · Vladimir Sementsov-Ogievskiy
Quoted 16 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data
must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove
bitmaps from the image after loading. But it defined in qcow2 spec
and
must be handled. Also, it can be used in future, if async schemes of
bitmap loading/saving are implemented.Quoted 7 lines
We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 8 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them
from
the image when it's opened and then overwrite them when the image is
closed.Quoted 1 line
And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't
need to allocate any new bitmap space when closing it.Quoted 8 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on
close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).It's not a real problem, but it does take time, and I maintain that it's
time it doesn't need to take because you can just use thein_useflag.I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file
once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...Quoted 3 lines
Also, if I don't free them on open, I'll have to free them on remove of
bdrv dirty bitmap..Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted
by the user or when the image file is closed. Just setting the singlein_useflag for all of the auto-loaded bitmaps will basically not take any time.Quoted 5 lines
On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some
of the existing clusters.Quoted 6 lines
By the way, dealing with removal of bitmaps when they are deleted by the user is also positive when it comes to migration or read-only disks that are later reopened R/W: Currently, as far as I can see, you just keep the auto-load bitmaps in the image if the image is part of an incoming migration or opened read-only, but then you continue under the assumption that they are removed from the image. That's not very good.
You are right, I need to handle reopening more carefully. In my way, I need to delete bitmaps when reopening R/W and in yours - set
in_usebit.Quoted 7 lines
If you delete the bitmaps only when the user asks you to, then you can just return an error that the bitmap cannot be removed at this time because the image file is read-only or used in migration (and I don't think the latter can even happen when it's the user who has requested removal of the bitmap).
Quoted 7 lines
Quoted 2 lines
Max
Ok, I'll rewrite this way, using '
in_use' bit and trying to avoid allocate/free overhead.
Quoted 5 lines
Trying to reuse clusters of
in_usebitmaps (including contiguous allocations for bitmap tables) will complicate things a lot.. We can use extra clusters of onein_usebitmap to save another one, the same is for bitmap tables. Extra clusters of old bitmap table (in case of resize) can be used for saving other bitmap data, etc..
I feel like you're trying to optimize too much. When rewriting the bitmap data, reuse only the clusters you have to write to anyway.
Say the bitmap table looks like this when you open the image:
[sparse 0, cluster 42, cluster 45, sparse 0, cluster 46]
("sparse 0" means no cluster is allocated and the bitmap table entry is 0, thus creating a sparse bitmap where that part is supposed to be read as 0.)
Now during runtime, you set some bits, maybe you even clear the whole bitmap at some point because you're doing a backup, and then you set some other bits, so when you close the image, the bitmap table would have to look like this:
[sparse 0, sparse 0, data, data, data]
Now, when you want to save that bitmap data, you just walk through the existing bitmap table. The first entry is a sparse 0, and the bitmap is sparse there, too, so that can stay as it is.
The second entry points to data but the bitmap is now sparse. You can then free the cluster (cluster 42) and write "sparse 0" into the bitmap table entry.
The third entry points to data and the bitmap contains data there, so you just write the data to cluster 45.
The fourth entry is sparse but your bitmap contains data. So now you need to allocate a cluster, maybe that will cluster 67, and write your data there, and make the bitmap table entry point there.
The fifth entry finally is handled just like the third entry.
So afterwards, your bitmap table would look like this:
[sparse 0, sparse 0, cluster 45, cluster 67, cluster 46]
It sounds as if you're trying to reuse all the clusters, i.e. you'd ideally get:
[sparse 0, sparse 0, cluster 45, cluster 42, cluster 46]
May be possible, but I really wouldn't worry about that. I'd consider it much too difficult. Just reuse the existing bitmap table.
Max
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
In reply to Vladimir Sementsov-OgievskiyQuoted 69 lines · Vladimir Sementsov-Ogievskiy
Quoted 68 lines · Max Reitz
Quoted 67 lines · Max Reitz
Quoted 33 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 32 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and
must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 7 lines
Quoted 6 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them from
the image when it's opened and then overwrite them when the image is closed.And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
Quoted 7 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).It's not a real problem, but it does take time, and I maintain that it's time it doesn't need to take because you can just use the
in_useflag.I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...
Quoted 2 lines
Also, if I don't free them on open, I'll have to free them on remove of bdrv dirty bitmap..
Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted by the user or when the image file is closed. Just setting the single
in_useflag for all of the auto-loaded bitmaps will basically not take any time.On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some of the existing clusters.
Quoted 8 lines
Quoted 6 lines
By the way, dealing with removal of bitmaps when they are deleted by the user is also positive when it comes to migration or read-only disks that are later reopened R/W: Currently, as far as I can see, you just keep the auto-load bitmaps in the image if the image is part of an incoming migration or opened read-only, but then you continue under the assumption that they are removed from the image. That's not very good.
You are right, I need to handle reopening more carefully. In my way, I need to delete bitmaps when reopening R/W and in yours - set
in_usebit.
Now I think, that loading auto_loading bitmaps in qcow2_open is wrong. I should load them only in bdrv_open, to avoid reloading bitmaps on drv->bdrv_open/drv->bdrv_close (they are called from bdrv_snapshot_goto).
So, it would be like this:
on bdrv_open, after drv->bdrv_open call drv->load_autoloading_bitmaps,
which will load bitmaps, mark them in_use in the image (if it is writable), create corresponding BdrvDirtyBitmaps. If there _any_ conflicts with existing BdrvDirtyBitmaps then fail.
on bdrv_close, before drv->bdrv_close call drv->store_persitstent_bitmaps,
which will store persistent bitmaps, set in_use to false and _delete_ corresponding BdrvDirtyBitmaps.
Also, in qcow2_reopen_prepare in case of changing write-ability from false to true we need to mark corresponding bitmaps in the image as in_use.. And something like this for incoming migration too.
qcow2_open will only load header extension fields to bs->opaque, and check these fields. It will not load bitmap directory.
Quoted 14 lines
Quoted 10 lines
If you delete the bitmaps only when the user asks you to, then you can just return an error that the bitmap cannot be removed at this time because the image file is read-only or used in migration (and I don't think the latter can even happen when it's the user who has requested removal of the bitmap).
Max
Ok, I'll rewrite this way, using '
in_use' bit and trying to avoid allocate/free overhead.
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
Quoted 99 lines · Vladimir Sementsov-Ogievskiy
Quoted 89 lines · Vladimir Sementsov-Ogievskiy
Quoted 81 lines · Max Reitz
Quoted 75 lines · Max Reitz
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
24.10.2016 13:32, Vladimir Sementsov-Ogievskiy пишет:
Quoted 39 lines
21.10.2016 22:58, Max Reitz пишет:
Quoted 38 lines · Vladimir Sementsov-Ogievskiy
Quoted 33 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 24 lines · Vladimir Sementsov-Ogievskiy
Quoted 16 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data
must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove
bitmaps from the image after loading. But it defined in qcow2 spec
and
must be handled. Also, it can be used in future, if async schemes of
bitmap loading/saving are implemented.Quoted 7 lines
We also remove in-use bitmaps from the image on open.
Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Quoted 8 lines
Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them
from
the image when it's opened and then overwrite them when the image is
closed.Quoted 1 line
And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't
need to allocate any new bitmap space when closing it.Quoted 8 lines
Quoted 2 lines
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
Is it a real problem to reallocate space in qcow2? If so, to minimaze allocations, I will have to make a list of clusters of
in_usebitmaps on
close, and then save current bitmaps to these clusters (and allocated some additional clusters, or free extra clusters).It's not a real problem, but it does take time, and I maintain that it's
time it doesn't need to take because you can just use thein_useflag.I wouldn't worry about reusing clusters of other bitmaps. Of course we can do that later on in some optimization but not now.
I just mean the basic case of some auto-loaded bitmap that is not being deleted while the VM is running and is just saved back to the image file
once it is closed. I don't expect that users will always consume all of the auto-loaded bitmaps while the VM is active...Quoted 3 lines
Also, if I don't free them on open, I'll have to free them on remove of
bdrv dirty bitmap..Yes, so? That takes time, but that is something the user will probably expect. I wouldn't expect opening of the file to take that time.
Overall, it doesn't matter time-wise whether you free the bitmap data when opening the image file or when the dirty bitmap is actually deleted
by the user or when the image file is closed. Just setting the singlein_useflag for all of the auto-loaded bitmaps will basically not take any time.Quoted 5 lines
On the other hand, as soon as just a single auto-loaded bitmap survives a VM (or qemu-img) invocation, you will very, very likely safe at least some time because writing the bitmap to the disk can reuse at least some
of the existing clusters.Quoted 6 lines
By the way, dealing with removal of bitmaps when they are deleted by the user is also positive when it comes to migration or read-only disks that are later reopened R/W: Currently, as far as I can see, you just keep the auto-load bitmaps in the image if the image is part of an incoming migration or opened read-only, but then you continue under the assumption that they are removed from the image. That's not very good.
Quoted 2 lines
You are right, I need to handle reopening more carefully. In my way, I need to delete bitmaps when reopening R/W and in yours - set
in_usebit.Now I think, that loading
auto_loadingbitmaps inqcow2_openis wrong. I should load them only inbdrv_open, to avoid reloading bitmaps on drv->bdrv_open/drv->bdrv_close (they are called frombdrv_snapshot_goto).So, it would be like this:
Quoted 8 lines
on
bdrv_open, afterdrv->bdrv_opencalldrv->load_autoloading_bitmaps, which will load bitmaps, mark themin_usein the image (if it is writable), create corresponding BdrvDirtyBitmaps. If there _any_ conflicts with existing BdrvDirtyBitmaps then fail.on
bdrv_close, beforedrv->bdrv_closecalldrv->store_persitstent_bitmaps, which will store persistent bitmaps, setin_useto false and _delete_ corresponding BdrvDirtyBitmaps.
Sounds good.
Quoted 3 lines
Also, in
qcow2_reopen_preparein case of changing write-ability from false to true we need to mark corresponding bitmaps in the image asin_use.. And something like this for incoming migration too.
Right.
Quoted 2 lines
qcow2_openwill only load header extension fields tobs->opaque, and check these fields. It will not load bitmap directory.
Yes, that sounds good.
Max
Re: [Qemu-devel] [PATCH 12/22] qcow2-bitmap: add IN_USE flag
In reply to Vladimir Sementsov-OgievskiyQuoted 27 lines · Vladimir Sementsov-Ogievskiy
21.10.2016 22:58, Max Reitz пишет:
Quoted 26 lines · Vladimir Sementsov-Ogievskiy
Quoted 25 lines
07.10.2016 22:44, Max Reitz пишет:
Quoted 24 lines · Vladimir Sementsov-Ogievskiy
Quoted 11 lines
This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.With current implementation this flag is never set, as we just remove bitmaps from the image after loading. But it defined in qcow2 spec and must be handled. Also, it can be used in future, if async schemes of bitmap loading/saving are implemented.
Quoted 6 lines
We also remove in-use bitmaps from the image on open.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)Don't you want to make use of this flag? It would appear useful to me if
you just marked autoload bitmaps asin_useinstead of deleting them from
the image when it's opened and then overwrite them when the image is closed.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
And what is the use of it?
You don't need to free any bitmaps when opening the file, and you don't need to allocate any new bitmap space when closing it.
As bitmaps are sparce in file, I need to allocate new space when closing. Or free it...
May happen. But not necessarily, and it will probably still save time as you can reuse existing allocations and don't have to free everything.
Max
PATCH v7 13/22 qcow2-bitmap: check constraints
In reply to Vladimir Sementsov-OgievskiyCheck bitmap header constraints as specified in docs/specs/qcow2.txt
--- block/qcow2-bitmap.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
block/qcow2-bitmap.c +39 -0
@@ -154,6 +154,34 @@ static inline void bitmap_directory_to_be(uint8_t *dir, size_t size) } } +static int check_constraints (BlockDriverState *bs,Qcow2BitmapDirEntry *h) +{ +BDRVQcow2State *s = bs->opaque; +uint64_t phys_bitmap_bytes = + (uint64_t )h->bitmap_table_size * s->cluster_size; +uint64_t max_virtual_bits = (phys_bitmap_bytes *8 ) << h->granularity_bits; +int64_t nb_sectors =bdrv_nb_sectors (bs); + +if (nb_sectors <0 ) { +return nb_sectors; + } + +int fail = + ((h->bitmap_table_size ==0 ) != (h->bitmap_table_offset ==0 )) || + (h->bitmap_table_offset % s->cluster_size) || + (h->bitmap_table_size >BME_MAX_TABLE_SIZE ) || + (phys_bitmap_bytes >BME_MAX_PHYS_SIZE ) || + (h->bitmap_table_offset !=0 && + (nb_sectors <<BDRV_SECTOR_BITS ) > max_virtual_bits) || + (h->granularity_bits >BME_MAX_GRANULARITY_BITS ) || + (h->granularity_bits <BME_MIN_GRANULARITY_BITS ) || + (h->flags &BME_RESERVED_FLAGS ) || + (h->name_size >BME_MAX_NAME_SIZE ) || + (h->type !=BT_DIRTY_TRACKING_BITMAP ); + +return fail ? -EINVAL :0 ; +} +static void clear_bitmap_table (BlockDriverState *bs,uint64_t *bitmap_table,uint32_t bitmap_table_size) {
@@ -372,6 +400,12 @@ static uint8_t *directory_read(BlockDriverState *bs, bdrv_get_device_or_node_name(bs)); goto fail; } + + ret =check_constraints (bs, e); +if (ret <0 ) { +error_setg (errp,"Bitmap doesn't satisfy the constraints." ); + goto fail; + } }assert ((uint8_t *)e == dir_end);
@@ -713,6 +747,11 @@ static int store_bitmap(BlockDriverState *bs, entry->extra_data_size =0 ;memcpy (entry +1 , bm_name, entry->name_size); + ret =check_constraints (bs, entry); +if (ret <0 ) { + goto fail; + } +return 0 ; fail:
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 13/22] qcow2-bitmap: check constraints
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Check bitmap header constraints as specified in
docs/specs/qcow2.txtSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
I'd pull this patch to some previous point in the series because the previous patches would already require you to check these constraints (which you just haven't done until now).
Quoted 40 lines
block/qcow2-bitmap.c +28
-0@@ -154,6 +154,34 @@ static inline void bitmap_directory_to_be(uint8_t *dir, size_t size) } } +static int check_constraints (BlockDriverState *bs,Qcow2BitmapDirEntry *h) +{ +BDRVQcow2State *s = bs->opaque; +uint64_t phys_bitmap_bytes = + (uint64_t )h->bitmap_table_size * s->cluster_size; +uint64_t max_virtual_bits = (phys_bitmap_bytes *8 ) << h->granularity_bits; +int64_t nb_sectors =bdrv_nb_sectors (bs); + +if (nb_sectors <0 ) { +return nb_sectors; + } + +int fail = + ((h->bitmap_table_size ==0 ) != (h->bitmap_table_offset ==0 )) || + (h->bitmap_table_offset % s->cluster_size) || + (h->bitmap_table_size >BME_MAX_TABLE_SIZE ) || + (phys_bitmap_bytes >BME_MAX_PHYS_SIZE ) || + (h->bitmap_table_offset !=0 && + (nb_sectors <<BDRV_SECTOR_BITS ) > max_virtual_bits) || + (h->granularity_bits >BME_MAX_GRANULARITY_BITS ) || + (h->granularity_bits <BME_MIN_GRANULARITY_BITS ) || + (h->flags &BME_RESERVED_FLAGS ) || + (h->name_size >BME_MAX_NAME_SIZE ) || + (h->type !=BT_DIRTY_TRACKING_BITMAP ); + +return fail ? -EINVAL :0 ; +} +static void clear_bitmap_table (BlockDriverState *bs,uint64_t *bitmap_table,uint32_t bitmap_table_size) {
block/qcow2-bitmap.c static uint8_t *directory_read(BlockDriverState *bs, · line 406
Quoted 7 lines
block/qcow2-bitmap.c +4
-0@@ -372,6 +400,12 @@ static uint8_t *directory_read(BlockDriverState *bs, bdrv_get_device_or_node_name(bs)); goto fail; } + + ret =check_constraints (bs, e); +if (ret <0 ) { +error_setg (errp,"Bitmap doesn't satisfy the constraints." );
I think I'd at least mention the name of the bitmap; also, no full stop at the end of error messages.
Quoted 6 lines
block/qcow2-bitmap.c +2
-0+ goto fail; + } }assert ((uint8_t *)e == dir_end);
block/qcow2-bitmap.c static int store_bitmap(BlockDriverState *bs, · line 754
Quoted 8 lines
block/qcow2-bitmap.c +5
-0@@ -713,6 +747,11 @@ static int store_bitmap(BlockDriverState *bs, entry->extra_data_size =0 ;memcpy (entry +1 , bm_name, entry->name_size); + ret =check_constraints (bs, entry); +if (ret <0 ) { + goto fail; + } +
As I said in my second reply to patch 9, I think it's a bit too late if we detect that the bitmap is actually invalid at this point. We really should notice earlier.
Apart from what would actually better for the user, it is actually too late to check the constraints here, as you have already written the bitmap data to disk. You should always check the constraints before reading and also before writing, not afterwards.
Max
Quoted 4 lines
block/qcow2-bitmap.c +0
-0return 0 ; fail:
PATCH v7 14/22 qcow2: delete bitmaps on truncate
In reply to Vladimir Sementsov-OgievskiyThis will not touch loaded bitmaps, as they are alredy removed from the image.
--- block/qcow2-bitmap.c | 38 ++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 9 ++++++--- block/qcow2.h | 2 ++ 3 files changed, 46 insertions(+), 3 deletions(-)
block/qcow2-bitmap.c +38 -0
@@ -624,6 +624,44 @@ out:return ret; } + +int qcow2_delete_bitmaps (BlockDriverState *bs) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint8_t *dir; +uint64_t dir_size; +Qcow2BitmapDirEntry *e; + +if (s->nb_bitmaps ==0 ) { +/* No bitmaps - nothing to do */ +return 0 ; + } + + dir_size = s->bitmap_directory_size; + dir =directory_read (bs, s->bitmap_directory_offset, + s->bitmap_directory_size,NULL ); +if (dir ==NULL ) { + ret = -EINVAL ; + goto out; + } + + ret =directory_update (bs,NULL ,0 ,0 ); +if (ret <0 ) { + goto out; + } + +/* to be consistent, free bitmap only after successfull directory update */ +for_each_bitmap_dir_entry (e, dir, dir_size) { +free_bitmap_clusters (bs, e); + } + +out: +g_free (dir); + +return ret; +} +/* store_bitmap_data() * Store bitmap to image, filling bitamp table accordingly. */
block/qcow2.c +6 -3
@@ -2592,9 +2592,12 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset)/* cannot proceed if image has bitmaps */ if (s->nb_bitmaps) { -/* FIXME */ -error_report ("Can't resize an image which has bitmaps" ); -return -ENOTSUP ; +/* FIXME: not loaded bitmaps will be lost */ + ret =qcow2_delete_bitmaps (bs); +if (ret <0 ) { +error_report ("Can't remove bitmaps from qcow2 on truncate" ); +return ret; + } }/* shrinking is currently not supported */
block/qcow2.h +2 -0
@@ -606,6 +606,8 @@ int qcow2_read_snapshots(BlockDriverState *bs);/* qcow2-bitmap.c functions */ int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp); +int qcow2_delete_bitmaps (BlockDriverState *bs); +/* qcow2-cache.c functions */ Qcow2Cache *qcow2_cache_create (BlockDriverState *bs,int num_tables);int qcow2_cache_destroy (BlockDriverState * bs,Qcow2Cache *c);
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 14/22] qcow2: delete bitmaps on truncate
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
This will not touch loaded bitmaps, as they are alredy removed from the image.
Well, yes, and I don't think that's ideal. :-)
I think marking them in_use instead of deleting them would be better.
Also, I'm not so sure whether this patch is an improvement at all. Wouldn't it be better to have a qemu-img tool (and/or QMP interface) for bitmap management and then making the user delete all the bitmaps before the image can be resized?
Max
Quoted 91 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 38 ++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 9 ++++++--- block/qcow2.h | 2 ++ 3 files changed, 46 insertions(+), 3 deletions(-)block/qcow2-bitmap.c +38
-0@@ -624,6 +624,44 @@ out:return ret; } + +int qcow2_delete_bitmaps (BlockDriverState *bs) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +uint8_t *dir; +uint64_t dir_size; +Qcow2BitmapDirEntry *e; + +if (s->nb_bitmaps ==0 ) { +/* No bitmaps - nothing to do */ +return 0 ; + } + + dir_size = s->bitmap_directory_size; + dir =directory_read (bs, s->bitmap_directory_offset, + s->bitmap_directory_size,NULL ); +if (dir ==NULL ) { + ret = -EINVAL ; + goto out; + } + + ret =directory_update (bs,NULL ,0 ,0 ); +if (ret <0 ) { + goto out; + } + +/* to be consistent, free bitmap only after successfull directory update */ +for_each_bitmap_dir_entry (e, dir, dir_size) { +free_bitmap_clusters (bs, e); + } + +out: +g_free (dir); + +return ret; +} +/* store_bitmap_data() * Store bitmap to image, filling bitamp table accordingly. */ block/qcow2.c +6
-3@@ -2592,9 +2592,12 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset)/* cannot proceed if image has bitmaps */ if (s->nb_bitmaps) { -/* FIXME */ -error_report ("Can't resize an image which has bitmaps" ); -return -ENOTSUP ; +/* FIXME: not loaded bitmaps will be lost */ + ret =qcow2_delete_bitmaps (bs); +if (ret <0 ) { +error_report ("Can't remove bitmaps from qcow2 on truncate" ); +return ret; + } }/* shrinking is currently not supported */ block/qcow2.h +2
-0@@ -606,6 +606,8 @@ int qcow2_read_snapshots(BlockDriverState *bs);/* qcow2-bitmap.c functions */ int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp); +int qcow2_delete_bitmaps (BlockDriverState *bs); +/* qcow2-cache.c functions */ Qcow2Cache *qcow2_cache_create (BlockDriverState *bs,int num_tables);int qcow2_cache_destroy (BlockDriverState * bs,Qcow2Cache *c);
PATCH v7 15/22 qcow2-bitmap: add autoclear bit
In reply to Vladimir Sementsov-OgievskiyAdd autoclear bit for handling rewriting image by old qemu version.
If autoclear bit is not set, but bitmaps extension is found it would not be loaded and warning will be generated.
--- block/qcow2-bitmap.c | 4 ++++ block/qcow2.c | 12 ++++++++++-- block/qcow2.h | 9 +++++++++ 3 files changed, 23 insertions(+), 2 deletions(-)
block/qcow2-bitmap.c +4 -0
@@ -515,6 +515,10 @@ static int directory_update(BlockDriverState *bs, uint8_t *new_dir,if (ret <0 ) { goto fail; } + + s->autoclear_features |=QCOW2_AUTOCLEAR_DIRTY_BITMAPS ; + }else { + s->autoclear_features &= ~(uint64_t )QCOW2_AUTOCLEAR_DIRTY_BITMAPS ; } s->bitmap_directory_offset = new_offset; s->bitmap_directory_size = new_size;
block/qcow2.c +10 -2
@@ -165,6 +165,13 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,break ;case QCOW2_EXT_MAGIC_DIRTY_BITMAPS : +if (!(s->autoclear_features &QCOW2_AUTOCLEAR_DIRTY_BITMAPS )) { +fprintf (stderr, +"WARNING: bitmaps_ext: autoclear flag is not " +"set, all bitmaps will be considered as inconsistent" ); +break ; + } + ret =bdrv_pread (bs->file, offset, &bitmaps_ext, ext.len);if (ret <0 ) { error_setg_errno(errp, -ret, "ERROR: bitmaps_ext: "
@@ -1206,8 +1213,9 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, }/* Clear unknown autoclear feature bits */ -if (!bs->read_only && !(flags &BDRV_O_INACTIVE ) && s->autoclear_features) { - s->autoclear_features =0 ; +if (!bs->read_only && !(flags &BDRV_O_INACTIVE ) && + (s->autoclear_features & ~QCOW2_AUTOCLEAR_MASK )) { + s->autoclear_features &=QCOW2_AUTOCLEAR_MASK ; ret =qcow2_update_header (bs);if (ret <0 ) {error_setg_errno (errp, -ret,"Could not update qcow2 header" );
block/qcow2.h +9 -0
@@ -215,6 +215,15 @@ enum { QCOW2_COMPAT_FEAT_MASK = QCOW2_COMPAT_LAZY_REFCOUNTS, }; +/* Autoclear feature bits */ +enum { +QCOW2_AUTOCLEAR_DIRTY_BITMAPS_BITNR =0 , +QCOW2_AUTOCLEAR_DIRTY_BITMAPS = +1 <<QCOW2_AUTOCLEAR_DIRTY_BITMAPS_BITNR , + +QCOW2_AUTOCLEAR_MASK =QCOW2_AUTOCLEAR_DIRTY_BITMAPS , +}; +enum qcow2_discard_type {QCOW2_DISCARD_NEVER =0 ,QCOW2_DISCARD_ALWAYS ,
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 15/22] qcow2-bitmap: add autoclear bit
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Add autoclear bit for handling rewriting image by old qemu version.
"Add support for the autoclear bit [...]"?
Quoted 3 lines
If autoclear bit is not set, but bitmaps extension is found it would not be loaded and warning will be generated.
"If the autoclear bit is not set, but the bitmaps extension is found, the bitmaps will not be loaded and a warning will be generated."
Quoted 7 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 4 ++++ block/qcow2.c | 12 ++++++++++-- block/qcow2.h | 9 +++++++++ 3 files changed, 23 insertions(+), 2 deletions(-)
Apart from the above, the patch looks good, but why does it come so late in the series?
So with the commit message fixed:
Re: [Qemu-devel] [PATCH 15/22] qcow2-bitmap: add autoclear bit
07.10.2016 23:11, Max Reitz пишет:
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
Quoted 1 line
Add autoclear bit for handling rewriting image by old qemu version.
"Add support for the autoclear bit [...]"?
finally "Add support for the autoclear bit to handle rewriting image by old qemu version" - ok?
Or I can merge this patch into two earlier patches 06 and 07.
Quoted 19 lines
Quoted 3 lines
If autoclear bit is not set, but bitmaps extension is found it would not be loaded and warning will be generated.
"If the autoclear bit is not set, but the bitmaps extension is found, the bitmaps will not be loaded and a warning will be generated."
Quoted 6 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 4 ++++ block/qcow2.c | 12 ++++++++++-- block/qcow2.h | 9 +++++++++ 3 files changed, 23 insertions(+), 2 deletions(-)Apart from the above, the patch looks good, but why does it come so late in the series?
So with the commit message fixed:
Reviewed-by Max Reitz <[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 15/22] qcow2-bitmap: add autoclear bit
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
07.10.2016 23:11, Max Reitz пишет:
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
Quoted 1 line
Add autoclear bit for handling rewriting image by old qemu version.
"Add support for the autoclear bit [...]"?
finally "Add support for the autoclear bit to handle rewriting image by old qemu version" - ok?
Or I can merge this patch into two earlier patches 06 and 07.
Sounds good.
Max
PATCH v7 16/22 qmp: add persistent flag to block-dirty-bitmap-add
In reply to Vladimir Sementsov-OgievskiyAdd optional 'persistent' flag to qmp command block-dirty-bitmap-add. Default is false.
--- blockdev.c | 12 +++++++++++- qapi/block-core.json | 7 ++++++- qmp-commands.hx | 5 ++++- 3 files changed, 21 insertions(+), 3 deletions(-)
blockdev.c +11 -1
@@ -1991,6 +1991,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,/* AIO context taken and released within qmp_block_dirty_bitmap_add */ qmp_block_dirty_bitmap_add (action->node, action->name, action->has_granularity, action->granularity, + action->has_persistent, action->persistent, &local_err);if (!local_err) {
@@ -2694,10 +2695,12 @@ out:void qmp_block_dirty_bitmap_add (const char *node,const char *name,bool has_granularity,uint32_t granularity, +bool has_persistent,bool persistent,Error **errp) {AioContext *aio_context;BlockDriverState *bs; +BdrvDirtyBitmap *bitmap;if (!name || name[0 ] =='\0' ) {error_setg (errp,"Bitmap name cannot be empty" );
@@ -2723,7 +2726,14 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name, granularity =bdrv_get_default_bitmap_granularity (bs); } -bdrv_create_dirty_bitmap (bs, granularity, name, errp); +if (!has_persistent) { + persistent = false; + } + + bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); +if (bitmap !=NULL ) { +bdrv_dirty_bitmap_set_persistance (bitmap, persistent); + } out:aio_context_release (aio_context);
qapi/block-core.json +6 -1
@@ -1235,10 +1235,15 @@ # @granularity: #optional the bitmap granularity, default is 64k for # block-dirty-bitmap-add # +# @persistent: #optional the bitmap is persistent, i.e. it will be saved to +# corresponding block device on it's close. Default is false. +# For block-dirty-bitmap-add. (Since 2.8) +# # Since 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', - 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32' } } + 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', + '*persistent': 'bool' } } ## # @block-dirty-bitmap-add
qmp-commands.hx +4 -1
@@ -1441,7 +1441,7 @@ EQMP { .name = "block-dirty-bitmap-add", - .args_type = "node:B,name:s,granularity:i?", + .args_type = "node:B,name:s,granularity:i?,persistent:b?", .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add, },
@@ -1458,6 +1458,9 @@ Arguments: - "node": device/node on which to create dirty bitmap (json-string) - "name": name of the new dirty bitmap (json-string) - "granularity": granularity to track writes with (int, optional) +- "persistent": bitmap will be saved to corresponding block device + on it's close. Block driver should maintain persistent bitmaps + (json-bool, optional, default false) (Since 2.8) Example:
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 16/22] qmp: add persistent flag to block-dirty-bitmap-add
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
Add optional 'persistent' flag to qmp command block-dirty-bitmap-add. Default is false.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- blockdev.c | 12 +++++++++++- qapi/block-core.json | 7 ++++++- qmp-commands.hx | 5 ++++-
This and later patches have conflicts with recent changes in the tree that removed qmp-commands.hx; hopefully shouldn't impact review of this series too badly.
qapi/block-core.json line 1239
Quoted 7 lines
+++
b/qapi/block-core.jsonqapi/block-core.json +2
-0@@ -1235,10 +1235,15 @@ # @granularity: #optional the bitmap granularity, default is 64k for # block-dirty-bitmap-add # +# @persistent: #optional the bitmap is persistent, i.e. it will be saved to +# corresponding block device on it's close. Default is false.
s/corresponding/the corresponding/
s/it's/its/
("it's" is only appropriate if you can substitute "it is" or "it has")
qmp-commands.hx Arguments: · line 1462
Quoted 6 lines
qmp-commands.hx +2
-0@@ -1458,6 +1458,9 @@ Arguments: - "node": device/node on which to create dirty bitmap (json-string) - "name": name of the new dirty bitmap (json-string) - "granularity": granularity to track writes with (int, optional) +- "persistent": bitmap will be saved to corresponding block device + on it's close. Block driver should maintain persistent bitmaps
One nice benefit of rebasing on top of qmp-commands.hx being removed is that you don't have to repeat yourself :)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 16/22] qmp: add persistent flag to block-dirty-bitmap-add
07.10.2016 22:52, Eric Blake пишет:
Quoted 27 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines
Add optional 'persistent' flag to qmp command block-dirty-bitmap-add. Default is false.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- blockdev.c | 12 +++++++++++- qapi/block-core.json | 7 ++++++- qmp-commands.hx | 5 ++++-This and later patches have conflicts with recent changes in the tree that removed qmp-commands.hx; hopefully shouldn't impact review of this series too badly.
qapi/block-core.json line 1239
Quoted 7 lines
+++
b/qapi/block-core.jsonqapi/block-core.json +2
-0@@ -1235,10 +1235,15 @@ # @granularity: #optional the bitmap granularity, default is 64k for # block-dirty-bitmap-add # +# @persistent: #optional the bitmap is persistent, i.e. it will be saved to +# corresponding block device on it's close. Default is false.s/corresponding/the corresponding/
s/it's/its/("it's" is only appropriate if you can substitute "it is" or "it has")
Quoted 8 lines
qmp-commands.hx Arguments: · line 1462
Quoted 6 lines
qmp-commands.hx +2
-0@@ -1458,6 +1458,9 @@ Arguments: - "node": device/node on which to create dirty bitmap (json-string) - "name": name of the new dirty bitmap (json-string) - "granularity": granularity to track writes with (int, optional) +- "persistent": bitmap will be saved to corresponding block device + on it's close. Block driver should maintain persistent bitmapsOne nice benefit of rebasing on top of qmp-commands.hx being removed is that you don't have to repeat yourself :)
As I understand, duplicating description is still here, in docs/qmp-commands.txt. I just don't have to repeat parameters definition (which is good, anyway)
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 16/22] qmp: add persistent flag to block-dirty-bitmap-add
In reply to Vladimir Sementsov-OgievskiyQuoted 38 lines · Vladimir Sementsov-Ogievskiy
Add optional 'persistent' flag to qmp command block-dirty-bitmap-add. Default is false.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- blockdev.c | 12 +++++++++++- qapi/block-core.json | 7 ++++++- qmp-commands.hx | 5 ++++- 3 files changed, 21 insertions(+), 3 deletions(-)blockdev.c +3
-0@@ -1991,6 +1991,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,/* AIO context taken and released within qmp_block_dirty_bitmap_add */ qmp_block_dirty_bitmap_add (action->node, action->name, action->has_granularity, action->granularity, + action->has_persistent, action->persistent, &local_err);if (!local_err) {@@ -2694,10 +2695,12 @@ out:void qmp_block_dirty_bitmap_add (const char *node,const char *name,bool has_granularity,uint32_t granularity, +bool has_persistent,bool persistent,Error **errp) {AioContext *aio_context;BlockDriverState *bs; +BdrvDirtyBitmap *bitmap;if (!name || name[0 ] =='\0' ) {error_setg (errp,"Bitmap name cannot be empty" );@@ -2723,7 +2726,14 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name, granularity =bdrv_get_default_bitmap_granularity (bs);
blockdev.c void qmp_block_dirty_bitmap_add(const char *node, const char *name, · line 2735
Quoted 10 lines
blockdev.c +7
-1} -bdrv_create_dirty_bitmap (bs, granularity, name, errp); +if (!has_persistent) { + persistent = false; + } + + bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); +if (bitmap !=NULL ) { +bdrv_dirty_bitmap_set_persistance (bitmap, persistent);
As I said before, I think this command should be able to return errors and make use of that to reject making bitmaps persistent when the respective block driver cannot handle them.
Quoted 9 lines
blockdev.c +1
-0+ } out:aio_context_release (aio_context);qapi/block-core.json +0
-0
qapi/block-core.json line 1240
Quoted 6 lines
qapi/block-core.json +3
-0@@ -1235,10 +1235,15 @@ # @granularity: #optional the bitmap granularity, default is 64k for # block-dirty-bitmap-add # +# @persistent: #optional the bitmap is persistent, i.e. it will be saved to +# corresponding block device on it's close. Default is false. +# For block-dirty-bitmap-add. (Since 2.8)
I'm not sure what the "For block-dirty-bitmap-add." is supposed to mean, because this whole struct is for block-dirty-bitmap-add (and for block-dirty-bitmap-add transactions, to be exact, but @persistent will surely work there, too, won't it?).
Also, I'd say "will be saved to the corresponding block device image file" instead of just "block device", because in my understanding a block device and its image file are two separate things.
qapi/block-core.json line 1246
Quoted 7 lines
qapi/block-core.json +3
-1+# # Since 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', - 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32' } } + 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', + '*persistent': 'bool' } }
I think normally we'd align the new line so that the opening ' of '*persistent' is under the opening ' of 'node'.
Quoted 17 lines
qapi/block-core.json +0
-0## # @block-dirty-bitmap-addqmp-commands.hx +1
-1@@ -1441,7 +1441,7 @@ EQMP { .name = "block-dirty-bitmap-add", - .args_type = "node:B,name:s,granularity:i?", + .args_type = "node:B,name:s,granularity:i?,persistent:b?", .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add, },
qmp-commands.hx Arguments: · line 1463
Quoted 6 lines
qmp-commands.hx +3
-0@@ -1458,6 +1458,9 @@ Arguments: - "node": device/node on which to create dirty bitmap (json-string) - "name": name of the new dirty bitmap (json-string) - "granularity": granularity to track writes with (int, optional) +- "persistent": bitmap will be saved to corresponding block device + on it's close. Block driver should maintain persistent bitmaps + (json-bool, optional, default false) (Since 2.8)
And I don't know what the user is supposed to make of the information that block drivers will take care of maintaining persistent bitmaps. All they care about is that it will be stored in the corresponding image file, so in my opinion it would be better to just omit the last sentence here.
Max
Quoted 4 lines
qmp-commands.hx +0
-0Example:
Re: [Qemu-devel] [PATCH 16/22] qmp: add persistent flag to block-dirty-bitmap-add
Quoted 2 lines · Max Reitz
On 30.09.2016 12:53, Vladimir Sementsov-Ogievskiy wrote:
[...]
Quoted 20 lines
Quoted 9 lines
blockdev.c +1
-0+ } out:aio_context_release (aio_context);qapi/block-core.json +0
-0qapi/block-core.json line 1240
Quoted 6 lines
qapi/block-core.json +3
-0@@ -1235,10 +1235,15 @@ # @granularity: #optional the bitmap granularity, default is 64k for # block-dirty-bitmap-add # +# @persistent: #optional the bitmap is persistent, i.e. it will be saved to +# corresponding block device on it's close. Default is false. +# For block-dirty-bitmap-add. (Since 2.8)I'm not sure what the "For block-dirty-bitmap-add." is supposed to mean, because this whole struct is for block-dirty-bitmap-add (and for block-dirty-bitmap-add transactions, to be exact, but @persistent will surely work there, too, won't it?).
Quoted 3 lines
Also, I'd say "will be saved to the corresponding block device image file" instead of just "block device", because in my understanding a block device and its image file are two separate things.
Hmm.. But 'its close' is block device close, not file close. And we call common bdrv_* function to save it, so we actually save it to device, and then the device puzzles out, how to actually save it.
Quoted 27 lines
qapi/block-core.json line 1246
Quoted 8 lines
qapi/block-core.json +3
-1+# # Since 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', - 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32' } } + 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', + '*persistent': 'bool' } }I think normally we'd align the new line so that the opening ' of '*persistent' is under the opening ' of 'node'.
Quoted 16 lines
qapi/block-core.json +0
-0## # @block-dirty-bitmap-addqmp-commands.hx +1
-1@@ -1441,7 +1441,7 @@ EQMP { .name = "block-dirty-bitmap-add", - .args_type = "node:B,name:s,granularity:i?", + .args_type = "node:B,name:s,granularity:i?,persistent:b?", .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add, },
Quoted 12 lines
qmp-commands.hx Arguments: · line 1463
Quoted 7 lines
qmp-commands.hx +3
-0@@ -1458,6 +1458,9 @@ Arguments: - "node": device/node on which to create dirty bitmap (json-string) - "name": name of the new dirty bitmap (json-string) - "granularity": granularity to track writes with (int, optional) +- "persistent": bitmap will be saved to corresponding block device + on it's close. Block driver should maintain persistent bitmaps + (json-bool, optional, default false) (Since 2.8)And I don't know what the user is supposed to make of the information that block drivers will take care of maintaining persistent bitmaps. All they care about is that it will be stored in the corresponding image file, so in my opinion it would be better to just omit the last sentence here.
Users shoud know, that only qcow2 supports persistent bitmaps. Instead of last sentence I can write "For now only Qcow2 disks supports persistent bitmaps".
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 16/22] qmp: add persistent flag to block-dirty-bitmap-add
Quoted 27 lines · Vladimir Sementsov-Ogievskiy
Quoted 2 lines · Max Reitz
On 30.09.2016 12:53, Vladimir Sementsov-Ogievskiy wrote:
[...]
Quoted 21 lines
Quoted 12 lines
blockdev.c +1
-0+ } out:aio_context_release (aio_context);qapi/block-core.json +1
-0@@ -1235,10 +1235,15 @@ # @granularity: #optional the bitmap granularity, default is 64k for # block-dirty-bitmap-add # +# @persistent: #optional the bitmap is persistent, i.e. it will beQuoted 4 lines
saved to
+# corresponding block device on it's close. Default is
false.
+# For block-dirty-bitmap-add. (Since 2.8)I'm not sure what the "For block-dirty-bitmap-add." is supposed to mean, because this whole struct is for block-dirty-bitmap-add (and for block-dirty-bitmap-add transactions, to be exact, but @persistent will surely work there, too, won't it?).
Quoted 5 lines
Quoted 3 lines
Also, I'd say "will be saved to the corresponding block device image file" instead of just "block device", because in my understanding a block device and its image file are two separate things.
Hmm.. But 'its close' is block device close, not file close.
In my understanding, it is the file close.
Quoted 3 lines
And we call
common bdrv_* function to save it, so we actually save it to device, and then the device puzzles out, how to actually save it.
Well, OK, it depends on what you mean by "block device". There are many things we call a "block device", but nowadays I think it mostly refers to either a guest block device or a BlockBackend (and as of lately, we're more and more trying to hide the BlockBackend from the user, so you could argue that it's only the guest device now).
The bdrv_* functions operate on block layer BDS nodes, and I don't think we call them "block devices" (at least not any more).
In any case, I think for users the term "block device" refers to either the device presented to the guest or to all of the block layer stuff that's underneath, and it's not quite clear how you could save a bitmap to that, or what it's supposed to mean to "close" a block device (you can remove it, you can destroy it, you can delete it, but "close" it?).
But saying that it will be saved to the image file that is attached to the block device will make it absolutely clear what we mean.
So what you have called a "device" here is neither what I'd call a device (I'd call it a "node" or "BDS"), nor what I think users would call a device. Also, it's not the BDS that puzzles out how to save it but some block driver.
Quoted 37 lines
Quoted 28 lines
qapi/block-core.json line 1246
Quoted 7 lines
qapi/block-core.json +3
-1+# # Since 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', - 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32' } } + 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', + '*persistent': 'bool' } }I think normally we'd align the new line so that the opening ' of '*persistent' is under the opening ' of 'node'.
Quoted 18 lines
### @block-dirty-bitmap-add
qmp-commands.hx +1
-1@@ -1441,7 +1441,7 @@ EQMP { .name = "block-dirty-bitmap-add", - .args_type = "node:B,name:s,granularity:i?", + .args_type = "node:B,name:s,granularity:i?,persistent:b?", .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add, }, @@ -1458,6 +1458,9 @@ Arguments: - "node": device/node on which to create dirty bitmap (json-string)- "name": name of the new dirty bitmap (json-string)
- "granularity": granularity to track writes with (int, optional)
+- "persistent": bitmap will be saved to corresponding block deviceQuoted 8 lines
Quoted 3 lines
+ on it's close. Block driver should maintain
persistent bitmaps
+ (json-bool, optional, default false) (Since 2.8)And I don't know what the user is supposed to make of the information that block drivers will take care of maintaining persistent bitmaps. All they care about is that it will be stored in the corresponding image file, so in my opinion it would be better to just omit the last sentence here.
Quoted 3 lines
Users shoud know, that only qcow2 supports persistent bitmaps. Instead of last sentence I can write "For now only Qcow2 disks supports persistent bitmaps".
s/supports/support/, but yes, that sounds preferable to me.
Max
Re: [Qemu-devel] [PATCH 16/22] qmp: add persistent flag to block-dirty-bitmap-add
Quoted 51 lines · Max Reitz
Quoted 26 lines · Vladimir Sementsov-Ogievskiy
Quoted 2 lines · Max Reitz
On 30.09.2016 12:53, Vladimir Sementsov-Ogievskiy wrote:
[...]
Quoted 21 lines
Quoted 12 lines
blockdev.c +1
-0+ } out:aio_context_release (aio_context);qapi/block-core.json +1
-0@@ -1235,10 +1235,15 @@ # @granularity: #optional the bitmap granularity, default is 64k for # block-dirty-bitmap-add # +# @persistent: #optional the bitmap is persistent, i.e. it will beQuoted 4 lines
saved to
+# corresponding block device on it's close. Default is
false.
+# For block-dirty-bitmap-add. (Since 2.8)I'm not sure what the "For block-dirty-bitmap-add." is supposed to mean, because this whole struct is for block-dirty-bitmap-add (and for block-dirty-bitmap-add transactions, to be exact, but @persistent will surely work there, too, won't it?).
Quoted 4 lines
Quoted 3 lines
Also, I'd say "will be saved to the corresponding block device image file" instead of just "block device", because in my understanding a block device and its image file are two separate things.
Hmm.. But 'its close' is block device close, not file close.
In my understanding, it is the file close.
Quoted 3 lines
And we call
common bdrv_* function to save it, so we actually save it to device, and then the device puzzles out, how to actually save it.Well, OK, it depends on what you mean by "block device". There are many things we call a "block device", but nowadays I think it mostly refers to either a guest block device or a BlockBackend (and as of lately, we're more and more trying to hide the BlockBackend from the user, so you could argue that it's only the guest device now).
The bdrv_* functions operate on block layer BDS nodes, and I don't think we call them "block devices" (at least not any more).
In any case, I think for users the term "block device" refers to either the device presented to the guest or to all of the block layer stuff that's underneath, and it's not quite clear how you could save a bitmap to that, or what it's supposed to mean to "close" a block device (you can remove it, you can destroy it, you can delete it, but "close" it?).
Quoted 8 lines
But saying that it will be saved to the image file that is attached to the block device will make it absolutely clear what we mean.
So what you have called a "device" here is neither what I'd call a device (I'd call it a "node" or "BDS"), nor what I think users would call a device. Also, it's not the BDS that puzzles out how to save it but some block driver.
Ok, thank you.
Quoted 43 lines
Quoted 36 lines
Quoted 28 lines
qapi/block-core.json line 1246
Quoted 7 lines
qapi/block-core.json +3
-1+# # Since 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', - 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32' } } + 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', + '*persistent': 'bool' } }I think normally we'd align the new line so that the opening ' of '*persistent' is under the opening ' of 'node'.
Quoted 18 lines
### @block-dirty-bitmap-add
qmp-commands.hx +1
-1@@ -1441,7 +1441,7 @@ EQMP { .name = "block-dirty-bitmap-add", - .args_type = "node:B,name:s,granularity:i?", + .args_type = "node:B,name:s,granularity:i?,persistent:b?", .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add, }, @@ -1458,6 +1458,9 @@ Arguments: - "node": device/node on which to create dirty bitmap (json-string)- "name": name of the new dirty bitmap (json-string)
- "granularity": granularity to track writes with (int, optional)
+- "persistent": bitmap will be saved to corresponding block deviceQuoted 8 lines
Quoted 3 lines
+ on it's close. Block driver should maintain
persistent bitmaps
+ (json-bool, optional, default false) (Since 2.8)And I don't know what the user is supposed to make of the information that block drivers will take care of maintaining persistent bitmaps. All they care about is that it will be stored in the corresponding image file, so in my opinion it would be better to just omit the last sentence here.
Quoted 3 lines
Users shoud know, that only qcow2 supports persistent bitmaps. Instead of last sentence I can write "For now only Qcow2 disks supports persistent bitmaps".
s/supports/support/, but yes, that sounds preferable to me.
Max
Signature · 2 lines
Best regards, Vladimir
PATCH v7 17/22 qmp: add autoload parameter to block-dirty-bitmap-add
In reply to Vladimir Sementsov-OgievskiyOptional. Default is false.
--- blockdev.c | 22 ++++++++++++++++++++-- qapi/block-core.json | 7 ++++++- qmp-commands.hx | 5 ++++- 3 files changed, 30 insertions(+), 4 deletions(-)
blockdev.c +20 -2
@@ -1992,6 +1992,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,qmp_block_dirty_bitmap_add (action->node, action->name, action->has_granularity, action->granularity, action->has_persistent, action->persistent, + action->has_autoload, action->autoload, &local_err);if (!local_err) {
@@ -2696,6 +2697,7 @@ out:void qmp_block_dirty_bitmap_add (const char *node,const char *name,bool has_granularity,uint32_t granularity,bool has_persistent,bool persistent, +bool has_autoload,bool autoload,Error **errp) {AioContext *aio_context;
@@ -2729,10 +2731,26 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,if (!has_persistent) { persistent = false; } +if (!has_autoload) { + autoload = false; + } + +if (autoload && !persistent) { +error_setg (errp,"Autoload flag must be used only for persistent" +"bitmaps" ); + goto out; + } bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); -if (bitmap !=NULL ) { -bdrv_dirty_bitmap_set_persistance (bitmap, persistent); +if (bitmap ==NULL ) { + goto out; + } + +if (persistent) { +bdrv_dirty_bitmap_set_persistance (bitmap, true); +if (autoload) { +bdrv_dirty_bitmap_set_autoload (bitmap, true); + } } out:
qapi/block-core.json +6 -1
@@ -1239,11 +1239,16 @@ # corresponding block device on it's close. Default is false. # For block-dirty-bitmap-add. (Since 2.8) # +# @autoload: #optional the bitmap will be autoloaded on it's storage image +# open. This flag is only for persistent bitmap and needed to inform +# block driver that bitmap should be autoloaded on the next image +# open. Default is false. For block-dirty-bitmap-add. (Since 2.8) +# # Since 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', - '*persistent': 'bool' } } + '*persistent': 'bool', '*autoload': 'bool' } } ## # @block-dirty-bitmap-add
qmp-commands.hx +4 -1
@@ -1441,7 +1441,7 @@ EQMP { .name = "block-dirty-bitmap-add", - .args_type = "node:B,name:s,granularity:i?,persistent:b?", + .args_type = "node:B,name:s,granularity:i?,persistent:b?,autoload:b?", .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add, },
@@ -1461,6 +1461,9 @@ Arguments: - "persistent": bitmap will be saved to corresponding block device on it's close. Block driver should maintain persistent bitmaps (json-bool, optional, default false) (Since 2.8) +- "autoload": only for persistent bitmaps. Bitmap will be autoloaded on it's + storage image open. (json-bool, optional, default false) + (Since 2.8) Example:
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 17/22] qmp: add autoload parameter to block-dirty-bitmap-add
Quoted 12 lines · Vladimir Sementsov-Ogievskiy
Optional. Default is false.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- blockdev.c | 22 ++++++++++++++++++++-- qapi/block-core.json | 7 ++++++- qmp-commands.hx | 5 ++++- 3 files changed, 30 insertions(+), 4 deletions(-)+++
b/qapi/block-core.json
qapi/block-core.json line 1242
Quoted 4 lines
qapi/block-core.json +1
-0@@ -1239,11 +1239,16 @@ # corresponding block device on it's close. Default is false. # For block-dirty-bitmap-add. (Since 2.8) # +# @autoload: #optional the bitmap will be autoloaded on it's storage image
s/it's/its/
qapi/block-core.json line 1243
Quoted 1 line
qapi/block-core.json +1
-0+# open. This flag is only for persistent bitmap and needed to inform
s/bitmap/bitmaps,/
s/needed/is needed/
qapi/block-core.json line 1244
Quoted 1 line
qapi/block-core.json +1
-0+# block driver that bitmap should be autoloaded on the next image
s/block/the block/
s/bitmap/the bitmap/
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 17/22] qmp: add autoload parameter to block-dirty-bitmap-add
In reply to Vladimir Sementsov-OgievskiyQuoted 9 lines · Vladimir Sementsov-Ogievskiy
Optional. Default is false.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- blockdev.c | 22 ++++++++++++++++++++-- qapi/block-core.json | 7 ++++++- qmp-commands.hx | 5 ++++- 3 files changed, 30 insertions(+), 4 deletions(-)
Design question: I see that being able to specify these flags when creating bitmaps is useful. However, would about a way for the user to change these flags on an existing dirty bitmap? Would you consider that useful?
(Of course, if so, it can always be added later, we don't need it now.)
Quoted 21 lines
blockdev.c +2
-0@@ -1992,6 +1992,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,qmp_block_dirty_bitmap_add (action->node, action->name, action->has_granularity, action->granularity, action->has_persistent, action->persistent, + action->has_autoload, action->autoload, &local_err);if (!local_err) {@@ -2696,6 +2697,7 @@ out:void qmp_block_dirty_bitmap_add (const char *node,const char *name,bool has_granularity,uint32_t granularity,bool has_persistent,bool persistent, +bool has_autoload,bool autoload,Error **errp) {AioContext *aio_context;
blockdev.c void qmp_block_dirty_bitmap_add(const char *node, const char *name, · line 2740
Quoted 10 lines
blockdev.c +7
-0@@ -2729,10 +2731,26 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,if (!has_persistent) { persistent = false; } +if (!has_autoload) { + autoload = false; + } + +if (autoload && !persistent) { +error_setg (errp,"Autoload flag must be used only for persistent" +"bitmaps" );
Missing space between "persistent" and "bitmaps".
Also, technically I think you should throw this error if has_autoload is true instead of autoload. I would consider it wrong if a user specified autoload at all, even autoload=false, without setting persistent=true. But if you disagree, then please keep the condition the way it is.
Quoted 5 lines
blockdev.c +2
-1+ goto out; + } bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); -if (bitmap !=NULL ) {
blockdev.c void qmp_block_dirty_bitmap_add(const char *node, const char *name, · line 2753
Quoted 10 lines
blockdev.c +9
-1-bdrv_dirty_bitmap_set_persistance (bitmap, persistent); +if (bitmap ==NULL ) { + goto out; + } + +if (persistent) { +bdrv_dirty_bitmap_set_persistance (bitmap, true); +if (autoload) { +bdrv_dirty_bitmap_set_autoload (bitmap, true); + }
By the way, a simpler way to do the same would be just
bdrv_dirty_bitmap_set_persistance(bitmap, persistent); bdrv_dirty_bitmap_set_autoload(bitmap, autoload);
But if you prefer this explicit style, that's fine, too.
qapi/block-core.json line 1243
Quoted 13 lines
blockdev.c +0
-0} out:qapi/block-core.json +2
-0@@ -1239,11 +1239,16 @@ # corresponding block device on it's close. Default is false. # For block-dirty-bitmap-add. (Since 2.8) # +# @autoload: #optional the bitmap will be autoloaded on it's storage image +# open.
I'd rephrase this like "The bitmap will be automatically loaded when the image it is stored in is opened." (Or keep "autoloaded" instead of "automatically loaded", that doesn't really matter.)
qapi/block-core.json line 1246
Quoted 3 lines
qapi/block-core.json +2
-0This flag is only for persistent bitmap and needed to inform +# block driver that bitmap should be autoloaded on the next image +# open.
Again, the user doesn't really have to care how the internals work. All they need to know is that the image will be automatically loaded when the image is opened the next time, and that is something the first sentence told them already.
(Of course, "This flag may only be specified for persistent bitmaps" should stay.)
qapi/block-core.json line 1247
Quoted 1 line
qapi/block-core.json +0
-0Default is false. For block-dirty-bitmap-add. (Since 2.8)
Again, I don't see the point of the last sentence.
Max
Quoted 34 lines
qapi/block-core.json +2
-1+# # Since 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', - '*persistent': 'bool' } } + '*persistent': 'bool', '*autoload': 'bool' } }##
# @block-dirty-bitmap-addqmp-commands.hx +4
-1@@ -1441,7 +1441,7 @@ EQMP { .name = "block-dirty-bitmap-add", - .args_type = "node:B,name:s,granularity:i?,persistent:b?", + .args_type = "node:B,name:s,granularity:i?,persistent:b?,autoload:b?", .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add, },@@ -1461,6 +1461,9 @@ Arguments: - "persistent": bitmap will be saved to corresponding block device on it's close. Block driver should maintain persistent bitmaps (json-bool, optional, default false) (Since 2.8) +- "autoload": only for persistent bitmaps. Bitmap will be autoloaded on it's + storage image open. (json-bool, optional, default false) + (Since 2.8) Example:
Re: [Qemu-devel] [PATCH 17/22] qmp: add autoload parameter to block-dirty-bitmap-add
Quoted 10 lines · Max Reitz
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
Optional. Default is false.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- blockdev.c | 22 ++++++++++++++++++++-- qapi/block-core.json | 7 ++++++- qmp-commands.hx | 5 ++++- 3 files changed, 30 insertions(+), 4 deletions(-)
Quoted 6 lines
Design question: I see that being able to specify these flags when creating bitmaps is useful. However, would about a way for the user to change these flags on an existing dirty bitmap? Would you consider that useful?
(Of course, if so, it can always be added later, we don't need it now.)
I vote for adding later, if needed. For now we don't have such scenarios.
Signature · 2 lines
Best regards, Vladimir
PATCH v7 18/22 qapi: add md5 checksum of last dirty bitmap level to query-block
In reply to Vladimir Sementsov-Ogievskiy--- block/dirty-bitmap.c | 1 + include/qemu/hbitmap.h | 8 ++++++++ qapi/block-core.json | 5 ++++- util/hbitmap.c | 8 ++++++++ 4 files changed, 21 insertions(+), 1 deletion(-)
block/dirty-bitmap.c +1 -0
@@ -359,6 +359,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs) info->has_name = !!bm->name; info->name =g_strdup (bm->name); info->status =bdrv_dirty_bitmap_status (bm); + info->md5 =hbitmap_md5 (bm->bitmap); entry->value = info; *plist = entry; plist = &entry->next;
include/qemu/hbitmap.h +8 -0
@@ -240,6 +240,14 @@ void hbitmap_deserialize_ones(HBitmap *hb, uint64_t start, uint64_t count,void hbitmap_deserialize_finish (HBitmap *hb);/** +* hbitmap_md5: +* @bitmap: HBitmap to operate on. +* +* Returns md5 checksum of the last level. +*/ +char *hbitmap_md5 (const HBitmap *bitmap); + +/** * hbitmap_free: * @hb: HBitmap to operate on. *
qapi/block-core.json +4 -1
@@ -419,11 +419,14 @@ # # @status: current status of the dirty bitmap (since 2.4) # +# @md5: md5 checksum (as a hexadecimal string) of the last bitmap level +# (since 2.8) +# # Since: 1.3 ## { 'struct': 'BlockDirtyInfo', 'data': {'*name': 'str', 'count': 'int', 'granularity': 'uint32', - 'status': 'DirtyBitmapStatus'} } + 'status': 'DirtyBitmapStatus', 'md5': 'str'} } ## # @BlockInfo:
util/hbitmap.c +8 -0
@@ -707,3 +707,11 @@ void hbitmap_free_meta(HBitmap *hb)hbitmap_free (hb->meta); hb->meta =NULL ; } + +char *hbitmap_md5 (const HBitmap *bitmap) +{ +uint64_t size = +MAX ((bitmap->size +BITS_PER_LONG -1 ) >>BITS_PER_LEVEL ,1 ); +const guchar *data = (const guchar *)bitmap->levels[HBITMAP_LEVELS -1 ]; +return g_compute_checksum_for_data (G_CHECKSUM_MD5 , data, size); +}
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 18/22] qapi: add md5 checksum of last dirty bitmap level to query-block
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
Reviewed-by John Snow <[email protected]>Reviewed-by Eric Blake <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/dirty-bitmap.c | 1 + include/qemu/hbitmap.h | 8 ++++++++ qapi/block-core.json | 5 ++++- util/hbitmap.c | 8 ++++++++ 4 files changed, 21 insertions(+), 1 deletion(-)
Having read John's and Eric's comments, I won't block this patch, but I won't give an R-b either.
It's probably true that this will not significantly slow down the query-block call, but doing this only for debugging does not seem right to me.
I'm not sure what the right way would be to get this information out (...maybe make it optional and set it only if qtest_enabled() is true?), but in my opinion this is not the right way.
Since I'm not the maintainer of the bitmap code (Fam and John are, even though their MAINTAINERS patch is not in master still...), I can't and won't block this, though.
Max
Re: [Qemu-devel] [PATCH 18/22] qapi: add md5 checksum of last dirty bitmap level to query-block
Quoted 14 lines · Max Reitz
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
Reviewed-by John Snow <[email protected]>Reviewed-by Eric Blake <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/dirty-bitmap.c | 1 + include/qemu/hbitmap.h | 8 ++++++++ qapi/block-core.json | 5 ++++- util/hbitmap.c | 8 ++++++++ 4 files changed, 21 insertions(+), 1 deletion(-)Having read John's and Eric's comments, I won't block this patch, but I won't give an R-b either.
Quoted 7 lines
It's probably true that this will not significantly slow down the query-block call, but doing this only for debugging does not seem right to me.
I'm not sure what the right way would be to get this information out (...maybe make it optional and set it only if
qtest_enabled()is true?), but in my opinion this is not the right way.
By the way, the cleanest way I can come up with (which I didn't write about in my first reply because it's not so trivial) would be some kind of debugging QMP command convention. For instance, we could say that all debugging commands have an x-debug- prefix, and then you could add an x-debug-get-bitmap-md5 to read the MD5 hash of a named bitmap. That would appear to be the cleanest way to do this to me.
Max
Quoted 6 lines
Since I'm not the maintainer of the bitmap code (Fam and John are, even though their
MAINTAINERSpatch is not in master still...), I can't and won't block this, though.Max
Re: [Qemu-devel] [PATCH 18/22] qapi: add md5 checksum of last dirty bitmap level to query-block
Quoted 10 lines · Max Reitz
Quoted 3 lines
I'm not sure what the right way would be to get this information out (...maybe make it optional and set it only if
qtest_enabled()is true?), but in my opinion this is not the right way.By the way, the cleanest way I can come up with (which I didn't write about in my first reply because it's not so trivial) would be some kind of debugging QMP command convention. For instance, we could say that all debugging commands have an x-debug- prefix, and then you could add an x-debug-get-bitmap-md5 to read the MD5 hash of a named bitmap. That would appear to be the cleanest way to do this to me.
Anything named x- is automatically not stable, and therefore free to use in the testsuite without having to worry about keeping it backwards compatible (libvirt won't touch x- commands). The suggestion of x-debug- to mark it as specifically for debug use is reasonable.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v7 19/22 iotests: test qcow2 persistent dirty bitmap
In reply to Vladimir Sementsov-Ogievskiy--- tests/qemu-iotests/165 | 87 ++++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/165.out | 5 +++ tests/qemu-iotests/group | 1 + 3 files changed, 93 insertions(+) create mode 100755 tests/qemu-iotests/165 create mode 100644 tests/qemu-iotests/165.out
tests/qemu-iotests/165 new file +87 -0
@@ -0,0 +1,87 @@ +#!/usr/bin/env python +# +# Tests for persistent dirty bitmaps. +# +# Copyright: Vladimir Sementsov-Ogievskiy 2015-2016 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import iotests +from iotests import qemu_img + +disk = os.path.join(iotests.test_dir, 'disk') +disk_size = 0x40000000 # 1G + +# regions for qemu_io: (start, count) in bytes +regions1 = ((0, 0x100000), + (0x200000, 0x100000)) + +regions2 = ((0x10000000, 0x20000), + (0x39990000, 0x10000)) + +class TestPersistentDirtyBitmap(iotests.QMPTestCase): + + def setUp(self): + qemu_img('create', '-f', iotests.imgfmt, disk, str(disk_size)) + + def tearDown(self): + os.remove(disk) + + def mkVm(self): + return iotests.VM().add_drive(disk) + + def getMd5(self): + result = self.vm.qmp('query-block'); + return result['return'][0]['dirty-bitmaps'][0]['md5'] + + def checkBitmap(self, md5): + result = self.vm.qmp('query-block'); + self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5); + + def writeRegions(self, regions): + for r in regions: + self.vm.hmp_qemu_io('drive0', + 'write %d %d' % r) + + def qmpAddBitmap(self): + self.vm.qmp('block-dirty-bitmap-add', node='drive0', + name='bitmap0', persistent=True, autoload=True) + + def test_persistent(self): + self.vm = self.mkVm() + self.vm.launch() + self.qmpAddBitmap() + + self.writeRegions(regions1) + md5 = self.getMd5() + + self.vm.shutdown() + self.vm = self.mkVm() + self.vm.launch() + + self.checkBitmap(md5) + self.writeRegions(regions2) + md5 = self.getMd5() + + self.vm.shutdown() + self.vm.launch() + + self.checkBitmap(md5) + + self.vm.shutdown() + +if __name__ == '__main__': + iotests.main(supported_fmts=['qcow2'])
tests/qemu-iotests/165.out new file +5 -0
@@ -0,0 +1,5 @@ +. +---------------------------------------------------------------------- +Ran 1 tests + +OK
tests/qemu-iotests/group +1 -0
@@ -161,3 +161,4 @@ 159 rw auto quick 160 rw auto quick 162 auto quick +165 rw auto quick
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 19/22] iotests: test qcow2 persistent dirty bitmap
Quoted 38 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/165 | 87 ++++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/165.out | 5 +++ tests/qemu-iotests/group | 1 + 3 files changed, 93 insertions(+) create mode 100755 tests/qemu-iotests/165 create mode 100644 tests/qemu-iotests/165.outtests/qemu-iotests/165 new file +23
-0@@ -0,0 +1,87 @@ +#!/usr/bin/env python +# +# Tests for persistent dirty bitmaps. +# +# Copyright: Vladimir Sementsov-Ogievskiy 2015-2016 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import iotests +from iotests import qemu_img
tests/qemu-iotests/165 line 33
Quoted 10 lines
tests/qemu-iotests/165 new file +10
-0+ +disk = os.path.join(iotests.test_dir, 'disk') +disk_size = 0x40000000 # 1G + +# regions for qemu_io: (start, count) in bytes +regions1 = ((0, 0x100000), + (0x200000, 0x100000)) + +regions2 = ((0x10000000, 0x20000), + (0x39990000, 0x10000))
Did you mean 0x3fff0000?
Looks good apart from that (and the fact that I don't like the @md5 field in query-block too much, but that's a matter for patch 18).
Max
Quoted 75 lines
tests/qemu-iotests/165 new file +54
-0+ +class TestPersistentDirtyBitmap(iotests.QMPTestCase): + + def setUp(self): + qemu_img('create', '-f', iotests.imgfmt, disk, str(disk_size)) + + def tearDown(self): + os.remove(disk) + + def mkVm(self): + return iotests.VM().add_drive(disk) + + def getMd5(self): + result = self.vm.qmp('query-block'); + return result['return'][0]['dirty-bitmaps'][0]['md5'] + + def checkBitmap(self, md5): + result = self.vm.qmp('query-block'); + self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5); + + def writeRegions(self, regions): + for r in regions: + self.vm.hmp_qemu_io('drive0', + 'write %d %d' % r) + + def qmpAddBitmap(self): + self.vm.qmp('block-dirty-bitmap-add', node='drive0', + name='bitmap0', persistent=True, autoload=True) + + def test_persistent(self): + self.vm = self.mkVm() + self.vm.launch() + self.qmpAddBitmap() + + self.writeRegions(regions1) + md5 = self.getMd5() + + self.vm.shutdown() + self.vm = self.mkVm() + self.vm.launch() + + self.checkBitmap(md5) + self.writeRegions(regions2) + md5 = self.getMd5() + + self.vm.shutdown() + self.vm.launch() + + self.checkBitmap(md5) + + self.vm.shutdown() + +if __name__ == '__main__': + iotests.main(supported_fmts=['qcow2'])tests/qemu-iotests/165.out new file +5
-0@@ -0,0 +1,5 @@ +. +---------------------------------------------------------------------- +Ran 1 tests + +OKtests/qemu-iotests/group +1
-0@@ -161,3 +161,4 @@ 159 rw auto quick 160 rw auto quick 162 auto quick +165 rw auto quick
PATCH v7 20/22 qcow2-dirty-bitmap: refcounts
In reply to Vladimir Sementsov-OgievskiyCalculate refcounts for dirty bitmaps.
--- block/qcow2-bitmap.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2-refcount.c | 14 +++++++----- block/qcow2.h | 5 +++++ 3 files changed, 74 insertions(+), 5 deletions(-)
block/qcow2-bitmap.c +60 -0
@@ -901,3 +901,63 @@ out:g_free (new_dir);g_free (dir); } + +int qcow2_dirty_bitmaps_refcounts (BlockDriverState *bs, +BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size) +{ +BDRVQcow2State *s = bs->opaque; +uint8_t *dir; +Qcow2BitmapDirEntry *e; +Error *local_err =NULL ; + +int i; +int ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, + s->bitmap_directory_offset, + s->bitmap_directory_size); +if (ret <0 ) { +return ret; + } + + dir =directory_read (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, &local_err); +if (dir ==NULL ) { +error_report_err (local_err); +return -EINVAL ; + } + +for_each_bitmap_dir_entry (e, dir, s->bitmap_directory_size) { +uint64_t *bitmap_table =NULL ; + + ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, + e->bitmap_table_offset, + e->bitmap_table_size); +if (ret <0 ) { +return ret; + } + + ret =bitmap_table_load (bs, e, &bitmap_table); +if (ret <0 ) { +return ret; + } + +for (i =0 ; i < e->bitmap_table_size; ++i) { +uint64_t off =be64_to_cpu (bitmap_table[i]); +if (off <=1 ) { +continue ; + } + + ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, + off, s->cluster_size); +if (ret <0 ) { +g_free (bitmap_table); +return ret; + } + } + +g_free (bitmap_table); + } + +return 0 ; +}
block/qcow2-refcount.c +9 -5
@@ -1309,11 +1309,9 @@ static int realloc_refcount_array(BDRVQcow2State *s, void **array,* * Modifies the number of errors in res. */ -static int inc_refcounts (BlockDriverState *bs, -BdrvCheckResult *res, -void **refcount_table, -int64_t *refcount_table_size, -int64_t offset,int64_t size) +int inc_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table,int64_t *refcount_table_size, +int64_t offset,int64_t size) {BDRVQcow2State *s = bs->opaque;uint64_t start, last, cluster_offset, k, refcount;
@@ -1843,6 +1841,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,return ret; } +/* dirty bitmaps */ + ret =qcow2_dirty_bitmaps_refcounts (bs, res, refcount_table, nb_clusters); +if (ret <0 ) { +return ret; + } +return check_refblocks (bs, res, fix, rebuild, refcount_table, nb_clusters); }
block/qcow2.h +5 -0
@@ -563,6 +563,9 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, int64_t size);int qcow2_pre_write_overlap_check (BlockDriverState *bs,int ign,int64_t offset,int64_t size); +int inc_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table,int64_t *refcount_table_size, +int64_t offset,int64_t size); int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order, BlockDriverAmendStatusCB *status_cb,
@@ -616,6 +619,8 @@ int qcow2_read_snapshots(BlockDriverState *bs);int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp);int qcow2_delete_bitmaps (BlockDriverState *bs); +int qcow2_dirty_bitmaps_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table,int64_t *refcount_table_size);/* qcow2-cache.c functions */ Qcow2Cache *qcow2_cache_create (BlockDriverState *bs,int num_tables);
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 20/22] qcow2-dirty-bitmap: refcounts
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Calculate refcounts for dirty bitmaps.
The commit message should mention that this is for qcow2's qemu-img check implementation.
Quoted 13 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2-refcount.c | 14 +++++++----- block/qcow2.h | 5 +++++ 3 files changed, 74 insertions(+), 5 deletions(-)block/qcow2-bitmap.c +0
-0
block/qcow2-bitmap.c out: · line 908
Quoted 8 lines
block/qcow2-bitmap.c +5
-0@@ -901,3 +901,63 @@ out:g_free (new_dir);g_free (dir); } + +int qcow2_dirty_bitmaps_refcounts (BlockDriverState *bs, +BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size)
I'd rename this function to make clear that this is for checking the refcounts, e.g. to "qcow2_check_dirty_bitmaps_refcounts" or "qcow2_count_dirty_bitmaps_refcounts" or just "qcow2_check_dirty_bitmaps". Probably the last one is the best because this function should ideally perform a full check of the dirty bitmaps extension.
Quoted 8 lines
block/qcow2-bitmap.c +8
-0+{ +BDRVQcow2State *s = bs->opaque; +uint8_t *dir; +Qcow2BitmapDirEntry *e; +Error *local_err =NULL ; + +int i; + int ret = inc_refcounts(bs, res, refcount_table, refcount_table_size,
block/qcow2-bitmap.c out: · line 926
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ s->bitmap_directory_offset, + s->bitmap_directory_size); +if (ret <0 ) { +return ret; + } + + dir =directory_read (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, &local_err); +if (dir ==NULL ) { +error_report_err (local_err);
I think you should increment res->corruptions here.
block/qcow2-bitmap.c out: · line 931
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+return -EINVAL ; + } + +for_each_bitmap_dir_entry (e, dir, s->bitmap_directory_size) { +uint64_t *bitmap_table =NULL ;
I think you should call check_constraints() here.
block/qcow2-bitmap.c out: · line 935
Quoted 4 lines
block/qcow2-bitmap.c +4
-0+ + ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, + e->bitmap_table_offset, + e->bitmap_table_size);
Probably rather e->bitmap_table_size * sizeof(uint64_t).
block/qcow2-bitmap.c out: · line 941
Quoted 6 lines
block/qcow2-bitmap.c +6
-0+if (ret <0 ) { +return ret; + } + + ret =bitmap_table_load (bs, e, &bitmap_table); +if (ret <0 ) {
Again, it would make sense to increment res->corruptions here.
block/qcow2-bitmap.c out: · line 949
Quoted 8 lines
block/qcow2-bitmap.c +8
-0+return ret; + } + +for (i =0 ; i < e->bitmap_table_size; ++i) { +uint64_t off =be64_to_cpu (bitmap_table[i]); +if (off <=1 ) { +continue ; + }
As I said in some other patch, I'd write this condition differently (with an offset mask, etc.).
Also, you should check that the offset is aligned to a cluster boundary and that no unknown flags are set.
block/qcow2-bitmap.c out: · line 962
Quoted 13 lines
block/qcow2-bitmap.c +13
-0+ + ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, + off, s->cluster_size); +if (ret <0 ) { +g_free (bitmap_table); +return ret; + } + } + +g_free (bitmap_table); + } + +return 0 ;
dir is leaked here.
Quoted 7 lines
block/qcow2-bitmap.c +1
-0+}block/qcow2-refcount.c +0
-0@@ -1309,11 +1309,9 @@ static int realloc_refcount_array(BDRVQcow2State *s, void **array,*
block/qcow2-refcount.c static int realloc_refcount_array(BDRVQcow2State *s, void **array, · line 1314
Quoted 10 lines
block/qcow2-refcount.c +3
-5* Modifies the number of errors in res. */ -static int inc_refcounts (BlockDriverState *bs, -BdrvCheckResult *res, -void **refcount_table, -int64_t *refcount_table_size, -int64_t offset,int64_t size) +int inc_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table,int64_t *refcount_table_size, +int64_t offset,int64_t size)
First, if you make this function public, you have to give it a qcow2_ prefix.
Second, if this function is public, it should have a name that makes sense. inc_refcounts() sounds as if it's the same as update_refcount() with an addend of 1. I'd rename it qcow2_inc_refcounts_imrt(), because that's probably the shortest name I can come up with (and qcow2-refcount.c explains what an IMRT is in some comment).
Quoted 31 lines
block/qcow2-refcount.c +6
-0{BDRVQcow2State *s = bs->opaque;uint64_t start, last, cluster_offset, k, refcount;@@ -1843,6 +1841,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,return ret; } +/* dirty bitmaps */ + ret =qcow2_dirty_bitmaps_refcounts (bs, res, refcount_table, nb_clusters); +if (ret <0 ) { +return ret; + } +return check_refblocks (bs, res, fix, rebuild, refcount_table, nb_clusters); }block/qcow2.h +3
-0@@ -563,6 +563,9 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, int64_t size);int qcow2_pre_write_overlap_check (BlockDriverState *bs,int ign,int64_t offset,int64_t size); +int inc_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table,int64_t *refcount_table_size, +int64_t offset,int64_t size); int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order, BlockDriverAmendStatusCB *status_cb,
block/qcow2.h int qcow2_read_snapshots(BlockDriverState *bs); · line 623
Quoted 5 lines
block/qcow2.h +2
-0@@ -616,6 +619,8 @@ int qcow2_read_snapshots(BlockDriverState *bs);int qcow2_read_bitmaps (BlockDriverState *bs,Error **errp);int qcow2_delete_bitmaps (BlockDriverState *bs); +int qcow2_dirty_bitmaps_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table,int64_t *refcount_table_size);
Normally we try to align parameters along the opening parenthesis as long as there is enough space, and there is enough space here.
Max
Quoted 4 lines
block/qcow2.h +0
-0/* qcow2-cache.c functions */ Qcow2Cache *qcow2_cache_create (BlockDriverState *bs,int num_tables);
PATCH v7 21/22 specs/qcow2: fix bitmap granularity qemu-specific note
In reply to Vladimir Sementsov-Ogievskiy--- docs/specs/qcow2.txt | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
docs/specs/qcow2.txt +1 -2
@@ -472,8 +472,7 @@ Structure of a bitmap directory entry: 17: granularity_bits Granularity bits. Valid values: 0 - 63. - Note: Qemu currently doesn't support granularity_bits - greater than 31. + Note: Qemu currently support only values 9 - 31. Granularity is calculated as granularity = 1 << granularity_bits
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 21/22] specs/qcow2: fix bitmap granularity qemu-specific note
docs/specs/qcow2.txt Structure of a bitmap directory entry: · line 473
Quoted 12 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- docs/specs/qcow2.txt | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)docs/specs/qcow2.txt +0
-0@@ -472,8 +472,7 @@ Structure of a bitmap directory entry: 17: granularity_bits Granularity bits. Valid values: 0 - 63.
Can we really theoretically go as low as 0, or are we constrained by the fact that cluster_bits must be at least 9?
docs/specs/qcow2.txt Structure of a bitmap directory entry: · line 475
Quoted 4 lines
docs/specs/qcow2.txt +1
-2- Note: Qemu currently doesn't support granularity_bits - greater than 31. + Note: Qemu currently support only values 9 - 31.
s/support/supports/
Quoted 4 lines
docs/specs/qcow2.txt +0
-0Granularity is calculated as granularity = 1 << granularity_bits
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 21/22] specs/qcow2: fix bitmap granularity qemu-specific note
Quoted 6 lines · Eric Blake
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- docs/specs/qcow2.txt | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
Quoted 9 lines
docs/specs/qcow2.txt Structure of a bitmap directory entry: · line 473
Quoted 7 lines
docs/specs/qcow2.txt +0
-0@@ -472,8 +472,7 @@ Structure of a bitmap directory entry: 17: granularity_bits Granularity bits. Valid values: 0 - 63.Can we really theoretically go as low as 0, or are we constrained by the fact that
cluster_bitsmust be at least 9?
Bitmaps in qcow2 are abstract, they are not necessary related to BdrvDirtyBitmaps.. We do not know how the will be used in future, so there no reasons for hard restriction. However values near zero seems not very useful. I do not think we should change this. Note about Qemu is enough.
Quoted 11 lines
docs/specs/qcow2.txt Structure of a bitmap directory entry: · line 475
Quoted 5 lines
docs/specs/qcow2.txt +1
-2- Note: Qemu currently doesn't support granularity_bits - greater than 31. + Note: Qemu currently support only values 9 - 31.s/support/supports/
Quoted 4 lines
docs/specs/qcow2.txt +0
-0Granularity is calculated as granularity = 1 << granularity_bits
Signature · 2 lines
Best regards, Vladimir
PATCH v7 22/22 specs/qcow2: do not use wording 'bitmap header'
In reply to Vladimir Sementsov-OgievskiyFor now bitmap directory entry some times called 'bitmap header'. This patch leaves only one name - 'bitmap directory entry'. The name 'bitmap header' creates misunderstandings with 'qcow2 header' and 'qcow2 bitmap header extension' (which is extension of qcow2 header)
--- docs/specs/qcow2.txt | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
docs/specs/qcow2.txt +2 -3
@@ -201,7 +201,7 @@ The fields of the bitmaps extension are: 8 - 15: bitmap_directory_size Size of the bitmap directory in bytes. It is the cumulative - size of all (nb_bitmaps) bitmap headers. + size of all (nb_bitmaps) bitmap directory entries. 16 - 23: bitmap_directory_offset Offset into the image file at which the bitmap directory
@@ -426,8 +426,7 @@ Each bitmap saved in the image is described in a bitmap directory entry. The bitmap directory is a contiguous area in the image file, whose starting offset and length are given by the header extension fields bitmap_directory_offset and bitmap_directory_size. The entries of the bitmap directory have variable -length, depending on the lengths of the bitmap name and extra data. These -entries are also called bitmap headers. +length, depending on the lengths of the bitmap name and extra data. Structure of a bitmap directory entry:
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 22/22] specs/qcow2: do not use wording 'bitmap header'
Quoted 1 line · Vladimir Sementsov-Ogievskiy
For now bitmap directory entry some times called 'bitmap header'. This
Awkward wording; maybe:
A bitmap directory entry is sometimes called a 'bitmap header'.
Quoted 4 lines
patch leaves only one name - 'bitmap directory entry'. The name 'bitmap header' creates misunderstandings with 'qcow2 header' and 'qcow2 bitmap header extension' (which is extension of qcow2 header)
Doc touchup patches like this are fairly uncontroversial, and could be moved earlier in the series so that they can be merged rather than having to be rebased multiple times.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
v7:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v7
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
It should be noted that (at least my) block-next is only valid during freeze, after that it becomes stale. I assume the patches you require from my block-next branch are the ones from the "Dirty bitmap changes for migration/persistence work" series, which I had to drop from my pull requests, however, due to some issues on Big Endian machines.
The only reason they are still in my block-next branch is, as I said, that that branch is just stale. I won't overwrite it for the time being, though, so this series can still be applied on top.
Max
Quoted 11 lines · Max Reitz
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
v7:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v7
based on block-next (https://github.com/XanClic/qemu/commits/block-next)It should be noted that (at least my) block-next is only valid during freeze, after that it becomes stale. I assume the patches you require from my block-next branch are the ones from the "Dirty bitmap changes for migration/persistence work" series, which I had to drop from my pull requests, however, due to some issues on Big Endian machines.
Quoted 6 lines
The only reason they are still in my block-next branch is, as I said, that that branch is just stale. I won't overwrite it for the time being, though, so this series can still be applied on top.
Max
Lemme fix up the 32bit problems and resend that out right now. (At least it's not lock correctness.)
--js