qcow2: persistent dirty bitmaps
v14 cover qcow2: persistent dirty bitmaps
Hi all!
There is a new update of qcow2-bitmap series - v14.
web: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=qcow2-bitmap-v14
git: https://src.openvz.org/scm/~vsementsov/qemu.git (tag qcow2-bitmap-v14)
v14:
07: use '|=' to update need_update_header
add John's r-b
add Max's r-b
09: remove unused bitmap_table_to_cpu()
left Max's r-b, hope it's ok
add John's r-b
10: remove extra new line
add John's r-b
11: add John's r-b
12: add John's r-b
13: small fixes by John's review:
- remove weird g_free of NULL pointer from
if (tb == NULL) {
g_free(tb);
return -EINVAL;
}
- remove extra comment "/* errp is already set */"
- s/"Too large bitmap directory"/"Bitmap directory is too large"/
left Max's r-b, hope you don't mind
22: add Max's r-b
23: add Max's r-b
24: add Max's r-b
25: new patch to improve error message on check_constraints_on_bitmap fail
v13: Just a fix for style checker.
13: line over 80
14: line over 80
22: s/if () \n{/if () {/
v12:
07: do not update header in qcow2_read_extensions, instead do it in the
end of qcow2_open, where it is updated also to clear unknown autoclear features.
Wrong bdrv_is_root_node used instead of bdrv_is_read_only is fixed automatically.
08: add Max's r-b
09: contextual: s/raw_bsd/raw-format/
qcow2-bitmap.c: copyright s/2016/2017/
fix compilation: move update_header_sync() to this patch
add Max's r-b
13: update_header_sync() is moved to 09
add Max's r-b
15: add Max's r-b
16: As qmp-commands.txt is removed from master, remove it here too.
Sentence "For now only Qcow2 disks support persistent bitmaps.
Default is false." moved to qapi/block-core.json.
Hope that is OK, Max's r-b is not dropped.
17: qmp-commands.txt deleted, r-b is not dropped too.
18: s/is failed/has failed/, add Max's r-b
19: copyright: s/2016/2017/
21: drop "res->corruptions > 0 => ret = -EINVAL", add Max's r-b
22: actually, patch is replaced with a new one, however some parts are the same.
instead of changing bdrv_release_dirty_bitmap behaviour, create new bdrv_remove_persistent_dirty_bitmap
23: improve comment, about not-exists is not an error
24: new patch
Old 24 (qcow2-bitmap: cache bitmap list in BDRVQcow2State) will be sent separately, to be applied after these series.
Also: about bdrv_dirty_bitmap_set_persistance: I've decided not change its behaviour for now and just call bdrv_remove_persistent_dirty_bitmap from qmp_block_dirty_bitmap_remove. Reasons:
1. Do not change reviewed part.
2. I'm not sure, that this complication of BdrvDirtyBitmap.persistent semantics is good (current semantics are just: .persistent means that bitmap should be saved on disk close). .persistent actually is not very related to "is there stored version of the bitmap in the image".
3. It may be done later.
4. It is not actually needed for now, as the only usage is dropping bitmap in bdrv_remove_persistent_dirty_bitmap. May be in future it will be needed, when we will have qmp interfaces for finer control of persistent dirty bitmaps.
v11:
Fix automatic build test fail.
18: wording - "ASCII representation of SHA256 ..."
24: fix redeclaration error. (This is still RFC, may be not needed patch,
see description in v10 below).
v10:
07: rm John's r-b
not add Max's r-b
fix subject s/dirty bitmaps/bitmaps
improve WARNING message
remove header ext if autoclear bit is unset, through qcow2_updata_header() - r-b blocking change
08: move bdrv_load_autoloading_dirty_bitmaps under asserts block (Max)
fix typo in commit message
move bdrv_load_autoloading_dirty_bitmaps after asserts
John's r-b
09: rewrite check_dir_entry
remove extra feature cluster_size=0 from check_table_entry
which clears autoclear bit before trying to update bitmap directory bitmap_list_store - do not free old clusters if in_place=true
changes in comments, fix indents
add functions
bitmap_list_count
update_ext_header_and_dir_in_place (unset autoclear bit before trying to
modify bitmap directory)
(for usage in qcow2_load_autoloading_dirty_bitmaps)
11: add node name to error report
Max's r-b
13: add type Qcow2BitmapTable
move from bm.table_* to bm.table.*
rewrite check_constraints_on_bitmap
flush(bs->file) instead of flush(bs) in update_ext_header_and_dir assert(DIV_ROUND_UP(bm_size, dsc) == tb_size);
and assert(write_size <= s->cluster_size); in store_bitmap_data
fix: s/tb_size/tb_size * sizeof(tb[0]) in store_bitmap
in qcow2_store_persistent_dirty_bitmaps:
fail if !can_write
fix counting of new_nb_bitmaps and new_dir_size
free old clusters _after_ successful directory update (aggregate old bitmap
tables into drop_tables)
2 14: rename to can_store_new_dirty_bitmap
Max's r-b
15: rename to can_store_new_dirty_bitmap
rm extra !!
16: Max's r-b
rename to can_store_new_dirty_bitmap
indenting
Since s/2.8/2.9
17: Max's r-b
Since s/2.8/2.9
18: hashing can fail in comment for qapi
Since s/2.8/2.9
remove dead comment
19: indentation
Max's r-b
20: Max's r-b
21: fix error handling
return 0 if nb_bitmaps == 0
new patches:
22-23: remove bitmap from file on bdrv_release_dirty_bitmap
24: experimental, RFC: cache bitmap list to bs, to not reload it from file.
I'm not sure that is needed now, but if you want I can merge it to earlier patches.
v9:
rebase on master!
01-04,06,07: John's r-b
03: rewording in comment ("The concurrent resetting ..."), John's r-b 07: reword error messages
commit message: dirty bitmap -> bitmap
09: fix uninitialized ret in bitmap_list_store() (fix compilation warning) 18: fix compilation of tests/test-hbitmap
also, change constants names for qcow2: DIRTY_BITMAP -> BITMAP (as it is not dirty bitmaps extension, but just bitmaps extension), QCOW -> QCOW2 (bitmaps are only for qcow2)
v8:
Patches 01-06 are mostly unchanged, except for spelling and wording. 02 - add Eric's r-b
03,04 - add Max's r-b
07-09 is refactored "bitmap-read" part of the feature. Main changes:
- introduce bitmap list - normal list, which represents bitmap directory. Function bitmap_list_load loads bitmap directory, checks everything and creates normal list.
- introduce .bdrv_load_autoloading_dirty_bitmaps : instead of loading bitmaps in qcow2_open, lets load them only in bdrv_open, to avoid reloading in bdrv_snapshot_goto
- do not delete bitmaps after read, but mark them IN_USE
10 has contextual changes and rewording of comment. I've added Max's r-b, hope it's ok.
11: add error_report("Persistent bitmaps are lost") in case of failed bitmap store
12: add Max's r-b
13 is refactored "bitmap-store" part of the feature. see 07-09 description
- for now I just free old clusters and allocate new. This will be improved with a separate patch.
patch about "delete bitmaps on truncate" is removed. This case will be handled later. IN_USE, autoclear, check-constraints things are merged into other patches.
14-15 are mew patches, to early check possibility of creating persistent bitmap with
specified name and granularity in specified BDS
16-17: spelling, rewording, indenting, tiny code simplifying, check can_store (by 14-15),
s/if (autoload && !persistent)/if (has_autoload && !persistent)/, rebase (deleted qmp-commands.hx)
18: alternative to md5 in query-block:
- separated qmp command -x-debug-block-dirty-bitmap-sha256
- as adviced by Daniel P. Berrange in my parallel thread:
- sha256 instead of md5
- use qcrypto_hash_* instead of GChecksum
- fix bug =) (size was wrong in hbitmap_md5)
19: s/3999/3fff/, use x-debug-block-dirty-bitmap-sha256
20: new patch to rename and publish inc_refcounts
21: some fixes and refactoring mostyly by Max's comments.
Max, Eric, great tanks for your review!
I hope, I've covered most of your comments by this update.
TODO:
- handle reopening image RO->RW and incoming migration, set IN_USE for existing loaded bitmaps in these cases.
- reuse old, already allocated data clusters for bitmaps storing
- truncate bitmaps in the image on truncate
v7:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v7
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
- a lot of refactoring and reordering of patches.
- dead code removed (bdrv_dirty_bitmap_load, etc.)
- do not maintain extra data for now
- do not store dirty bitmap directory in memory (as we use it seldom, we can just reread if needed)
By Kevin's review:
01 - commit message changed: fix->improvement (as it was not a bug) 03 - r-b
04 - r-b
05 - add 21 patch to fix spec, also, removed all (I hope) mentions of
"Bitmap Header", switch to one unified name for it - "Bitmap Directory Entry", to avoid misunderstanding with Qcow2 header. (also, add patch 22, to fix it in spec)
v6.06 - improve for_each_dir_entry loop, reorder patches, other small fixes v6.07 ~> v7.09 - dead code removed, I've moved to one function
.bdrv_store_persistent_bitmaps and have wrapper and callback in one patch (with also some other staff. If it not ok, I can split them)
v6.08 - about keeping bitmap directory instead of bitmap list: no I don't keep
it at all.
v6.16 - old bdrv_ bitmap-storing related functions are removed. The new one is
bdrv_store_persistent_bitmaps.
v6:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v6
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
There are a lot of changes, reorderings and additions in comparement with v5. One principal thing: now bitmaps are removed from image after loading instead of marking them in_use. It is simpler and we do not need to store superfluous data. Also, we are no more interested in command line interface to dirty bitmaps. So it is dropped. If someone needs it I can add it later.
Vladimir Sementsov-Ogievskiy (25):
specs/qcow2: fix bitmap granularity qemu-specific note specs/qcow2: do not use wording 'bitmap header'
hbitmap: improve dirty iter
tests: add hbitmap iter test
block: fix bdrv_dirty_bitmap_granularity signature block/dirty-bitmap: add deserialize_ones func
qcow2: add bitmaps extension
block: introduce auto-loading bitmaps
qcow2: add .bdrv_load_autoloading_dirty_bitmaps block/dirty-bitmap: add autoload field to BdrvDirtyBitmap
block: introduce persistent dirty bitmaps block/dirty-bitmap: add bdrv_dirty_bitmap_next()
qcow2: add .bdrv_store_persistent_dirty_bitmaps()
block: add bdrv_can_store_new_dirty_bitmap
qcow2: add .bdrv_can_store_new_dirty_bitmap
qmp: add persistent flag to block-dirty-bitmap-add
qmp: add autoload parameter to block-dirty-bitmap-add
qmp: add x-debug-block-dirty-bitmap-sha256
iotests: test qcow2 persistent dirty bitmap
qcow2-refcount: rename inc_refcounts() and make it public
qcow2-bitmap: refcounts
block/dirty-bitmap: add bdrv_remove_persistent_dirty_bitmap
qcow2: add .bdrv_remove_persistent_dirty_bitmap
qmp: block-dirty-bitmap-remove: remove persistent
qcow2-bitmap: improve check_constraints_on_bitmap
block.c | 68 +++ block/Makefile.objs | 2 +- block/dirty-bitmap.c | 80 ++- block/qcow2-bitmap.c | 1360 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2-refcount.c | 59 +- block/qcow2.c | 128 +++- block/qcow2.h | 42 ++ blockdev.c | 71 ++- docs/specs/qcow2.txt | 8 +- include/block/block.h | 5 + include/block/block_int.h | 12 + include/block/dirty-bitmap.h | 21 +- include/qemu/hbitmap.h | 49 +- qapi/block-core.json | 39 +- tests/Makefile.include | 2 +- tests/qemu-iotests/165 | 89 +++ tests/qemu-iotests/165.out | 5 + tests/qemu-iotests/group | 1 + tests/test-hbitmap.c | 19 + util/hbitmap.c | 51 +- 20 files changed, 2047 insertions(+), 64 deletions(-) create mode 100644 block/qcow2-bitmap.c create mode 100755 tests/qemu-iotests/165 create mode 100644 tests/qemu-iotests/165.out
Signature · 1 line
1.8.3.1
PATCH 1/25 specs/qcow2: fix bitmap granularity qemu-specific note
--- docs/specs/qcow2.txt | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
docs/specs/qcow2.txt +1 -2
@@ -472,8 +472,7 @@ Structure of a bitmap directory entry: 17: granularity_bits Granularity bits. Valid values: 0 - 63. - Note: Qemu currently doesn't support granularity_bits - greater than 31. + Note: Qemu currently supports only values 9 - 31. Granularity is calculated as granularity = 1 << granularity_bits
Signature · 1 line
1.8.3.1
PATCH 2/25 specs/qcow2: do not use wording 'bitmap header'
In reply to Vladimir Sementsov-OgievskiyA bitmap directory entry is sometimes called a 'bitmap header'. This patch leaves only one name - 'bitmap directory entry'. The name 'bitmap header' creates misunderstandings with 'qcow2 header' and 'qcow2 bitmap header extension' (which is extension of qcow2 header)
--- docs/specs/qcow2.txt | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
docs/specs/qcow2.txt +2 -3
@@ -201,7 +201,7 @@ The fields of the bitmaps extension are: 8 - 15: bitmap_directory_size Size of the bitmap directory in bytes. It is the cumulative - size of all (nb_bitmaps) bitmap headers. + size of all (nb_bitmaps) bitmap directory entries. 16 - 23: bitmap_directory_offset Offset into the image file at which the bitmap directory
@@ -426,8 +426,7 @@ Each bitmap saved in the image is described in a bitmap directory entry. The bitmap directory is a contiguous area in the image file, whose starting offset and length are given by the header extension fields bitmap_directory_offset and bitmap_directory_size. The entries of the bitmap directory have variable -length, depending on the lengths of the bitmap name and extra data. These -entries are also called bitmap headers. +length, depending on the lengths of the bitmap name and extra data. Structure of a bitmap directory entry:
Signature · 1 line
1.8.3.1
PATCH 3/25 hbitmap: improve dirty iter
In reply to Vladimir Sementsov-OgievskiyMake dirty iter resistant to resetting bits in corresponding HBitmap.
--- include/qemu/hbitmap.h | 26 ++++---------------------- util/hbitmap.c | 23 ++++++++++++++++++++++- 2 files changed, 26 insertions(+), 23 deletions(-)
include/qemu/hbitmap.h +4 -22
@@ -256,10 +256,9 @@ void hbitmap_free(HBitmap *hb);* the lowest-numbered bit that is set in @hb, starting at @first. * * Concurrent setting of bits is acceptable, and will at worst cause the -* iteration to miss some of those bits. Resetting bits before the current -* position of the iterator is also okay. However, concurrent resetting of -* bits can lead to unexpected behavior if the iterator has not yet reached -* those bits. +* iteration to miss some of those bits. +* +* The concurrent resetting of bits is OK. */ void hbitmap_iter_init (HBitmapIter *hbi,const HBitmap *hb,uint64_t first);
@@ -298,24 +297,7 @@ void hbitmap_free_meta(HBitmap *hb);* Return the next bit that is set in @hbi's associated HBitmap, * or -1 if all remaining bits are zero. */ -static inline int64_t hbitmap_iter_next (HBitmapIter *hbi) -{ -unsigned long cur = hbi->cur[HBITMAP_LEVELS -1 ]; -int64_t item; - -if (cur ==0 ) { - cur =hbitmap_iter_skip_words (hbi); -if (cur ==0 ) { -return -1 ; - } - } - -/* The next call will resume work from the next bit. */ - hbi->cur[HBITMAP_LEVELS -1 ] = cur & (cur -1 ); - item = ((uint64_t )hbi->pos <<BITS_PER_LEVEL ) +ctzl (cur); - -return item << hbi->granularity; -} +int64_t hbitmap_iter_next (HBitmapIter *hbi);/** * hbitmap_iter_next_word:
util/hbitmap.c +22 -1
@@ -106,8 +106,9 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)unsigned long cur;do { - cur = hbi->cur[--i]; + i--; pos >>=BITS_PER_LEVEL ; + cur = hbi->cur[i] & hb->levels[i][pos]; }while (cur ==0 );/* Check for end of iteration. We always use fewer than BITS_PER_LONG
@@ -139,6 +140,26 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)return cur; } +int64_t hbitmap_iter_next (HBitmapIter *hbi) +{ +unsigned long cur = hbi->cur[HBITMAP_LEVELS -1 ] & + hbi->hb->levels[HBITMAP_LEVELS -1 ][hbi->pos]; +int64_t item; + +if (cur ==0 ) { + cur =hbitmap_iter_skip_words (hbi); +if (cur ==0 ) { +return -1 ; + } + } + +/* The next call will resume work from the next bit. */ + hbi->cur[HBITMAP_LEVELS -1 ] = cur & (cur -1 ); + item = ((uint64_t )hbi->pos <<BITS_PER_LEVEL ) +ctzl (cur); + +return item << hbi->granularity; +} +void hbitmap_iter_init (HBitmapIter *hbi,const HBitmap *hb,uint64_t first) {unsigned i, bit;
Signature · 1 line
1.8.3.1
PATCH 4/25 tests: add hbitmap iter test
In reply to Vladimir Sementsov-OgievskiyTest that hbitmap iter is resistant to bitmap resetting.
--- tests/test-hbitmap.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
tests/test-hbitmap.c +19 -0
@@ -909,6 +909,22 @@ static void hbitmap_test_add(const char *testpath, hbitmap_test_teardown); } +static void test_hbitmap_iter_and_reset (TestHBitmapData *data, +const void *unused) +{ +HBitmapIter hbi; + +hbitmap_test_init (data,L1 *2 ,0 ); +hbitmap_set (data->hb,0 , data->size); + +hbitmap_iter_init (&hbi, data->hb,BITS_PER_LONG -1 ); + +hbitmap_iter_next (&hbi); + +hbitmap_reset_all (data->hb); +hbitmap_iter_next (&hbi); +} +int main (int argc,char **argv) {g_test_init (&argc, &argv,NULL );
@@ -966,6 +982,9 @@ int main(int argc, char **argv) test_hbitmap_serialize_part);hbitmap_test_add ("/hbitmap/serialize/zeroes" , test_hbitmap_serialize_zeroes); + +hbitmap_test_add ("/hbitmap/iter/iter_and_reset" , + test_hbitmap_iter_and_reset);g_test_run ();return 0 ;
Signature · 1 line
1.8.3.1
PATCH 5/25 block: fix bdrv_dirty_bitmap_granularity signature
In reply to Vladimir Sementsov-OgievskiyMake getter signature const-correct. This allows other functions with const dirty bitmap parameter use bdrv_dirty_bitmap_granularity().
--- block/dirty-bitmap.c | 2 +- include/block/dirty-bitmap.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
block/dirty-bitmap.c +1 -1
@@ -388,7 +388,7 @@ uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs)return granularity; } -uint32_t bdrv_dirty_bitmap_granularity (BdrvDirtyBitmap *bitmap) +uint32_t bdrv_dirty_bitmap_granularity (const BdrvDirtyBitmap *bitmap) {return BDRV_SECTOR_SIZE <<hbitmap_granularity (bitmap->bitmap); }
include/block/dirty-bitmap.h +1 -1
@@ -29,7 +29,7 @@ void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap);void bdrv_enable_dirty_bitmap (BdrvDirtyBitmap *bitmap);BlockDirtyInfoList *bdrv_query_dirty_bitmaps (BlockDriverState *bs);uint32_t bdrv_get_default_bitmap_granularity (BlockDriverState *bs); -uint32_t bdrv_dirty_bitmap_granularity (BdrvDirtyBitmap *bitmap); +uint32_t bdrv_dirty_bitmap_granularity (const BdrvDirtyBitmap *bitmap);uint32_t bdrv_dirty_bitmap_meta_granularity (BdrvDirtyBitmap *bitmap);bool bdrv_dirty_bitmap_enabled (BdrvDirtyBitmap *bitmap);bool bdrv_dirty_bitmap_frozen (BdrvDirtyBitmap *bitmap);
Signature · 1 line
1.8.3.1
PATCH 6/25 block/dirty-bitmap: add deserialize_ones func
In reply to Vladimir Sementsov-OgievskiyAdd bdrv_dirty_bitmap_deserialize_ones() function, which is needed for qcow2 bitmap loading, to handle unallocated bitmap parts, marked as all-ones.
--- block/dirty-bitmap.c | 7 +++++++ include/block/dirty-bitmap.h | 3 +++ include/qemu/hbitmap.h | 15 +++++++++++++++ util/hbitmap.c | 17 +++++++++++++++++ 4 files changed, 42 insertions(+)
block/dirty-bitmap.c +7 -0
@@ -499,6 +499,13 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,hbitmap_deserialize_zeroes (bitmap->bitmap, start, count, finish); } +void bdrv_dirty_bitmap_deserialize_ones (BdrvDirtyBitmap *bitmap, +uint64_t start,uint64_t count, +bool finish) +{ +hbitmap_deserialize_ones (bitmap->bitmap, start, count, finish); +} +void bdrv_dirty_bitmap_deserialize_finish (BdrvDirtyBitmap *bitmap) {hbitmap_deserialize_finish (bitmap->bitmap);
include/block/dirty-bitmap.h +3 -0
@@ -70,6 +70,9 @@ void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap,void bdrv_dirty_bitmap_deserialize_zeroes (BdrvDirtyBitmap *bitmap,uint64_t start,uint64_t count,bool finish); +void bdrv_dirty_bitmap_deserialize_ones (BdrvDirtyBitmap *bitmap, +uint64_t start,uint64_t count, +bool finish);void bdrv_dirty_bitmap_deserialize_finish (BdrvDirtyBitmap *bitmap);#endif
include/qemu/hbitmap.h +15 -0
@@ -229,6 +229,21 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count, bool finish);/** +* hbitmap_deserialize_ones +* @hb: HBitmap to operate on. +* @start: First bit to restore. +* @count: Number of bits to restore. +* @finish: Whether to call hbitmap_deserialize_finish automatically. +* +* Fills the bitmap with ones. +* +* If @finish is false, caller must call hbitmap_serialize_finish before using +* the bitmap. +*/ +void hbitmap_deserialize_ones (HBitmap *hb,uint64_t start,uint64_t count, +bool finish); + +/** * hbitmap_deserialize_finish * @hb: HBitmap to operate on. *
util/hbitmap.c +17 -0
@@ -551,6 +551,23 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count, } } +void hbitmap_deserialize_ones (HBitmap *hb,uint64_t start,uint64_t count, +bool finish) +{ +uint64_t el_count; +unsigned long *first; + +if (!count) { +return ; + } +serialization_chunk (hb, start, count, &first, &el_count); + +memset (first,0xff , el_count *sizeof (unsigned long )); +if (finish) { +hbitmap_deserialize_finish (hb); + } +} +void hbitmap_deserialize_finish (HBitmap *bitmap) {int64_t i, size, prev_size;
Signature · 1 line
1.8.3.1
PATCH 7/25 qcow2: add bitmaps extension
In reply to Vladimir Sementsov-OgievskiyAdd bitmap extension as specified in docs/specs/qcow2.txt. For now, just mirror extension header into Qcow2 state and check constraints.
For now, disable image resize if it has bitmaps. It will be fixed later.
--- block/qcow2.c | 123 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--- block/qcow2.h | 24 ++++++++++++ 2 files changed, 142 insertions(+), 5 deletions(-)
block/qcow2.c +118 -5
@@ -63,6 +63,7 @@ typedef struct {#define QCOW2_EXT_MAGIC_END 0#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA#define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857 +#define QCOW2_EXT_MAGIC_BITMAPS 0x23852875static int qcow2_probe (const uint8_t *buf,int buf_size,const char *filename) {
@@ -86,12 +87,17 @@ static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)*/ static int qcow2_read_extensions (BlockDriverState *bs,uint64_t start_offset,uint64_t end_offset,void **p_feature_table, -Error **errp) +bool *need_update_header,Error **errp) {BDRVQcow2State *s = bs->opaque;QCowExtension ext;uint64_t offset;int ret; +Qcow2BitmapHeaderExt bitmaps_ext; + +if (need_update_header !=NULL ) { + *need_update_header = false; + }#ifdef DEBUG_EXT printf ("qcow2_read_extensions: start=%ld end=%ld\n" , start_offset, end_offset);
@@ -162,6 +168,85 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, }break ; +case QCOW2_EXT_MAGIC_BITMAPS : +if (ext.len !=sizeof (bitmaps_ext)) { +error_setg_errno (errp, -ret,"bitmaps_ext: " +"Invalid extension length" ); +return -EINVAL ; + } + +if (!(s->autoclear_features &QCOW2_AUTOCLEAR_BITMAPS )) { +fprintf (stderr, +"WARNING: a program lacking bitmap support modified " +"this file, so all bitmaps are now considered " +"inconsistent. Some clusters may be leaked, run " +"'qemu-img check -r' on the image file to fix." ); +if (need_update_header !=NULL ) { + *need_update_header = true; + } +break ; + } + + ret =bdrv_pread (bs->file, offset, &bitmaps_ext, ext.len); +if (ret <0 ) { +error_setg_errno (errp, -ret,"bitmaps_ext: " +"Could not read ext header" ); +return ret; + } + +if (bitmaps_ext.reserved32 !=0 ) { +error_setg_errno (errp, -ret,"bitmaps_ext: " +"Reserved field is not zero" ); +return -EINVAL ; + } + +be32_to_cpus (&bitmaps_ext.nb_bitmaps); +be64_to_cpus (&bitmaps_ext.bitmap_directory_size); +be64_to_cpus (&bitmaps_ext.bitmap_directory_offset); + +if (bitmaps_ext.nb_bitmaps >QCOW2_MAX_BITMAPS ) { +error_setg (errp, +"bitmaps_ext: File %s has %" PRIu32" bitmaps, " +"exceeding the QEMU supported maximum of %d" , + bs->filename, bitmaps_ext.nb_bitmaps, +QCOW2_MAX_BITMAPS ); +return -EINVAL ; + } + +if (bitmaps_ext.nb_bitmaps ==0 ) { +error_setg (errp,"found bitmaps extension with zero bitmaps" ); +return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_offset & (s->cluster_size -1 )) { +error_setg (errp,"bitmaps_ext: " +"invalid bitmap directory offset" ); +return -EINVAL ; + } + +if (bitmaps_ext.bitmap_directory_size > +QCOW2_MAX_BITMAP_DIRECTORY_SIZE ) { +error_setg (errp,"bitmaps_ext: " +"bitmap directory size (%" PRIu64") exceeds " +"the maximum supported size (%d)" , + bitmaps_ext.bitmap_directory_size, +QCOW2_MAX_BITMAP_DIRECTORY_SIZE ); +return -EINVAL ; + } + + s->nb_bitmaps = bitmaps_ext.nb_bitmaps; + s->bitmap_directory_offset = + bitmaps_ext.bitmap_directory_offset; + s->bitmap_directory_size = + bitmaps_ext.bitmap_directory_size; + +#ifdef DEBUG_EXT +printf ("Qcow2: Got bitmaps extension: " +"offset=%" PRIu64" nb_bitmaps=%" PRIu32"\n" , + s->bitmap_directory_offset, s->nb_bitmaps); +#endif +break ; +default :/* unknown magic - save it in case we need to rewrite the header */ {
@@ -824,6 +909,7 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,Error *local_err =NULL ;uint64_t ext_end;uint64_t l1_vm_state_index; +bool need_update_header = false; ret =bdrv_pread (bs->file,0 , &header,sizeof (header));if (ret <0 ) {
@@ -929,7 +1015,7 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK ) {void *feature_table =NULL ;qcow2_read_extensions (bs, header.header_length, ext_end, - &feature_table,NULL ); + &feature_table,NULL ,NULL );report_unsupported_feature (errp, feature_table, s->incompatible_features & ~QCOW2_INCOMPAT_MASK );
@@ -1116,7 +1202,7 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,/* read qcow2 extensions */ if (qcow2_read_extensions (bs, header.header_length, ext_end,NULL , - &local_err)) { + &need_update_header, &local_err)) {error_propagate (errp, local_err); ret = -EINVAL ; goto fail;
@@ -1152,8 +1238,10 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, }/* Clear unknown autoclear feature bits */ -if (!bs->read_only && !(flags &BDRV_O_INACTIVE ) && s->autoclear_features) { - s->autoclear_features =0 ; + need_update_header |= s->autoclear_features & ~QCOW2_AUTOCLEAR_MASK ; + +if (need_update_header && !bs->read_only && !(flags &BDRV_O_INACTIVE )) { + s->autoclear_features &=QCOW2_AUTOCLEAR_MASK ; ret =qcow2_update_header (bs);if (ret <0 ) {error_setg_errno (errp, -ret,"Could not update qcow2 header" );
@@ -1953,6 +2041,24 @@ int qcow2_update_header(BlockDriverState *bs) buflen -= ret; } +if (s->nb_bitmaps >0 ) { +Qcow2BitmapHeaderExt bitmaps_header = { + .nb_bitmaps =cpu_to_be32 (s->nb_bitmaps), + .bitmap_directory_size = +cpu_to_be64 (s->bitmap_directory_size), + .bitmap_directory_offset = +cpu_to_be64 (s->bitmap_directory_offset) + }; + ret =header_ext_add (buf,QCOW2_EXT_MAGIC_BITMAPS , + &bitmaps_header,sizeof (bitmaps_header), + buflen); +if (ret <0 ) { + goto fail; + } + buf += ret; + buflen -= ret; + } +/* Keep unknown header extensions */ QLIST_FOREACH(uext, &s->unknown_header_ext, next) { ret =header_ext_add (buf, uext->magic, uext->data, uext->len, buflen);
@@ -2528,6 +2634,13 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset)return -ENOTSUP ; } +/* cannot proceed if image has bitmaps */ +if (s->nb_bitmaps) { +/* TODO: resize bitmaps in the image */ +error_report ("Can't resize an image which has bitmaps" ); +return -ENOTSUP ; + } +/* shrinking is currently not supported */ if (offset < bs->total_sectors *512 ) {error_report ("qcow2 doesn't support shrinking images yet" );
block/qcow2.h +24 -0
@@ -52,6 +52,10 @@* space for snapshot names and IDs */ #define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS) +/* Bitmap header extension constraints */ +#define QCOW2_MAX_BITMAPS 65535 +#define QCOW2_MAX_BITMAP_DIRECTORY_SIZE (1024 * QCOW2_MAX_BITMAPS) +/* indicate that the refcount of the referenced cluster is exactly one. */ #define QCOW_OFLAG_COPIED (1ULL << 63)/* indicate that the cluster is compressed (they never have the copied flag) */
@@ -195,6 +199,15 @@ enum { QCOW2_COMPAT_FEAT_MASK = QCOW2_COMPAT_LAZY_REFCOUNTS, }; +/* Autoclear feature bits */ +enum { +QCOW2_AUTOCLEAR_BITMAPS_BITNR =0 , +QCOW2_AUTOCLEAR_BITMAPS = +1 <<QCOW2_AUTOCLEAR_BITMAPS_BITNR , + +QCOW2_AUTOCLEAR_MASK =QCOW2_AUTOCLEAR_BITMAPS , +}; +enum qcow2_discard_type {QCOW2_DISCARD_NEVER =0 ,QCOW2_DISCARD_ALWAYS ,
@@ -222,6 +235,13 @@ typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,typedef void Qcow2SetRefcountFunc (void *refcount_array,uint64_t index,uint64_t value); +typedef struct Qcow2BitmapHeaderExt { +uint32_t nb_bitmaps; +uint32_t reserved32; +uint64_t bitmap_directory_size; +uint64_t bitmap_directory_offset; +} QEMU_PACKED Qcow2BitmapHeaderExt; + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size;
@@ -263,6 +283,10 @@ typedef struct BDRVQcow2State {unsigned int nb_snapshots;QCowSnapshot *snapshots; +uint32_t nb_bitmaps; +uint64_t bitmap_directory_size; +uint64_t bitmap_directory_offset; +int flags;int qcow_version;bool use_lazy_refcounts;
Signature · 1 line
1.8.3.1
PATCH 8/25 block: introduce auto-loading bitmaps
In reply to Vladimir Sementsov-OgievskiyAuto loading bitmaps are bitmaps stored in the disk image, which should be loaded when the image is opened and become BdrvDirtyBitmaps for the corresponding drive.
--- block.c | 14 ++++++++++++++ include/block/block.h | 2 ++ include/block/block_int.h | 3 +++ 3 files changed, 19 insertions(+)
block.c +14 -0
@@ -1141,6 +1141,13 @@ static int bdrv_open_common(BlockDriverState *bs, BdrvChild *file,assert (bdrv_min_mem_align (bs) !=0 );assert (is_power_of_2 (bs->bl.request_alignment)); +bdrv_load_autoloading_dirty_bitmaps (bs, &local_err); +if (local_err) { +error_propagate (errp, local_err); + ret = -EINVAL ; + goto free_and_fail; + } +qemu_opts_del (opts);return 0 ;
@@ -4093,3 +4100,10 @@ void bdrv_del_child(BlockDriverState *parent_bs, BdrvChild *child, Error **errp) parent_bs->drv->bdrv_del_child (parent_bs, child, errp); } + +void bdrv_load_autoloading_dirty_bitmaps (BlockDriverState *bs,Error **errp) +{ +if (bs->drv && bs->drv->bdrv_load_autoloading_dirty_bitmaps) { + bs->drv->bdrv_load_autoloading_dirty_bitmaps (bs, errp); + } +}
include/block/block.h +2 -0
@@ -551,4 +551,6 @@ void bdrv_add_child(BlockDriverState *parent, BlockDriverState *child, Error **errp);void bdrv_del_child (BlockDriverState *parent,BdrvChild *child,Error **errp); +void bdrv_load_autoloading_dirty_bitmaps (BlockDriverState *bs,Error **errp); +#endif
include/block/block_int.h +3 -0
@@ -320,6 +320,9 @@ struct BlockDriver {void (*bdrv_del_child)(BlockDriverState *parent,BdrvChild *child,Error **errp); +void (*bdrv_load_autoloading_dirty_bitmaps)(BlockDriverState *bs, +Error **errp); +QLIST_ENTRY (BlockDriver) list; };
Signature · 1 line
1.8.3.1
PATCH 9/25 qcow2: add .bdrv_load_autoloading_dirty_bitmaps
In reply to Vladimir Sementsov-OgievskiyAuto loading bitmaps are bitmaps in Qcow2, with the AUTO flag set. They are loaded when the image is opened and become BdrvDirtyBitmaps for the corresponding drive.
Extra data in bitmaps is not supported for now.
--- block/Makefile.objs | 2 +- block/qcow2-bitmap.c | 724 +++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 2 + block/qcow2.h | 3 + 4 files changed, 730 insertions(+), 1 deletion(-) create mode 100644 block/qcow2-bitmap.c
block/Makefile.objs +1 -1
@@ -1,5 +1,5 @@ block-obj-y += raw-format.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o dmg.o -block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o +block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-bitmap.o block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-y += qed-check.o block-obj-y += vhdx.o vhdx-endian.o vhdx-log.o
block/qcow2-bitmap.c new file +724 -0
724 lines not on this page · raw message
block/qcow2.c +2 -0
@@ -3538,6 +3538,8 @@ BlockDriver bdrv_qcow2 = { .bdrv_detach_aio_context = qcow2_detach_aio_context, .bdrv_attach_aio_context = qcow2_attach_aio_context, + + .bdrv_load_autoloading_dirty_bitmaps = qcow2_load_autoloading_dirty_bitmaps, };static void bdrv_qcow2_init (void )
block/qcow2.h +3 -0
@@ -613,4 +613,7 @@ int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset, void **table);void qcow2_cache_put (BlockDriverState *bs,Qcow2Cache *c,void **table); +/* qcow2-bitmap.c functions */ +void qcow2_load_autoloading_dirty_bitmaps (BlockDriverState *bs,Error **errp); +#endif
Signature · 1 line
1.8.3.1
PATCH 10/25 block/dirty-bitmap: add autoload field to BdrvDirtyBitmap
In reply to Vladimir Sementsov-OgievskiyMirror AUTO flag from Qcow2 bitmap in BdrvDirtyBitmap. This will be needed in future, to save this flag back to Qcow2 for persistent bitmaps.
--- block/dirty-bitmap.c | 15 +++++++++++++++ block/qcow2-bitmap.c | 2 ++ include/block/dirty-bitmap.h | 2 ++ 3 files changed, 19 insertions(+)
block/dirty-bitmap.c +15 -0
@@ -44,6 +44,8 @@ struct BdrvDirtyBitmap {int64_t size;/* Size of the bitmap (Number of sectors) */ bool disabled;/* Bitmap is read-only */ int active_iterators;/* How many iterators are active */ +bool autoload;/* For persistent bitmaps: bitmap must be +autoloaded on image opening */ QLIST_ENTRY (BdrvDirtyBitmap) list; };
@@ -70,6 +72,7 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)assert (!bdrv_dirty_bitmap_frozen (bitmap));g_free (bitmap->name); bitmap->name =NULL ; + bitmap->autoload = false; } BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
@@ -238,6 +241,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs, bitmap->name =NULL ; successor->name = name; bitmap->successor =NULL ; + successor->autoload = bitmap->autoload; + bitmap->autoload = false;bdrv_release_dirty_bitmap (bs, bitmap);return successor;
@@ -540,3 +545,13 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap) {return hbitmap_count (bitmap->meta); } + +void bdrv_dirty_bitmap_set_autoload (BdrvDirtyBitmap *bitmap,bool autoload) +{ + bitmap->autoload = autoload; +} + +bool bdrv_dirty_bitmap_get_autoload (const BdrvDirtyBitmap *bitmap) +{ +return bitmap->autoload; +}
block/qcow2-bitmap.c +2 -0
@@ -697,6 +697,8 @@ void qcow2_load_autoloading_dirty_bitmaps(BlockDriverState *bs, Error **errp)if (bitmap ==NULL ) { goto fail; } + +bdrv_dirty_bitmap_set_autoload (bitmap, true); bm->flags |=BME_FLAG_IN_USE ; created_dirty_bitmaps =g_slist_append (created_dirty_bitmaps, bitmap);
include/block/dirty-bitmap.h +2 -0
@@ -75,4 +75,6 @@ void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap, bool finish);void bdrv_dirty_bitmap_deserialize_finish (BdrvDirtyBitmap *bitmap); +void bdrv_dirty_bitmap_set_autoload (BdrvDirtyBitmap *bitmap,bool autoload); +bool bdrv_dirty_bitmap_get_autoload (const BdrvDirtyBitmap *bitmap);#endif
Signature · 1 line
1.8.3.1
PATCH 11/25 block: introduce persistent dirty bitmaps
In reply to Vladimir Sementsov-OgievskiyNew field BdrvDirtyBitmap.persistent means, that bitmap should be saved on bdrv_close, using format driver. Format driver should maintain bitmap storing.
--- block.c | 32 ++++++++++++++++++++++++++++++++ block/dirty-bitmap.c | 26 ++++++++++++++++++++++++++ block/qcow2-bitmap.c | 1 + include/block/block.h | 1 + include/block/block_int.h | 2 ++ include/block/dirty-bitmap.h | 6 ++++++ 6 files changed, 68 insertions(+)
block.c +32 -0
@@ -2321,6 +2321,7 @@ void bdrv_reopen_abort(BDRVReopenState *reopen_state)static void bdrv_close (BlockDriverState *bs) {BdrvAioNotifier *ban, *ban_next; +Error *local_err =NULL ;assert (!bs->job);assert (!bs->refcnt);
@@ -2329,6 +2330,12 @@ static void bdrv_close(BlockDriverState *bs)bdrv_flush (bs);bdrv_drain (bs);/* in case flush left pending I/O */ +bdrv_store_persistent_dirty_bitmaps (bs, &local_err); +if (local_err !=NULL ) { +error_report_err (local_err); +error_report ("Persistent bitmaps are lost for node '%s'" , +bdrv_get_device_or_node_name (bs)); + }bdrv_release_named_dirty_bitmaps (bs);assert (QLIST_EMPTY (&bs->dirty_bitmaps));
@@ -4107,3 +4114,28 @@ void bdrv_load_autoloading_dirty_bitmaps(BlockDriverState *bs, Error **errp) bs->drv->bdrv_load_autoloading_dirty_bitmaps (bs, errp); } } + +void bdrv_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp) +{ +BlockDriver *drv = bs->drv; + +if (!bdrv_has_persistent_bitmaps (bs)) { +return ; + } + +if (!drv) { +error_setg_errno (errp,ENOMEDIUM , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return ; + } + +if (!drv->bdrv_store_persistent_dirty_bitmaps) { +error_setg_errno (errp,ENOTSUP , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return ; + } + + drv->bdrv_store_persistent_dirty_bitmaps (bs, errp); +}
block/dirty-bitmap.c +26 -0
@@ -44,6 +44,7 @@ struct BdrvDirtyBitmap {int64_t size;/* Size of the bitmap (Number of sectors) */ bool disabled;/* Bitmap is read-only */ int active_iterators;/* How many iterators are active */ +bool persistent;/* bitmap must be saved to owner disk image */ bool autoload;/* For persistent bitmaps: bitmap must be autoloaded on image opening */ QLIST_ENTRY (BdrvDirtyBitmap) list;
@@ -72,6 +73,7 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)assert (!bdrv_dirty_bitmap_frozen (bitmap));g_free (bitmap->name); bitmap->name =NULL ; + bitmap->persistent = false; bitmap->autoload = false; }
@@ -241,6 +243,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs, bitmap->name =NULL ; successor->name = name; bitmap->successor =NULL ; + successor->persistent = bitmap->persistent; + bitmap->persistent = false; successor->autoload = bitmap->autoload; bitmap->autoload = false;bdrv_release_dirty_bitmap (bs, bitmap);
@@ -555,3 +559,25 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap) {return bitmap->autoload; } + +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap,bool persistent) +{ + bitmap->persistent = persistent; +} + +bool bdrv_dirty_bitmap_get_persistance (BdrvDirtyBitmap *bitmap) +{ +return bitmap->persistent; +} + +bool bdrv_has_persistent_bitmaps (BlockDriverState *bs) +{ +BdrvDirtyBitmap *bm; + QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) { +if (bm->persistent) { +return true; + } + } + +return false; +}
block/qcow2-bitmap.c +1 -0
@@ -698,6 +698,7 @@ void qcow2_load_autoloading_dirty_bitmaps(BlockDriverState *bs, Error **errp) goto fail; } +bdrv_dirty_bitmap_set_persistance (bitmap, true);bdrv_dirty_bitmap_set_autoload (bitmap, true); bm->flags |=BME_FLAG_IN_USE ; created_dirty_bitmaps =
include/block/block.h +1 -0
@@ -552,5 +552,6 @@ void bdrv_add_child(BlockDriverState *parent, BlockDriverState *child,void bdrv_del_child (BlockDriverState *parent,BdrvChild *child,Error **errp);void bdrv_load_autoloading_dirty_bitmaps (BlockDriverState *bs,Error **errp); +void bdrv_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp);#endif
include/block/block_int.h +2 -0
@@ -322,6 +322,8 @@ struct BlockDriver {void (*bdrv_load_autoloading_dirty_bitmaps)(BlockDriverState *bs,Error **errp); +void (*bdrv_store_persistent_dirty_bitmaps)(BlockDriverState *bs, +Error **errp);QLIST_ENTRY (BlockDriver) list; };
include/block/dirty-bitmap.h +6 -0
@@ -77,4 +77,10 @@ void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);void bdrv_dirty_bitmap_set_autoload (BdrvDirtyBitmap *bitmap,bool autoload);bool bdrv_dirty_bitmap_get_autoload (const BdrvDirtyBitmap *bitmap); +void bdrv_dirty_bitmap_set_persistance (BdrvDirtyBitmap *bitmap, +bool persistent); +bool bdrv_dirty_bitmap_get_persistance (BdrvDirtyBitmap *bitmap); + +bool bdrv_has_persistent_bitmaps (BlockDriverState *bs); +#endif
Signature · 1 line
1.8.3.1
PATCH 12/25 block/dirty-bitmap: add bdrv_dirty_bitmap_next()
In reply to Vladimir Sementsov-Ogievskiy--- block/dirty-bitmap.c | 7 +++++++ include/block/dirty-bitmap.h | 3 +++ 2 files changed, 10 insertions(+)
block/dirty-bitmap.c +7 -0
@@ -581,3 +581,10 @@ bool bdrv_has_persistent_bitmaps(BlockDriverState *bs)return false; } + +BdrvDirtyBitmap *bdrv_dirty_bitmap_next (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap) +{ +return bitmap ==NULL ?QLIST_FIRST (&bs->dirty_bitmaps) : +QLIST_NEXT (bitmap, list); +}
include/block/dirty-bitmap.h +3 -0
@@ -81,6 +81,9 @@ void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap *bitmap, bool persistent);bool bdrv_dirty_bitmap_get_persistance (BdrvDirtyBitmap *bitmap); +BdrvDirtyBitmap *bdrv_dirty_bitmap_next (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap); +bool bdrv_has_persistent_bitmaps (BlockDriverState *bs);#endif
Signature · 1 line
1.8.3.1
PATCH 13/25 qcow2: add .bdrv_store_persistent_dirty_bitmaps()
In reply to Vladimir Sementsov-OgievskiyRealize block bitmap storing interface, to allow qcow2 images store persistent bitmaps.
--- block/qcow2-bitmap.c | 487 +++++++++++++++++++++++++++++++++++++++++++++++++-- block/qcow2.c | 1 + block/qcow2.h | 1 + 3 files changed, 474 insertions(+), 15 deletions(-)
block/qcow2-bitmap.c +472 -15
@@ -28,6 +28,7 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "exec/log.h" +#include "qemu/cutils.h" #include "block/block_int.h" #include "block/qcow2.h"
@@ -43,6 +44,10 @@#define BME_MIN_GRANULARITY_BITS 9#define BME_MAX_NAME_SIZE 1023 +#if BME_MAX_TABLE_SIZE *8ULL >INT_MAX +#error In the code bitmap table physical size assumed to fit into int +#endif +/* Bitmap directory entry flags */ #define BME_RESERVED_FLAGS 0xfffffffcU#define BME_FLAG_IN_USE 1
@@ -67,13 +72,22 @@ typedef struct QEMU_PACKED Qcow2BitmapDirEntry {/* name follows */ } Qcow2BitmapDirEntry; +typedef struct Qcow2BitmapTable { +uint64_t offset; +uint32_t size;/* number of 64bit entries */ +QSIMPLEQ_ENTRY (Qcow2BitmapTable) entry; +}Qcow2BitmapTable ; +typedef QSIMPLEQ_HEAD(Qcow2BitmapTableList, Qcow2BitmapTable) +Qcow2BitmapTableList ; +typedef struct Qcow2Bitmap { -uint64_t table_offset; -uint32_t table_size; +Qcow2BitmapTable table;uint32_t flags;uint8_t granularity_bits;char *name; +BdrvDirtyBitmap *dirty_bitmap; +QSIMPLEQ_ENTRY (Qcow2Bitmap) entry; }Qcow2Bitmap ; typedef QSIMPLEQ_HEAD(Qcow2BitmapList, Qcow2Bitmap) Qcow2BitmapList;
@@ -108,6 +122,15 @@ static int update_header_sync(BlockDriverState *bs)return 0 ; } +static inline void bitmap_table_to_be (uint64_t *bitmap_table,size_t size) +{ +size_t i; + +for (i =0 ; i < size; ++i) { +cpu_to_be64s (&bitmap_table[i]); + } +} +static int check_table_entry (uint64_t entry,int cluster_size) {uint64_t offset;
@@ -135,7 +158,50 @@ static int check_table_entry(uint64_t entry, int cluster_size)return 0 ; } -static int bitmap_table_load (BlockDriverState *bs,Qcow2Bitmap *bm, +static int check_constraints_on_bitmap (BlockDriverState *bs, +const char *name, +uint32_t granularity) +{ +BDRVQcow2State *s = bs->opaque; +int granularity_bits =ctz32 (granularity); +int64_t len =bdrv_getlength (bs); +bool fail; + +assert (granularity >0 ); +assert ((granularity & (granularity -1 )) ==0 ); + +if (len <0 ) { +return len; + } + + fail = (granularity_bits >BME_MAX_GRANULARITY_BITS ) || + (granularity_bits <BME_MIN_GRANULARITY_BITS ) || + (len > (uint64_t )BME_MAX_PHYS_SIZE << granularity_bits) || + (len > (uint64_t )BME_MAX_TABLE_SIZE * s->cluster_size << + granularity_bits) || + (strlen (name) >BME_MAX_NAME_SIZE ); + +return fail ? -EINVAL :0 ; +} + +static void clear_bitmap_table (BlockDriverState *bs,uint64_t *bitmap_table, +uint32_t bitmap_table_size) +{ +BDRVQcow2State *s = bs->opaque; +int i; + +for (i =0 ; i < bitmap_table_size; ++i) { +uint64_t addr = bitmap_table[i] &BME_TABLE_ENTRY_OFFSET_MASK ; +if (!addr) { +continue ; + } + +qcow2_free_clusters (bs, addr, s->cluster_size,QCOW2_DISCARD_OTHER ); + bitmap_table[i] =0 ; + } +} + +static int bitmap_table_load (BlockDriverState *bs,Qcow2BitmapTable *tb,uint64_t **bitmap_table) {int ret;
@@ -143,20 +209,20 @@ static int bitmap_table_load(BlockDriverState *bs, Qcow2Bitmap *bm,uint32_t i;uint64_t *table; -assert (bm->table_size !=0 ); - table =g_try_new (uint64_t, bm->table_size); +assert (tb->size !=0 ); + table =g_try_new (uint64_t, tb->size);if (table ==NULL ) {return -ENOMEM ; } -assert (bm->table_size <=BME_MAX_TABLE_SIZE ); - ret =bdrv_pread (bs->file, bm->table_offset, - table, bm->table_size *sizeof (uint64_t )); +assert (tb->size <=BME_MAX_TABLE_SIZE ); + ret =bdrv_pread (bs->file, tb->offset, + table, tb->size *sizeof (uint64_t ));if (ret <0 ) { goto fail; } -for (i =0 ; i < bm->table_size; ++i) { +for (i =0 ; i < tb->size; ++i) {be64_to_cpus (&table[i]); ret =check_table_entry (table[i], s->cluster_size);if (ret <0 ) {
@@ -173,6 +239,28 @@ fail:return ret; } +static int free_bitmap_clusters (BlockDriverState *bs,Qcow2BitmapTable *tb) +{ +int ret; +uint64_t *bitmap_table; + + ret =bitmap_table_load (bs, tb, &bitmap_table); +if (ret <0 ) { +assert (bitmap_table ==NULL ); +return ret; + } + +clear_bitmap_table (bs, bitmap_table, tb->size); +qcow2_free_clusters (bs, tb->offset, tb->size *sizeof (uint64_t ), +QCOW2_DISCARD_OTHER ); +g_free (bitmap_table); + + tb->offset =0 ; + tb->size =0 ; + +return 0 ; +} +/* This function returns the number of disk sectors covered by a single cluster * of bitmap data. */ static uint64_t disk_sectors_in_bitmap_cluster(const BDRVQcow2State *s,
@@ -254,7 +342,7 @@ static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs, goto fail; } - ret =bitmap_table_load (bs, bm, &bitmap_table); + ret =bitmap_table_load (bs, &bm->table, &bitmap_table);if (ret <0 ) {error_setg_errno (errp, -ret, "Could not read bitmap_table table from image for "
@@ -268,7 +356,7 @@ static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs, goto fail; } - ret =load_bitmap_data (bs, bitmap_table, bm->table_size, + ret =load_bitmap_data (bs, bitmap_table, bm->table.size, bitmap);if (ret <0 ) { error_setg_errno(errp, -ret, "Could not read bitmap '%s' from image",
@@ -504,8 +592,8 @@ static Qcow2BitmapList *bitmap_list_load(BlockDriverState *bs, uint64_t offset, } bm =g_new (Qcow2Bitmap,1 ); - bm->table_offset = e->bitmap_table_offset; - bm->table_size = e->bitmap_table_size; + bm->table.offset = e->bitmap_table_offset; + bm->table.size = e->bitmap_table_size; bm->flags = e->flags; bm->granularity_bits = e->granularity_bits; bm->name =dir_entry_copy_name (e);
@@ -573,8 +661,8 @@ static int bitmap_list_store(BlockDriverState *bs, Qcow2BitmapList *bm_list, e = (Qcow2BitmapDirEntry *)dir; QSIMPLEQ_FOREACH(bm, bm_list, entry) { - e->bitmap_table_offset = bm->table_offset; - e->bitmap_table_size = bm->table_size; + e->bitmap_table_offset = bm->table.offset; + e->bitmap_table_size = bm->table.size; e->flags = bm->flags; e->type =BT_DIRTY_TRACKING_BITMAP ; e->granularity_bits = bm->granularity_bits;
@@ -666,6 +754,69 @@ static int update_ext_header_and_dir_in_place(BlockDriverState *bs,return update_header_sync (bs); } +static int update_ext_header_and_dir (BlockDriverState *bs, +Qcow2BitmapList *bm_list) +{ +BDRVQcow2State *s = bs->opaque; +int ret; +uint64_t new_offset =0 ; +uint64_t new_size =0 ; +uint32_t new_nb_bitmaps =0 ; +uint64_t old_offset = s->bitmap_directory_offset; +uint64_t old_size = s->bitmap_directory_size; +uint32_t old_nb_bitmaps = s->nb_bitmaps; +uint64_t old_autocl = s->autoclear_features; + +if (bm_list !=NULL && !QSIMPLEQ_EMPTY (bm_list)) { + new_nb_bitmaps =bitmap_list_count (bm_list); + +if (new_nb_bitmaps >QCOW2_MAX_BITMAPS ) { +return -EINVAL ; + } + + ret =bitmap_list_store (bs, bm_list, &new_offset, &new_size, false); +if (ret <0 ) { +return ret; + } + + ret =bdrv_flush (bs->file->bs); +if (ret <0 ) { + goto fail; + } + + s->autoclear_features |=QCOW2_AUTOCLEAR_BITMAPS ; + }else { + s->autoclear_features &= ~(uint64_t )QCOW2_AUTOCLEAR_BITMAPS ; + } + + s->bitmap_directory_offset = new_offset; + s->bitmap_directory_size = new_size; + s->nb_bitmaps = new_nb_bitmaps; + + ret =update_header_sync (bs); +if (ret <0 ) { + goto fail; + } + +if (old_size >0 ) { +qcow2_free_clusters (bs, old_offset, old_size,QCOW2_DISCARD_OTHER ); + } + +return 0 ; + +fail: +if (new_offset >0 ) { +qcow2_free_clusters (bs, new_offset, new_size,QCOW2_DISCARD_OTHER ); + } + + s->bitmap_directory_offset = old_offset; + s->bitmap_directory_size = old_size; + s->nb_bitmaps = old_nb_bitmaps; + s->autoclear_features = old_autocl; + +return ret; +} +/* for g_slist_foreach for GSList of BdrvDirtyBitmap* elements */ static void release_dirty_bitmap_helper (gpointer bitmap,gpointer bs)
@@ -725,3 +876,309 @@ fail:g_slist_free (created_dirty_bitmaps);bitmap_list_free (bm_list); } + +/* store_bitmap_data() +* Store bitmap to image, filling bitmap table accordingly. +*/ +static uint64_t *store_bitmap_data (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap, +uint32_t *bitmap_table_size,Error **errp) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +int64_t sector; +uint64_t dsc; +uint64_t bm_size =bdrv_dirty_bitmap_size (bitmap); +const char *bm_name =bdrv_dirty_bitmap_name (bitmap); +uint8_t *buf =NULL ; +BdrvDirtyBitmapIter *dbi; +uint64_t *tb; +uint64_t tb_size = +size_to_clusters (s, +bdrv_dirty_bitmap_serialization_size (bitmap,0 , bm_size)); + +if (tb_size >BME_MAX_TABLE_SIZE || + tb_size * s->cluster_size >BME_MAX_PHYS_SIZE ) + { +error_setg (errp,"Bitmap '%s' is too big" , bm_name); +return NULL ; + } + + tb =g_try_new0 (uint64_t, tb_size); +if (tb ==NULL ) { +error_setg (errp,"No memory" ); +return NULL ; + } + + dbi =bdrv_dirty_iter_new (bitmap,0 ); + buf =g_malloc (s->cluster_size); + dsc =disk_sectors_in_bitmap_cluster (s, bitmap); +assert (DIV_ROUND_UP (bm_size, dsc) == tb_size); + +while ((sector =bdrv_dirty_iter_next (dbi)) !=-1 ) { +uint64_t cluster = sector / dsc; +uint64_t end, write_size; +int64_t off; + + sector = cluster * dsc; + end =MIN (bm_size, sector + dsc); + write_size = +bdrv_dirty_bitmap_serialization_size (bitmap, sector, end - sector); +assert (write_size <= s->cluster_size); + + off =qcow2_alloc_clusters (bs, s->cluster_size); +if (off <0 ) { +error_setg_errno (errp, -off, +"Failed to allocate clusters for bitmap '%s'" , + bm_name); + goto fail; + } + tb[cluster] = off; + +bdrv_dirty_bitmap_serialize_part (bitmap, buf, sector, end - sector); +if (write_size < s->cluster_size) { +memset (buf + write_size,0 , s->cluster_size - write_size); + } + + ret =qcow2_pre_write_overlap_check (bs,0 , off, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Qcow2 overlap check failed" ); + goto fail; + } + + ret =bdrv_pwrite (bs->file, off, buf, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write bitmap '%s' to file" , + bm_name); + goto fail; + } + +if (end >= bm_size) { +break ; + } + +bdrv_set_dirty_iter (dbi, end); + } + + *bitmap_table_size = tb_size; +g_free (buf); +bdrv_dirty_iter_free (dbi); + +return tb; + +fail: +clear_bitmap_table (bs, tb, tb_size); +g_free (buf); +bdrv_dirty_iter_free (dbi); +g_free (tb); + +return NULL ; +} + +/* store_bitmap() +* Store bm->dirty_bitmap to qcow2. +* Set bm->table_offset and bm->table_size accordingly. +*/ +static int store_bitmap (BlockDriverState *bs,Qcow2Bitmap *bm,Error **errp) +{ +int ret; +uint64_t *tb; +int64_t tb_offset; +uint32_t tb_size; +BdrvDirtyBitmap *bitmap = bm->dirty_bitmap; +const char *bm_name; + +assert (bitmap !=NULL ); + + bm_name =bdrv_dirty_bitmap_name (bitmap); + + tb =store_bitmap_data (bs, bitmap, &tb_size, errp); +if (tb ==NULL ) { +return -EINVAL ; + } + +assert (tb_size <=BME_MAX_TABLE_SIZE ); + tb_offset =qcow2_alloc_clusters (bs, tb_size *sizeof (tb[0 ])); +if (tb_offset <0 ) { +error_setg_errno (errp, -tb_offset, +"Failed to allocate clusters for bitmap '%s'" , + bm_name); + goto fail; + } + + ret =qcow2_pre_write_overlap_check (bs,0 , tb_offset, + tb_size *sizeof (tb[0 ])); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Qcow2 overlap check failed" ); + goto fail; + } + +bitmap_table_to_be (tb, tb_size); + ret =bdrv_pwrite (bs->file, tb_offset, tb, tb_size *sizeof (tb[0 ])); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write bitmap '%s' to file" , + bm_name); + goto fail; + } + +g_free (tb); + + bm->table.offset = tb_offset; + bm->table.size = tb_size; + +return 0 ; + +fail: +clear_bitmap_table (bs, tb, tb_size); + +if (tb_offset >0 ) { +qcow2_free_clusters (bs, tb_offset, tb_size *sizeof (tb[0 ]), +QCOW2_DISCARD_OTHER ); + } + +g_free (tb); + +return ret; +} + +static Qcow2Bitmap *find_bitmap_by_name (Qcow2BitmapList *bm_list, +const char *name) +{ +Qcow2Bitmap *bm; + + QSIMPLEQ_FOREACH(bm, bm_list, entry) { +if (strcmp (name, bm->name) ==0 ) { +return bm; + } + } + +return NULL ; +} + +void qcow2_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp) +{ +BdrvDirtyBitmap *bitmap; +BDRVQcow2State *s = bs->opaque; +uint32_t new_nb_bitmaps = s->nb_bitmaps; +uint64_t new_dir_size = s->bitmap_directory_size; +int ret; +Qcow2BitmapList *bm_list; +Qcow2Bitmap *bm; +Qcow2BitmapTableList drop_tables; +Qcow2BitmapTable *tb, *tb_next; + +QSIMPLEQ_INIT (&drop_tables); + +if (!can_write (bs)) { +error_setg (errp,"No write access" ); +return ; + } + +if (!bdrv_has_persistent_bitmaps (bs)) { +/* nothing to do */ +return ; + } + +if (s->nb_bitmaps ==0 ) { + bm_list =bitmap_list_new (); + }else { + bm_list =bitmap_list_load (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); +if (bm_list ==NULL ) { +return ; + } + } + +/* check constraints and names */ +for (bitmap =bdrv_dirty_bitmap_next (bs,NULL ); bitmap !=NULL ; + bitmap =bdrv_dirty_bitmap_next (bs, bitmap)) + { +const char *name =bdrv_dirty_bitmap_name (bitmap); +uint32_t granularity =bdrv_dirty_bitmap_granularity (bitmap); +Qcow2Bitmap *bm; + +if (!bdrv_dirty_bitmap_get_persistance (bitmap)) { +continue ; + } + +if (check_constraints_on_bitmap (bs, name, granularity) <0 ) { +error_setg (errp,"Bitmap '%s' doesn't satisfy the constraints" , + name); + goto fail; + } + + bm =find_bitmap_by_name (bm_list, name); +if (bm ==NULL ) { +if (++new_nb_bitmaps >QCOW2_MAX_BITMAPS ) { +error_setg (errp,"Too many persistent bitmaps" ); + goto fail; + } + + new_dir_size +=calc_dir_entry_size (strlen (name),0 ); +if (new_dir_size >QCOW2_MAX_BITMAP_DIRECTORY_SIZE ) { +error_setg (errp,"Bitmap directory is too large" ); + goto fail; + } + + bm =g_new0 (Qcow2Bitmap,1 ); + bm->name =g_strdup (name); +QSIMPLEQ_INSERT_TAIL (bm_list, bm, entry); + }else { +if (!(bm->flags &BME_FLAG_IN_USE )) { +error_setg (errp,"Bitmap '%s' already exists in the image" , + name); + goto fail; + } + tb =g_memdup (&bm->table,sizeof (bm->table)); + bm->table.offset =0 ; + bm->table.size =0 ; +QSIMPLEQ_INSERT_TAIL (&drop_tables, tb, entry); + } + bm->flags =bdrv_dirty_bitmap_get_autoload (bitmap) ?BME_FLAG_AUTO :0 ; + bm->granularity_bits =ctz32 (bdrv_dirty_bitmap_granularity (bitmap)); + bm->dirty_bitmap = bitmap; + } + +/* allocate clusters and store bitmaps */ + QSIMPLEQ_FOREACH(bm, bm_list, entry) { +if (bm->dirty_bitmap ==NULL ) { +continue ; + } + + ret =store_bitmap (bs, bm, errp); +if (ret <0 ) { + goto fail; + } + } + + ret =update_ext_header_and_dir (bs, bm_list); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to update bitmap extension" ); + goto fail; + } + +/* Bitmap directory was successfully updated, so, old data can be dropped. +* TODO it is better to reuse these clusters */ + QSIMPLEQ_FOREACH_SAFE(tb, &drop_tables, entry, tb_next) { +free_bitmap_clusters (bs, tb); +g_free (tb); + } + +bitmap_list_free (bm_list); +return ; + +fail: + QSIMPLEQ_FOREACH(bm, bm_list, entry) { +if (bm->dirty_bitmap ==NULL || bm->table.offset ==0 ) { +continue ; + } + +free_bitmap_clusters (bs, &bm->table); + } + + QSIMPLEQ_FOREACH_SAFE(tb, &drop_tables, entry, tb_next) { +g_free (tb); + } + +bitmap_list_free (bm_list); +}
block/qcow2.c +1 -0
@@ -3540,6 +3540,7 @@ BlockDriver bdrv_qcow2 = { .bdrv_attach_aio_context = qcow2_attach_aio_context, .bdrv_load_autoloading_dirty_bitmaps = qcow2_load_autoloading_dirty_bitmaps, + .bdrv_store_persistent_dirty_bitmaps = qcow2_store_persistent_dirty_bitmaps, };static void bdrv_qcow2_init (void )
block/qcow2.h +1 -0
@@ -615,5 +615,6 @@ void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table);/* qcow2-bitmap.c functions */ void qcow2_load_autoloading_dirty_bitmaps (BlockDriverState *bs,Error **errp); +void qcow2_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp);#endif
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 13/25] qcow2: add .bdrv_store_persistent_dirty_bitmaps()
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Realize block bitmap storing interface, to allow qcow2 images store persistent bitmaps.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>
PATCH 14/25 block: add bdrv_can_store_new_dirty_bitmap
In reply to Vladimir Sementsov-OgievskiyThis will be needed to check some restrictions before making bitmap persistent in qmp-block-dirty-bitmap-add (this functionality will be added by future patch)
--- block.c | 22 ++++++++++++++++++++++ include/block/block.h | 2 ++ include/block/block_int.h | 4 ++++ 3 files changed, 28 insertions(+)
block.c +22 -0
@@ -4139,3 +4139,25 @@ void bdrv_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp) drv->bdrv_store_persistent_dirty_bitmaps (bs, errp); } + +bool bdrv_can_store_new_dirty_bitmap (BlockDriverState *bs,const char *name, +uint32_t granularity,Error **errp) +{ +BlockDriver *drv = bs->drv; + +if (!drv) { +error_setg_errno (errp,ENOMEDIUM , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return false; + } + +if (!drv->bdrv_can_store_new_dirty_bitmap) { +error_setg_errno (errp,ENOTSUP , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return false; + } + +return drv->bdrv_can_store_new_dirty_bitmap (bs, name, granularity, errp); +}
include/block/block.h +2 -0
@@ -553,5 +553,7 @@ void bdrv_del_child(BlockDriverState *parent, BdrvChild *child, Error **errp);void bdrv_load_autoloading_dirty_bitmaps (BlockDriverState *bs,Error **errp);void bdrv_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp); +bool bdrv_can_store_new_dirty_bitmap (BlockDriverState *bs,const char *name, +uint32_t granularity,Error **errp);#endif
include/block/block_int.h +4 -0
@@ -324,6 +324,10 @@ struct BlockDriver { Error **errp);void (*bdrv_store_persistent_dirty_bitmaps)(BlockDriverState *bs,Error **errp); +bool (*bdrv_can_store_new_dirty_bitmap)(BlockDriverState *bs, +const char *name, +uint32_t granularity, +Error **errp);QLIST_ENTRY (BlockDriver) list; };
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 14/25] block: add bdrv_can_store_new_dirty_bitmap
Quoted 60 lines · Vladimir Sementsov-Ogievskiy
This will be needed to check some restrictions before making bitmap persistent in qmp-block-dirty-bitmap-add (this functionality will be added by future patch)
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>--- block.c | 22 ++++++++++++++++++++++ include/block/block.h | 2 ++ include/block/block_int.h | 4 ++++ 3 files changed, 28 insertions(+)block.c +22
-0@@ -4139,3 +4139,25 @@ void bdrv_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp) drv->bdrv_store_persistent_dirty_bitmaps (bs, errp); } + +bool bdrv_can_store_new_dirty_bitmap (BlockDriverState *bs,const char *name, +uint32_t granularity,Error **errp) +{ +BlockDriver *drv = bs->drv; + +if (!drv) { +error_setg_errno (errp,ENOMEDIUM , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return false; + } + +if (!drv->bdrv_can_store_new_dirty_bitmap) { +error_setg_errno (errp,ENOTSUP , +"Can't store persistent bitmaps to %s" , +bdrv_get_device_or_node_name (bs)); +return false; + } + +return drv->bdrv_can_store_new_dirty_bitmap (bs, name, granularity, errp); +}include/block/block.h +2
-0@@ -553,5 +553,7 @@ void bdrv_del_child(BlockDriverState *parent, BdrvChild *child, Error **errp);void bdrv_load_autoloading_dirty_bitmaps (BlockDriverState *bs,Error **errp);void bdrv_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp); +bool bdrv_can_store_new_dirty_bitmap (BlockDriverState *bs,const char *name, +uint32_t granularity,Error **errp);#endif include/block/block_int.h +0
-0
include/block/block_int.h struct BlockDriver { · line 333
Quoted 10 lines
include/block/block_int.h +4
-0@@ -324,6 +324,10 @@ struct BlockDriver { Error **errp);void (*bdrv_store_persistent_dirty_bitmaps)(BlockDriverState *bs,Error **errp); +bool (*bdrv_can_store_new_dirty_bitmap)(BlockDriverState *bs, +const char *name, +uint32_t granularity, +Error **errp);QLIST_ENTRY (BlockDriver) list; };
PATCH 15/25 qcow2: add .bdrv_can_store_new_dirty_bitmap
In reply to Vladimir Sementsov-OgievskiyRealize .bdrv_can_store_new_dirty_bitmap interface.
--- block/qcow2-bitmap.c | 40 ++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 1 + block/qcow2.h | 4 ++++ 3 files changed, 45 insertions(+)
block/qcow2-bitmap.c +40 -0
@@ -1182,3 +1182,43 @@ fail:bitmap_list_free (bm_list); } + +bool qcow2_can_store_new_dirty_bitmap (BlockDriverState *bs, +const char *name, +uint32_t granularity, +Error **errp) +{ +BDRVQcow2State *s = bs->opaque; +const char *reason =NULL ; +bool found; +Qcow2BitmapList *bm_list; + +if (check_constraints_on_bitmap (bs, name, granularity) !=0 ) { + reason ="it doesn't satisfy the constraints" ; + goto common_errp; + } + +if (s->nb_bitmaps ==0 ) { +return true; + } + + bm_list =bitmap_list_load (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); +if (bm_list ==NULL ) { +return false; + } + + found =find_bitmap_by_name (bm_list, name); +bitmap_list_free (bm_list); +if (found) { + reason ="bitmap with the same name is already stored" ; + goto common_errp; + } + +return true; + +common_errp: +error_setg (errp,"Can't make bitmap '%s' persistent in '%s', as %s." , + name,bdrv_get_device_or_node_name (bs), reason); +return false; +}
block/qcow2.c +1 -0
@@ -3541,6 +3541,7 @@ BlockDriver bdrv_qcow2 = { .bdrv_load_autoloading_dirty_bitmaps = qcow2_load_autoloading_dirty_bitmaps, .bdrv_store_persistent_dirty_bitmaps = qcow2_store_persistent_dirty_bitmaps, + .bdrv_can_store_new_dirty_bitmap = qcow2_can_store_new_dirty_bitmap, };static void bdrv_qcow2_init (void )
block/qcow2.h +4 -0
@@ -616,5 +616,9 @@ void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table);/* qcow2-bitmap.c functions */ void qcow2_load_autoloading_dirty_bitmaps (BlockDriverState *bs,Error **errp);void qcow2_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp); +bool qcow2_can_store_new_dirty_bitmap (BlockDriverState *bs, +const char *name, +uint32_t granularity, +Error **errp);#endif
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 15/25] qcow2: add .bdrv_can_store_new_dirty_bitmap
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Realize .bdrv_can_store_new_dirty_bitmap interface.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>--- block/qcow2-bitmap.c | 40 ++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 1 + block/qcow2.h | 4 ++++ 3 files changed, 45 insertions(+)block/qcow2-bitmap.c +5
-0@@ -1182,3 +1182,43 @@ fail:bitmap_list_free (bm_list); } + +bool qcow2_can_store_new_dirty_bitmap (BlockDriverState *bs, +const char *name, +uint32_t granularity, +Error **errp)
block/qcow2-bitmap.c fail: · line 1199
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+{ +BDRVQcow2State *s = bs->opaque; +const char *reason =NULL ; +bool found; +Qcow2BitmapList *bm_list; + +if (check_constraints_on_bitmap (bs, name, granularity) !=0 ) { + reason ="it doesn't satisfy the constraints" ; + goto common_errp; + }
"the constraints" are still entirely opaque to the caller, it may not be obvious that it's because of the name length.
(But maybe you fix this in patch 25.)
block/qcow2-bitmap.c fail: · line 1209
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ +if (s->nb_bitmaps ==0 ) { +return true; + } + + bm_list =bitmap_list_load (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); +if (bm_list ==NULL ) { +return false; + }
Another opportunity for caching the bm_list.
Quoted 5 lines
block/qcow2-bitmap.c +5
-0+ + found =find_bitmap_by_name (bm_list, name); +bitmap_list_free (bm_list); +if (found) { + reason ="bitmap with the same name is already stored" ;
block/qcow2-bitmap.c fail: · line 1224
Quoted 10 lines
block/qcow2-bitmap.c +10
-0+ goto common_errp; + } + +return true; + +common_errp: +error_setg (errp,"Can't make bitmap '%s' persistent in '%s', as %s." , + name,bdrv_get_device_or_node_name (bs), reason); +return false; +}
We don't check total numbers of bitmaps here? (I'll keep reading ahead in the series.)
Quoted 18 lines
block/qcow2.c +1
-0@@ -3541,6 +3541,7 @@ BlockDriver bdrv_qcow2 = { .bdrv_load_autoloading_dirty_bitmaps = qcow2_load_autoloading_dirty_bitmaps, .bdrv_store_persistent_dirty_bitmaps = qcow2_store_persistent_dirty_bitmaps, + .bdrv_can_store_new_dirty_bitmap = qcow2_can_store_new_dirty_bitmap, };static void bdrv_qcow2_init (void )block/qcow2.h +0
-0
block/qcow2.h void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table); · line 624
Quoted 9 lines
block/qcow2.h +4
-0@@ -616,5 +616,9 @@ void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table);/* qcow2-bitmap.c functions */ void qcow2_load_autoloading_dirty_bitmaps (BlockDriverState *bs,Error **errp);void qcow2_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp); +bool qcow2_can_store_new_dirty_bitmap (BlockDriverState *bs, +const char *name, +uint32_t granularity, +Error **errp);#endif
Benefit of the doubt:
PATCH 16/25 qmp: add persistent flag to block-dirty-bitmap-add
In reply to Vladimir Sementsov-OgievskiyAdd optional 'persistent' flag to qmp command block-dirty-bitmap-add. Default is false.
--- blockdev.c | 18 +++++++++++++++++- qapi/block-core.json | 8 +++++++- 2 files changed, 24 insertions(+), 2 deletions(-)
blockdev.c +17 -1
@@ -1967,6 +1967,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,/* AIO context taken and released within qmp_block_dirty_bitmap_add */ qmp_block_dirty_bitmap_add (action->node, action->name, action->has_granularity, action->granularity, + action->has_persistent, action->persistent, &local_err);if (!local_err) {
@@ -2696,10 +2697,12 @@ out:void qmp_block_dirty_bitmap_add (const char *node,const char *name,bool has_granularity,uint32_t granularity, +bool has_persistent,bool persistent,Error **errp) {AioContext *aio_context;BlockDriverState *bs; +BdrvDirtyBitmap *bitmap;if (!name || name[0 ] =='\0' ) {error_setg (errp,"Bitmap name cannot be empty" );
@@ -2725,7 +2728,20 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name, granularity =bdrv_get_default_bitmap_granularity (bs); } -bdrv_create_dirty_bitmap (bs, granularity, name, errp); +if (!has_persistent) { + persistent = false; + } + +if (persistent && + !bdrv_can_store_new_dirty_bitmap (bs, name, granularity, errp)) + { + goto out; + } + + bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); +if (bitmap !=NULL ) { +bdrv_dirty_bitmap_set_persistance (bitmap, persistent); + } out:aio_context_release (aio_context);
qapi/block-core.json +7 -1
@@ -1559,10 +1559,16 @@ # @granularity: #optional the bitmap granularity, default is 64k for # block-dirty-bitmap-add # +# @persistent: #optional the bitmap is persistent, i.e. it will be saved to the +# corresponding block device image file on its close. For now only +# Qcow2 disks support persistent bitmaps. Default is false. +# (Since 2.9) +# # Since: 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', - 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32' } } + 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', + '*persistent': 'bool' } } ## # @block-dirty-bitmap-add:
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 16/25] qmp: add persistent flag to block-dirty-bitmap-add
Quoted 41 lines · Vladimir Sementsov-Ogievskiy
Add optional 'persistent' flag to qmp command block-dirty-bitmap-add. Default is false.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>Reviewed-by Max Reitz <[email protected]>--- blockdev.c | 18 +++++++++++++++++- qapi/block-core.json | 8 +++++++- 2 files changed, 24 insertions(+), 2 deletions(-)blockdev.c +3
-1@@ -1967,6 +1967,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,/* AIO context taken and released within qmp_block_dirty_bitmap_add */ qmp_block_dirty_bitmap_add (action->node, action->name, action->has_granularity, action->granularity, + action->has_persistent, action->persistent, &local_err);if (!local_err) {@@ -2696,10 +2697,12 @@ out:void qmp_block_dirty_bitmap_add (const char *node,const char *name,bool has_granularity,uint32_t granularity, +bool has_persistent,bool persistent,Error **errp) {AioContext *aio_context;BlockDriverState *bs; +BdrvDirtyBitmap *bitmap;if (!name || name[0 ] =='\0' ) {error_setg (errp,"Bitmap name cannot be empty" );@@ -2725,7 +2728,20 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name, granularity =bdrv_get_default_bitmap_granularity (bs); } -bdrv_create_dirty_bitmap (bs, granularity, name, errp);
blockdev.c void qmp_block_dirty_bitmap_add(const char *node, const char *name, · line 2740
Quoted 10 lines
blockdev.c +10
-0+if (!has_persistent) { + persistent = false; + } + +if (persistent && + !bdrv_can_store_new_dirty_bitmap (bs, name, granularity, errp)) + { + goto out; + } +
Yeah, I don't see where you're checking against MAX_BITMAPS in the qcow2 implementation for can_store.
Quoted 20 lines
blockdev.c +4
-0+ bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); +if (bitmap !=NULL ) { +bdrv_dirty_bitmap_set_persistance (bitmap, persistent); + } out:aio_context_release (aio_context);qapi/block-core.json +4
-0@@ -1559,10 +1559,16 @@ # @granularity: #optional the bitmap granularity, default is 64k for # block-dirty-bitmap-add # +# @persistent: #optional the bitmap is persistent, i.e. it will be saved to the +# corresponding block device image file on its close. For now only +# Qcow2 disks support persistent bitmaps. Default is false. +# (Since 2.9)
qapi/block-core.json line 1574
Quoted 10 lines
qapi/block-core.json +3
-1+# # Since: 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', - 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32' } } + 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', + '*persistent': 'bool' } } ## # @block-dirty-bitmap-add:
Looks good otherwise.
PATCH 17/25 qmp: add autoload parameter to block-dirty-bitmap-add
In reply to Vladimir Sementsov-OgievskiyOptional. Default is false.
--- blockdev.c | 18 ++++++++++++++++-- qapi/block-core.json | 6 +++++- 2 files changed, 21 insertions(+), 3 deletions(-)
blockdev.c +16 -2
@@ -1968,6 +1968,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,qmp_block_dirty_bitmap_add (action->node, action->name, action->has_granularity, action->granularity, action->has_persistent, action->persistent, + action->has_autoload, action->autoload, &local_err);if (!local_err) {
@@ -2698,6 +2699,7 @@ out:void qmp_block_dirty_bitmap_add (const char *node,const char *name,bool has_granularity,uint32_t granularity,bool has_persistent,bool persistent, +bool has_autoload,bool autoload,Error **errp) {AioContext *aio_context;
@@ -2731,6 +2733,15 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,if (!has_persistent) { persistent = false; } +if (!has_autoload) { + autoload = false; + } + +if (has_autoload && !persistent) { +error_setg (errp,"Autoload flag must be used only for persistent " +"bitmaps" ); + goto out; + }if (persistent && !bdrv_can_store_new_dirty_bitmap (bs, name, granularity, errp))
@@ -2739,10 +2750,13 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name, } bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); -if (bitmap !=NULL ) { -bdrv_dirty_bitmap_set_persistance (bitmap, persistent); +if (bitmap ==NULL ) { + goto out; } +bdrv_dirty_bitmap_set_persistance (bitmap, persistent); +bdrv_dirty_bitmap_set_autoload (bitmap, autoload); + out:aio_context_release (aio_context); }
qapi/block-core.json +5 -1
@@ -1564,11 +1564,15 @@ # Qcow2 disks support persistent bitmaps. Default is false. # (Since 2.9) # +# @autoload: #optional the bitmap will be automatically loaded when the image +# it is stored in is opened. This flag may only be specified for +# persistent bitmaps. Default is false. (Since 2.9) +# # Since: 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', - '*persistent': 'bool' } } + '*persistent': 'bool', '*autoload': 'bool' } } ## # @block-dirty-bitmap-add:
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 17/25] qmp: add autoload parameter to block-dirty-bitmap-add
Quoted 34 lines · Vladimir Sementsov-Ogievskiy
Optional. Default is false.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>Reviewed-by Max Reitz <[email protected]>--- blockdev.c | 18 ++++++++++++++++-- qapi/block-core.json | 6 +++++- 2 files changed, 21 insertions(+), 3 deletions(-)blockdev.c +2
-0@@ -1968,6 +1968,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,qmp_block_dirty_bitmap_add (action->node, action->name, action->has_granularity, action->granularity, action->has_persistent, action->persistent, + action->has_autoload, action->autoload, &local_err);if (!local_err) {@@ -2698,6 +2699,7 @@ out:void qmp_block_dirty_bitmap_add (const char *node,const char *name,bool has_granularity,uint32_t granularity,bool has_persistent,bool persistent, +bool has_autoload,bool autoload,Error **errp) {AioContext *aio_context;@@ -2731,6 +2733,15 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,if (!has_persistent) { persistent = false; }
blockdev.c void qmp_block_dirty_bitmap_add(const char *node, const char *name, · line 2745
Quoted 10 lines
blockdev.c +9
-0+if (!has_autoload) { + autoload = false; + } + +if (has_autoload && !persistent) { +error_setg (errp,"Autoload flag must be used only for persistent " +"bitmaps" ); + goto out; + }
Good, glad we have this case covered.
Quoted 30 lines
blockdev.c +5
-2if (persistent && !bdrv_can_store_new_dirty_bitmap (bs, name, granularity, errp))@@ -2739,10 +2750,13 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name, } bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp); -if (bitmap !=NULL ) { -bdrv_dirty_bitmap_set_persistance (bitmap, persistent); +if (bitmap ==NULL ) { + goto out; } +bdrv_dirty_bitmap_set_persistance (bitmap, persistent); +bdrv_dirty_bitmap_set_autoload (bitmap, autoload); + out:aio_context_release (aio_context); }qapi/block-core.json +3
-0@@ -1564,11 +1564,15 @@ # Qcow2 disks support persistent bitmaps. Default is false. # (Since 2.9) # +# @autoload: #optional the bitmap will be automatically loaded when the image +# it is stored in is opened. This flag may only be specified for +# persistent bitmaps. Default is false. (Since 2.9)
qapi/block-core.json line 1578
Quoted 10 lines
qapi/block-core.json +2
-1+# # Since: 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', - '*persistent': 'bool' } } + '*persistent': 'bool', '*autoload': 'bool' } } ## # @block-dirty-bitmap-add:
Looks good to me.
PATCH 18/25 qmp: add x-debug-block-dirty-bitmap-sha256
In reply to Vladimir Sementsov-Ogievskiy--- block/dirty-bitmap.c | 5 +++++ blockdev.c | 29 +++++++++++++++++++++++++++++ include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 8 ++++++++ qapi/block-core.json | 27 +++++++++++++++++++++++++++ tests/Makefile.include | 2 +- util/hbitmap.c | 11 +++++++++++ 7 files changed, 83 insertions(+), 1 deletion(-)
block/dirty-bitmap.c +5 -0
@@ -588,3 +588,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,return bitmap ==NULL ?QLIST_FIRST (&bs->dirty_bitmaps) :QLIST_NEXT (bitmap, list); } + +char *bdrv_dirty_bitmap_sha256 (const BdrvDirtyBitmap *bitmap,Error **errp) +{ +return hbitmap_sha256 (bitmap->bitmap, errp); +}
blockdev.c +29 -0
@@ -2820,6 +2820,35 @@ void qmp_block_dirty_bitmap_clear(const char *node, const char *name,aio_context_release (aio_context); } +BlockDirtyBitmapSha256 *qmp_x_debug_block_dirty_bitmap_sha256 (const char *node, +const char *name, +Error **errp) +{ +AioContext *aio_context; +BdrvDirtyBitmap *bitmap; +BlockDriverState *bs; +BlockDirtyBitmapSha256 *ret =NULL ; +char *sha256; + + bitmap =block_dirty_bitmap_lookup (node, name, &bs, &aio_context, errp); +if (!bitmap || !bs) { +return NULL ; + } + + sha256 =bdrv_dirty_bitmap_sha256 (bitmap, errp); +if (sha256 ==NULL ) { + goto out; + } + + ret =g_new (BlockDirtyBitmapSha256,1 ); + ret->sha256 = sha256; + +out: +aio_context_release (aio_context); + +return ret; +} +void hmp_drive_del (Monitor *mon,const QDict *qdict) {const char *id =qdict_get_str (qdict,"id" );
include/block/dirty-bitmap.h +2 -0
@@ -86,4 +86,6 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,bool bdrv_has_persistent_bitmaps (BlockDriverState *bs); +char *bdrv_dirty_bitmap_sha256 (const BdrvDirtyBitmap *bitmap,Error **errp); +#endif
include/qemu/hbitmap.h +8 -0
@@ -253,6 +253,14 @@ void hbitmap_deserialize_ones(HBitmap *hb, uint64_t start, uint64_t count,void hbitmap_deserialize_finish (HBitmap *hb);/** +* hbitmap_sha256: +* @bitmap: HBitmap to operate on. +* +* Returns SHA256 hash of the last level. +*/ +char *hbitmap_sha256 (const HBitmap *bitmap,Error **errp); + +/** * hbitmap_free: * @hb: HBitmap to operate on. *
qapi/block-core.json +27 -0
@@ -1642,6 +1642,33 @@ 'data': 'BlockDirtyBitmap' } ## +# @BlockDirtyBitmapSha256: +# +# SHA256 hash of dirty bitmap data +# +# @sha256: ASCII representation of SHA256 bitmap hash +# +# Since: 2.9 +## + { 'struct': 'BlockDirtyBitmapSha256', + 'data': {'sha256': 'str'} } + +## +# @x-debug-block-dirty-bitmap-sha256: +# +# Get bitmap SHA256 +# +# Returns: BlockDirtyBitmapSha256 on success +# If @node is not a valid block device, DeviceNotFound +# If @name is not found or if hashing has failed, GenericError with an +# explanation +# +# Since: 2.9 +## + { 'command': 'x-debug-block-dirty-bitmap-sha256', + 'data': 'BlockDirtyBitmap', 'returns': 'BlockDirtyBitmapSha256' } + +## # @blockdev-mirror: # # Start mirroring a block device's writes to a new destination.
tests/Makefile.include +1 -1
@@ -519,7 +519,7 @@ tests/test-blockjob$(EXESUF): tests/test-blockjob.o $(test-block-obj-y) $(test-u tests/test-blockjob-txn$(EXESUF): tests/test-blockjob-txn.o $(test-block-obj-y) $(test-util-obj-y) tests/test-thread-pool$(EXESUF): tests/test-thread-pool.o $(test-block-obj-y) tests/test-iov$(EXESUF): tests/test-iov.o $(test-util-obj-y) -tests/test-hbitmap$(EXESUF): tests/test-hbitmap.o $(test-util-obj-y) +tests/test-hbitmap$(EXESUF): tests/test-hbitmap.o $(test-util-obj-y) $(test-crypto-obj-y) tests/test-x86-cpuid$(EXESUF): tests/test-x86-cpuid.o tests/test-xbzrle$(EXESUF): tests/test-xbzrle.o migration/xbzrle.o page_cache.o $(test-util-obj-y) tests/test-cutils$(EXESUF): tests/test-cutils.o util/cutils.o
util/hbitmap.c +11 -0
@@ -13,6 +13,7 @@#include "qemu/hbitmap.h" #include "qemu/host-utils.h" #include "trace.h" +#include "crypto/hash.h" /* HBitmaps provides an array of bits. The bits are stored as usual in an * array of unsigned longs, but HBitmap is also optimized to provide fast
@@ -727,3 +728,13 @@ void hbitmap_free_meta(HBitmap *hb)hbitmap_free (hb->meta); hb->meta =NULL ; } + +char *hbitmap_sha256 (const HBitmap *bitmap,Error **errp) +{ +size_t size = bitmap->sizes[HBITMAP_LEVELS -1 ] *sizeof (unsigned long ); +char *data = (char *)bitmap->levels[HBITMAP_LEVELS -1 ]; +char *hash =NULL ; +qcrypto_hash_digest (QCRYPTO_HASH_ALG_SHA256 , data, size, &hash, errp); + +return hash; +}
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 18/25] qmp: add x-debug-block-dirty-bitmap-sha256
Quoted 163 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>--- block/dirty-bitmap.c | 5 +++++ blockdev.c | 29 +++++++++++++++++++++++++++++ include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 8 ++++++++ qapi/block-core.json | 27 +++++++++++++++++++++++++++ tests/Makefile.include | 2 +- util/hbitmap.c | 11 +++++++++++ 7 files changed, 83 insertions(+), 1 deletion(-)block/dirty-bitmap.c +5
-0@@ -588,3 +588,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,return bitmap ==NULL ?QLIST_FIRST (&bs->dirty_bitmaps) :QLIST_NEXT (bitmap, list); } + +char *bdrv_dirty_bitmap_sha256 (const BdrvDirtyBitmap *bitmap,Error **errp) +{ +return hbitmap_sha256 (bitmap->bitmap, errp); +}blockdev.c +29
-0@@ -2820,6 +2820,35 @@ void qmp_block_dirty_bitmap_clear(const char *node, const char *name,aio_context_release (aio_context); } +BlockDirtyBitmapSha256 *qmp_x_debug_block_dirty_bitmap_sha256 (const char *node, +const char *name, +Error **errp) +{ +AioContext *aio_context; +BdrvDirtyBitmap *bitmap; +BlockDriverState *bs; +BlockDirtyBitmapSha256 *ret =NULL ; +char *sha256; + + bitmap =block_dirty_bitmap_lookup (node, name, &bs, &aio_context, errp); +if (!bitmap || !bs) { +return NULL ; + } + + sha256 =bdrv_dirty_bitmap_sha256 (bitmap, errp); +if (sha256 ==NULL ) { + goto out; + } + + ret =g_new (BlockDirtyBitmapSha256,1 ); + ret->sha256 = sha256; + +out: +aio_context_release (aio_context); + +return ret; +} +void hmp_drive_del (Monitor *mon,const QDict *qdict) {const char *id =qdict_get_str (qdict,"id" );include/block/dirty-bitmap.h +2
-0@@ -86,4 +86,6 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,bool bdrv_has_persistent_bitmaps (BlockDriverState *bs); +char *bdrv_dirty_bitmap_sha256 (const BdrvDirtyBitmap *bitmap,Error **errp); +#endif include/qemu/hbitmap.h +8
-0@@ -253,6 +253,14 @@ void hbitmap_deserialize_ones(HBitmap *hb, uint64_t start, uint64_t count,void hbitmap_deserialize_finish (HBitmap *hb);/** +* hbitmap_sha256: +* @bitmap: HBitmap to operate on. +* +* Returns SHA256 hash of the last level. +*/ +char *hbitmap_sha256 (const HBitmap *bitmap,Error **errp); + +/** * hbitmap_free: * @hb: HBitmap to operate on. * qapi/block-core.json +27
-0@@ -1642,6 +1642,33 @@ 'data': 'BlockDirtyBitmap' } ## +# @BlockDirtyBitmapSha256: +# +# SHA256 hash of dirty bitmap data +# +# @sha256: ASCII representation of SHA256 bitmap hash +# +# Since: 2.9 +## + { 'struct': 'BlockDirtyBitmapSha256', + 'data': {'sha256': 'str'} } + +## +# @x-debug-block-dirty-bitmap-sha256: +# +# Get bitmap SHA256 +# +# Returns: BlockDirtyBitmapSha256 on success +# If @node is not a valid block device, DeviceNotFound +# If @name is not found or if hashing has failed, GenericError with an +# explanation +# +# Since: 2.9 +## + { 'command': 'x-debug-block-dirty-bitmap-sha256', + 'data': 'BlockDirtyBitmap', 'returns': 'BlockDirtyBitmapSha256' } + +## # @blockdev-mirror: # # Start mirroring a block device's writes to a new destination.tests/Makefile.include +1
-1@@ -519,7 +519,7 @@ tests/test-blockjob$(EXESUF): tests/test-blockjob.o $(test-block-obj-y) $(test-u tests/test-blockjob-txn$(EXESUF): tests/test-blockjob-txn.o $(test-block-obj-y) $(test-util-obj-y) tests/test-thread-pool$(EXESUF): tests/test-thread-pool.o $(test-block-obj-y) tests/test-iov$(EXESUF): tests/test-iov.o $(test-util-obj-y) -tests/test-hbitmap$(EXESUF): tests/test-hbitmap.o $(test-util-obj-y) +tests/test-hbitmap$(EXESUF): tests/test-hbitmap.o $(test-util-obj-y) $(test-crypto-obj-y) tests/test-x86-cpuid$(EXESUF): tests/test-x86-cpuid.o tests/test-xbzrle$(EXESUF): tests/test-xbzrle.o migration/xbzrle.o page_cache.o $(test-util-obj-y) tests/test-cutils$(EXESUF): tests/test-cutils.o util/cutils.outil/hbitmap.c +1
-0@@ -13,6 +13,7 @@#include "qemu/hbitmap.h" #include "qemu/host-utils.h" #include "trace.h" +#include "crypto/hash.h" /* HBitmaps provides an array of bits. The bits are stored as usual in an * array of unsigned longs, but HBitmap is also optimized to provide fast @@ -727,3 +728,13 @@ void hbitmap_free_meta(HBitmap *hb)hbitmap_free (hb->meta); hb->meta =NULL ; }
util/hbitmap.c void hbitmap_free_meta(HBitmap *hb) · line 740
Quoted 10 lines
util/hbitmap.c +10
-0+ +char *hbitmap_sha256 (const HBitmap *bitmap,Error **errp) +{ +size_t size = bitmap->sizes[HBITMAP_LEVELS -1 ] *sizeof (unsigned long ); +char *data = (char *)bitmap->levels[HBITMAP_LEVELS -1 ]; +char *hash =NULL ; +qcrypto_hash_digest (QCRYPTO_HASH_ALG_SHA256 , data, size, &hash, errp); + +return hash; +}
I've been fine with this ability as a testing implement since you began writing this series, but what I wonder is if there is some way that we can disable this comment for non-testing builds in some kind of formal way.
The x-prefix is a good hint that we don't want people using this, but it would be nice if we could actually further hide this interface from prying eyes.
Markus, do you have any thoughts on the matter?
Regardless, I am fine with this as-is, so:
PATCH 19/25 iotests: test qcow2 persistent dirty bitmap
In reply to Vladimir Sementsov-Ogievskiy--- tests/qemu-iotests/165 | 89 ++++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/165.out | 5 +++ tests/qemu-iotests/group | 1 + 3 files changed, 95 insertions(+) create mode 100755 tests/qemu-iotests/165 create mode 100644 tests/qemu-iotests/165.out
tests/qemu-iotests/165 new file +89 -0
@@ -0,0 +1,89 @@ +#!/usr/bin/env python +# +# Tests for persistent dirty bitmaps. +# +# Copyright: Vladimir Sementsov-Ogievskiy 2015-2017 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import iotests +from iotests import qemu_img + +disk = os.path.join(iotests.test_dir, 'disk') +disk_size = 0x40000000 # 1G + +# regions for qemu_io: (start, count) in bytes +regions1 = ((0, 0x100000), + (0x200000, 0x100000)) + +regions2 = ((0x10000000, 0x20000), + (0x3fff0000, 0x10000)) + +class TestPersistentDirtyBitmap(iotests.QMPTestCase): + + def setUp(self): + qemu_img('create', '-f', iotests.imgfmt, disk, str(disk_size)) + + def tearDown(self): + os.remove(disk) + + def mkVm(self): + return iotests.VM().add_drive(disk) + + def getSha256(self): + result = self.vm.qmp('x-debug-block-dirty-bitmap-sha256', + node='drive0', name='bitmap0') + return result['return']['sha256'] + + def checkBitmap(self, sha256): + result = self.vm.qmp('x-debug-block-dirty-bitmap-sha256', + node='drive0', name='bitmap0') + self.assert_qmp(result, 'return/sha256', sha256); + + def writeRegions(self, regions): + for r in regions: + self.vm.hmp_qemu_io('drive0', + 'write %d %d' % r) + + def qmpAddBitmap(self): + self.vm.qmp('block-dirty-bitmap-add', node='drive0', + name='bitmap0', persistent=True, autoload=True) + + def test_persistent(self): + self.vm = self.mkVm() + self.vm.launch() + self.qmpAddBitmap() + + self.writeRegions(regions1) + sha256 = self.getSha256() + + self.vm.shutdown() + self.vm = self.mkVm() + self.vm.launch() + + self.checkBitmap(sha256) + self.writeRegions(regions2) + sha256 = self.getSha256() + + self.vm.shutdown() + self.vm.launch() + + self.checkBitmap(sha256) + + self.vm.shutdown() + +if __name__ == '__main__': + iotests.main(supported_fmts=['qcow2'])
tests/qemu-iotests/165.out new file +5 -0
@@ -0,0 +1,5 @@ +. +---------------------------------------------------------------------- +Ran 1 tests + +OK
tests/qemu-iotests/group +1 -0
@@ -162,6 +162,7 @@ 159 rw auto quick 160 rw auto quick 162 auto quick +165 rw auto quick 170 rw auto quick 171 rw auto quick 172 auto
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 19/25] iotests: test qcow2 persistent dirty bitmap
Quoted 122 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>--- tests/qemu-iotests/165 | 89 ++++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/165.out | 5 +++ tests/qemu-iotests/group | 1 + 3 files changed, 95 insertions(+) create mode 100755 tests/qemu-iotests/165 create mode 100644 tests/qemu-iotests/165.outtests/qemu-iotests/165 new file +89
-0@@ -0,0 +1,89 @@ +#!/usr/bin/env python +# +# Tests for persistent dirty bitmaps. +# +# Copyright: Vladimir Sementsov-Ogievskiy 2015-2017 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import iotests +from iotests import qemu_img + +disk = os.path.join(iotests.test_dir, 'disk') +disk_size = 0x40000000 # 1G + +# regions for qemu_io: (start, count) in bytes +regions1 = ((0, 0x100000), + (0x200000, 0x100000)) + +regions2 = ((0x10000000, 0x20000), + (0x3fff0000, 0x10000)) + +class TestPersistentDirtyBitmap(iotests.QMPTestCase): + + def setUp(self): + qemu_img('create', '-f', iotests.imgfmt, disk, str(disk_size)) + + def tearDown(self): + os.remove(disk) + + def mkVm(self): + return iotests.VM().add_drive(disk) + + def getSha256(self): + result = self.vm.qmp('x-debug-block-dirty-bitmap-sha256', + node='drive0', name='bitmap0') + return result['return']['sha256'] + + def checkBitmap(self, sha256): + result = self.vm.qmp('x-debug-block-dirty-bitmap-sha256', + node='drive0', name='bitmap0') + self.assert_qmp(result, 'return/sha256', sha256); + + def writeRegions(self, regions): + for r in regions: + self.vm.hmp_qemu_io('drive0', + 'write %d %d' % r) + + def qmpAddBitmap(self): + self.vm.qmp('block-dirty-bitmap-add', node='drive0', + name='bitmap0', persistent=True, autoload=True) + + def test_persistent(self): + self.vm = self.mkVm() + self.vm.launch() + self.qmpAddBitmap() + + self.writeRegions(regions1) + sha256 = self.getSha256() + + self.vm.shutdown() + self.vm = self.mkVm() + self.vm.launch() + + self.checkBitmap(sha256) + self.writeRegions(regions2) + sha256 = self.getSha256() + + self.vm.shutdown() + self.vm.launch() + + self.checkBitmap(sha256) + + self.vm.shutdown() + +if __name__ == '__main__': + iotests.main(supported_fmts=['qcow2'])tests/qemu-iotests/165.out new file +5
-0@@ -0,0 +1,5 @@ +. +---------------------------------------------------------------------- +Ran 1 tests + +OKtests/qemu-iotests/group +0
-0
tests/qemu-iotests/group line 168
Quoted 7 lines
tests/qemu-iotests/group +1
-0@@ -162,6 +162,7 @@ 159 rw auto quick 160 rw auto quick 162 auto quick +165 rw auto quick 170 rw auto quick 171 rw auto quick 172 auto
Good enough for a basic test for now.
We'll want to test the error cases of the persistence and autoload flags eventually.
PATCH 20/25 qcow2-refcount: rename inc_refcounts() and make it public
In reply to Vladimir Sementsov-OgievskiyThis is needed for the following patch, which will introduce refcounts checking for qcow2 bitmaps.
--- block/qcow2-refcount.c | 53 ++++++++++++++++++++++++++------------------------ block/qcow2.h | 4 ++++ 2 files changed, 32 insertions(+), 25 deletions(-)
block/qcow2-refcount.c +28 -25
@@ -1309,11 +1309,10 @@ static int realloc_refcount_array(BDRVQcow2State *s, void **array,* * Modifies the number of errors in res. */ -static int inc_refcounts (BlockDriverState *bs, -BdrvCheckResult *res, -void **refcount_table, -int64_t *refcount_table_size, -int64_t offset,int64_t size) +int qcow2_inc_refcounts_imrt (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size, +int64_t offset,int64_t size) {BDRVQcow2State *s = bs->opaque;uint64_t start, last, cluster_offset, k, refcount;
@@ -1406,8 +1405,9 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, nb_csectors = ((l2_entry >> s->csize_shift) & s->csize_mask) +1 ; l2_entry &= s->cluster_offset_mask; - ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, - l2_entry & ~511 , nb_csectors *512 ); + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + l2_entry & ~511 , nb_csectors *512 );if (ret <0 ) { goto fail; }
@@ -1445,8 +1445,9 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, }/* Mark cluster as used */ - ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, - offset, s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + offset, s->cluster_size);if (ret <0 ) { goto fail; }
@@ -1498,8 +1499,8 @@ static int check_refcounts_l1(BlockDriverState *bs, l1_size2 = l1_size *sizeof (uint64_t );/* Mark L1 table as used */ - ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, - l1_table_offset, l1_size2); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, refcount_table_size, + l1_table_offset, l1_size2);if (ret <0 ) { goto fail; }
@@ -1528,8 +1529,9 @@ static int check_refcounts_l1(BlockDriverState *bs,if (l2_offset) {/* Mark L2 table as used */ l2_offset &=L1E_OFFSET_MASK ; - ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, - l2_offset, s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + l2_offset, s->cluster_size);if (ret <0 ) { goto fail; }
@@ -1744,14 +1746,15 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, } res->corruptions_fixed++; - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, - offset, s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, nb_clusters, + offset, s->cluster_size);if (ret <0 ) {return ret; }/* No need to check whether the refcount is now greater than 1: * This area was just allocated and zeroed, so it can only be -* exactly 1 after inc_refcounts() */ +* exactly 1 after qcow2_inc_refcounts_imrt() */ continue ; resize_fail:
@@ -1766,8 +1769,8 @@ resize_fail: }if (offset !=0 ) { - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, - offset, s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, nb_clusters, + offset, s->cluster_size);if (ret <0 ) {return ret; }
@@ -1807,8 +1810,8 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, }/* header */ - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, -0 , s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, nb_clusters, +0 , s->cluster_size);if (ret <0 ) {return ret; }
@@ -1829,16 +1832,16 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,return ret; } } - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, - s->snapshots_offset, s->snapshots_size); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, nb_clusters, + s->snapshots_offset, s->snapshots_size);if (ret <0 ) {return ret; }/* refcount data */ - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, - s->refcount_table_offset, - s->refcount_table_size *sizeof (uint64_t )); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, nb_clusters, + s->refcount_table_offset, + s->refcount_table_size *sizeof (uint64_t ));if (ret <0 ) {return ret; }
block/qcow2.h +4 -0
@@ -543,6 +543,10 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, int64_t size);int qcow2_pre_write_overlap_check (BlockDriverState *bs,int ign,int64_t offset,int64_t size); +int qcow2_inc_refcounts_imrt (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size, +int64_t offset,int64_t size); int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order, BlockDriverAmendStatusCB *status_cb,
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 20/25] qcow2-refcount: rename inc_refcounts() and make it public
Quoted 146 lines · Vladimir Sementsov-Ogievskiy
This is needed for the following patch, which will introduce refcounts checking for qcow2 bitmaps.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 53 ++++++++++++++++++++++++++------------------------ block/qcow2.h | 4 ++++ 2 files changed, 32 insertions(+), 25 deletions(-)block/qcow2-refcount.c +28
-25@@ -1309,11 +1309,10 @@ static int realloc_refcount_array(BDRVQcow2State *s, void **array,* * Modifies the number of errors in res. */ -static int inc_refcounts (BlockDriverState *bs, -BdrvCheckResult *res, -void **refcount_table, -int64_t *refcount_table_size, -int64_t offset,int64_t size) +int qcow2_inc_refcounts_imrt (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size, +int64_t offset,int64_t size) {BDRVQcow2State *s = bs->opaque;uint64_t start, last, cluster_offset, k, refcount;@@ -1406,8 +1405,9 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, nb_csectors = ((l2_entry >> s->csize_shift) & s->csize_mask) +1 ; l2_entry &= s->cluster_offset_mask; - ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, - l2_entry & ~511 , nb_csectors *512 ); + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + l2_entry & ~511 , nb_csectors *512 );if (ret <0 ) { goto fail; }@@ -1445,8 +1445,9 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, }/* Mark cluster as used */ - ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, - offset, s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + offset, s->cluster_size);if (ret <0 ) { goto fail; }@@ -1498,8 +1499,8 @@ static int check_refcounts_l1(BlockDriverState *bs, l1_size2 = l1_size *sizeof (uint64_t );/* Mark L1 table as used */ - ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, - l1_table_offset, l1_size2); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, refcount_table_size, + l1_table_offset, l1_size2);if (ret <0 ) { goto fail; }@@ -1528,8 +1529,9 @@ static int check_refcounts_l1(BlockDriverState *bs,if (l2_offset) {/* Mark L2 table as used */ l2_offset &=L1E_OFFSET_MASK ; - ret =inc_refcounts (bs, res, refcount_table, refcount_table_size, - l2_offset, s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + l2_offset, s->cluster_size);if (ret <0 ) { goto fail; }@@ -1744,14 +1746,15 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, } res->corruptions_fixed++; - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, - offset, s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, nb_clusters, + offset, s->cluster_size);if (ret <0 ) {return ret; }/* No need to check whether the refcount is now greater than 1: * This area was just allocated and zeroed, so it can only be -* exactly 1 after inc_refcounts() */ +* exactly 1 after qcow2_inc_refcounts_imrt() */ continue ; resize_fail:@@ -1766,8 +1769,8 @@ resize_fail: }if (offset !=0 ) { - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, - offset, s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, nb_clusters, + offset, s->cluster_size);if (ret <0 ) {return ret; }@@ -1807,8 +1810,8 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, }/* header */ - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, -0 , s->cluster_size); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, nb_clusters, +0 , s->cluster_size);if (ret <0 ) {return ret; }@@ -1829,16 +1832,16 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,return ret; } } - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, - s->snapshots_offset, s->snapshots_size); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, nb_clusters, + s->snapshots_offset, s->snapshots_size);if (ret <0 ) {return ret; }/* refcount data */ - ret =inc_refcounts (bs, res, refcount_table, nb_clusters, - s->refcount_table_offset, - s->refcount_table_size *sizeof (uint64_t )); + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, nb_clusters, + s->refcount_table_offset, + s->refcount_table_size *sizeof (uint64_t ));if (ret <0 ) {return ret; }block/qcow2.h +0
-0
block/qcow2.h int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, · line 552
Quoted 10 lines
block/qcow2.h +4
-0@@ -543,6 +543,10 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, int64_t size);int qcow2_pre_write_overlap_check (BlockDriverState *bs,int ign,int64_t offset,int64_t size); +int qcow2_inc_refcounts_imrt (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size, +int64_t offset,int64_t size); int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order, BlockDriverAmendStatusCB *status_cb,
Mechanically safe of course. Max and Kevin will be the experts for the implications in qcow2-land. (And Max already gave his blessings!)
PATCH 21/25 qcow2-bitmap: refcounts
In reply to Vladimir Sementsov-OgievskiyCalculate refcounts for qcow2 bitmaps. It is needed for qcow2's qemu-img check implementation.
--- block/qcow2-bitmap.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2-refcount.c | 6 ++++ block/qcow2.h | 3 ++ 3 files changed, 85 insertions(+)
block/qcow2-bitmap.c +76 -0
@@ -1222,3 +1222,79 @@ common_errp: name, bdrv_get_device_or_node_name(bs), reason);return false; } + +int qcow2_check_bitmaps_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +Qcow2BitmapList *bm_list; +Qcow2Bitmap *bm; + +if (s->nb_bitmaps ==0 ) { +return 0 ; + } + + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, refcount_table_size, + s->bitmap_directory_offset, + s->bitmap_directory_size); +if (ret <0 ) { +return ret; + } + + bm_list =bitmap_list_load (bs, s->bitmap_directory_offset, + s->bitmap_directory_size,NULL ); +if (bm_list ==NULL ) { + res->corruptions++; +return -EINVAL ; + } + + QSIMPLEQ_FOREACH(bm, bm_list, entry) { +uint64_t *bitmap_table =NULL ; +int i; + + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + bm->table.offset, + bm->table.size *sizeof (uint64_t )); +if (ret <0 ) { + goto out; + } + + ret =bitmap_table_load (bs, &bm->table, &bitmap_table); +if (ret <0 ) { + res->corruptions++; + goto out; + } + +for (i =0 ; i < bm->table.size; ++i) { +uint64_t entry = bitmap_table[i]; +uint64_t offset = entry &BME_TABLE_ENTRY_OFFSET_MASK ; + +if (check_table_entry (entry, s->cluster_size) <0 ) { + res->corruptions++; +continue ; + } + +if (offset ==0 ) { +continue ; + } + + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + offset, s->cluster_size); +if (ret <0 ) { +g_free (bitmap_table); + goto out; + } + } + +g_free (bitmap_table); + } + +out: +bitmap_list_free (bm_list); + +return ret; +}
block/qcow2-refcount.c +6 -0
@@ -1846,6 +1846,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,return ret; } +/* bitmaps */ + ret =qcow2_check_bitmaps_refcounts (bs, res, refcount_table, nb_clusters); +if (ret <0 ) { +return ret; + } +return check_refblocks (bs, res, fix, rebuild, refcount_table, nb_clusters); }
block/qcow2.h +3 -0
@@ -624,5 +624,8 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs,const char *name, uint32_t granularity, Error **errp); +int qcow2_check_bitmaps_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size);#endif
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 21/25] qcow2-bitmap: refcounts
Quoted 118 lines · Vladimir Sementsov-Ogievskiy
Calculate refcounts for qcow2 bitmaps. It is needed for qcow2's qemu-img check implementation.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>--- block/qcow2-bitmap.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2-refcount.c | 6 ++++ block/qcow2.h | 3 ++ 3 files changed, 85 insertions(+)block/qcow2-bitmap.c +76
-0@@ -1222,3 +1222,79 @@ common_errp: name, bdrv_get_device_or_node_name(bs), reason);return false; } + +int qcow2_check_bitmaps_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +Qcow2BitmapList *bm_list; +Qcow2Bitmap *bm; + +if (s->nb_bitmaps ==0 ) { +return 0 ; + } + + ret =qcow2_inc_refcounts_imrt (bs, res, refcount_table, refcount_table_size, + s->bitmap_directory_offset, + s->bitmap_directory_size); +if (ret <0 ) { +return ret; + } + + bm_list =bitmap_list_load (bs, s->bitmap_directory_offset, + s->bitmap_directory_size,NULL ); +if (bm_list ==NULL ) { + res->corruptions++; +return -EINVAL ; + } + + QSIMPLEQ_FOREACH(bm, bm_list, entry) { +uint64_t *bitmap_table =NULL ; +int i; + + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + bm->table.offset, + bm->table.size *sizeof (uint64_t )); +if (ret <0 ) { + goto out; + } + + ret =bitmap_table_load (bs, &bm->table, &bitmap_table); +if (ret <0 ) { + res->corruptions++; + goto out; + } + +for (i =0 ; i < bm->table.size; ++i) { +uint64_t entry = bitmap_table[i]; +uint64_t offset = entry &BME_TABLE_ENTRY_OFFSET_MASK ; + +if (check_table_entry (entry, s->cluster_size) <0 ) { + res->corruptions++; +continue ; + } + +if (offset ==0 ) { +continue ; + } + + ret =qcow2_inc_refcounts_imrt (bs, res, + refcount_table, refcount_table_size, + offset, s->cluster_size); +if (ret <0 ) { +g_free (bitmap_table); + goto out; + } + } + +g_free (bitmap_table); + } + +out: +bitmap_list_free (bm_list); + +return ret; +}block/qcow2-refcount.c +6
-0@@ -1846,6 +1846,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,return ret; } +/* bitmaps */ + ret =qcow2_check_bitmaps_refcounts (bs, res, refcount_table, nb_clusters); +if (ret <0 ) { +return ret; + } +return check_refblocks (bs, res, fix, rebuild, refcount_table, nb_clusters); }block/qcow2.h +0
-0
block/qcow2.h bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs, · line 631
Quoted 8 lines
block/qcow2.h +3
-0@@ -624,5 +624,8 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs,const char *name, uint32_t granularity, Error **errp); +int qcow2_check_bitmaps_refcounts (BlockDriverState *bs,BdrvCheckResult *res, +void **refcount_table, +int64_t *refcount_table_size);#endif
Looks OK.
PATCH 22/25 block/dirty-bitmap: add bdrv_remove_persistent_dirty_bitmap
In reply to Vladimir Sementsov-OgievskiyInterface for removing persistent bitmap from its storage.
--- block/dirty-bitmap.c | 18 ++++++++++++++++++ include/block/block_int.h | 3 +++ include/block/dirty-bitmap.h | 3 +++ 3 files changed, 24 insertions(+)
block/dirty-bitmap.c +18 -0
@@ -327,12 +327,30 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)/** * Release all named dirty bitmaps attached to a BDS (for use in bdrv_close()). * There must not be any frozen bitmaps attached. +* This function does not remove persistent bitmaps from the storage. */ void bdrv_release_named_dirty_bitmaps (BlockDriverState *bs) {bdrv_do_release_matching_dirty_bitmap (bs,NULL , true); } +/** +* Remove persistent dirty bitmap from the storage if it exists. +* Absence of bitmap is not an error, because we have the following scenario: +* BdrvDirtyBitmap can have .persistent = true but not yet saved and have no +* stored version. For such bitmap bdrv_remove_persistent_dirty_bitmap() should +* not fail. +* This function doesn't release corresponding BdrvDirtyBitmap. +*/ +void bdrv_remove_persistent_dirty_bitmap (BlockDriverState *bs, +const char *name, +Error **errp) +{ +if (bs->drv && bs->drv->bdrv_remove_persistent_dirty_bitmap) { + bs->drv->bdrv_remove_persistent_dirty_bitmap (bs, name, errp); + } +} +void bdrv_disable_dirty_bitmap (BdrvDirtyBitmap *bitmap) {assert (!bdrv_dirty_bitmap_frozen (bitmap));
include/block/block_int.h +3 -0
@@ -328,6 +328,9 @@ struct BlockDriver {const char *name, uint32_t granularity, Error **errp); +void (*bdrv_remove_persistent_dirty_bitmap)(BlockDriverState *bs, +const char *name, +Error **errp);QLIST_ENTRY (BlockDriver) list; };
include/block/dirty-bitmap.h +3 -0
@@ -25,6 +25,9 @@ BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,void bdrv_dirty_bitmap_make_anon (BdrvDirtyBitmap *bitmap);void bdrv_release_dirty_bitmap (BlockDriverState *bs,BdrvDirtyBitmap *bitmap);void bdrv_release_named_dirty_bitmaps (BlockDriverState *bs); +void bdrv_remove_persistent_dirty_bitmap (BlockDriverState *bs, +const char *name, +Error **errp);void bdrv_disable_dirty_bitmap (BdrvDirtyBitmap *bitmap);void bdrv_enable_dirty_bitmap (BdrvDirtyBitmap *bitmap);BlockDirtyInfoList *bdrv_query_dirty_bitmaps (BlockDriverState *bs);
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 22/25] block/dirty-bitmap: add bdrv_remove_persistent_dirty_bitmap
Quoted 65 lines · Vladimir Sementsov-Ogievskiy
Interface for removing persistent bitmap from its storage.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>--- block/dirty-bitmap.c | 18 ++++++++++++++++++ include/block/block_int.h | 3 +++ include/block/dirty-bitmap.h | 3 +++ 3 files changed, 24 insertions(+)block/dirty-bitmap.c +18
-0@@ -327,12 +327,30 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)/** * Release all named dirty bitmaps attached to a BDS (for use in bdrv_close()). * There must not be any frozen bitmaps attached. +* This function does not remove persistent bitmaps from the storage. */ void bdrv_release_named_dirty_bitmaps (BlockDriverState *bs) {bdrv_do_release_matching_dirty_bitmap (bs,NULL , true); } +/** +* Remove persistent dirty bitmap from the storage if it exists. +* Absence of bitmap is not an error, because we have the following scenario: +* BdrvDirtyBitmap can have .persistent = true but not yet saved and have no +* stored version. For such bitmap bdrv_remove_persistent_dirty_bitmap() should +* not fail. +* This function doesn't release corresponding BdrvDirtyBitmap. +*/ +void bdrv_remove_persistent_dirty_bitmap (BlockDriverState *bs, +const char *name, +Error **errp) +{ +if (bs->drv && bs->drv->bdrv_remove_persistent_dirty_bitmap) { + bs->drv->bdrv_remove_persistent_dirty_bitmap (bs, name, errp); + } +} +void bdrv_disable_dirty_bitmap (BdrvDirtyBitmap *bitmap) {assert (!bdrv_dirty_bitmap_frozen (bitmap));include/block/block_int.h +3
-0@@ -328,6 +328,9 @@ struct BlockDriver {const char *name, uint32_t granularity, Error **errp); +void (*bdrv_remove_persistent_dirty_bitmap)(BlockDriverState *bs, +const char *name, +Error **errp);QLIST_ENTRY (BlockDriver) list; };include/block/dirty-bitmap.h +0
-0
include/block/dirty-bitmap.h BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs, · line 33
Quoted 9 lines
include/block/dirty-bitmap.h +3
-0@@ -25,6 +25,9 @@ BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,void bdrv_dirty_bitmap_make_anon (BdrvDirtyBitmap *bitmap);void bdrv_release_dirty_bitmap (BlockDriverState *bs,BdrvDirtyBitmap *bitmap);void bdrv_release_named_dirty_bitmaps (BlockDriverState *bs); +void bdrv_remove_persistent_dirty_bitmap (BlockDriverState *bs, +const char *name, +Error **errp);void bdrv_disable_dirty_bitmap (BdrvDirtyBitmap *bitmap);void bdrv_enable_dirty_bitmap (BdrvDirtyBitmap *bitmap);BlockDirtyInfoList *bdrv_query_dirty_bitmaps (BlockDriverState *bs);
PATCH 23/25 qcow2: add .bdrv_remove_persistent_dirty_bitmap
In reply to Vladimir Sementsov-OgievskiyRealize .bdrv_remove_persistent_dirty_bitmap interface.
--- block/qcow2-bitmap.c | 41 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 1 + block/qcow2.h | 3 +++ 3 files changed, 45 insertions(+)
block/qcow2-bitmap.c +41 -0
@@ -1055,6 +1055,47 @@ static Qcow2Bitmap *find_bitmap_by_name(Qcow2BitmapList *bm_list,return NULL ; } +void qcow2_remove_persistent_dirty_bitmap (BlockDriverState *bs, +const char *name, +Error **errp) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +Qcow2Bitmap *bm; +Qcow2BitmapList *bm_list; + +if (s->nb_bitmaps ==0 ) { +/* Absence of the bitmap is not an error: see explanation above +* bdrv_remove_persistent_dirty_bitmap() definition. */ +return ; + } + + bm_list =bitmap_list_load (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); +if (bm_list ==NULL ) { +return ; + } + + bm =find_bitmap_by_name (bm_list, name); +if (bm ==NULL ) { + goto fail; + } + +QSIMPLEQ_REMOVE (bm_list, bm, Qcow2Bitmap, entry); + + ret =update_ext_header_and_dir (bs, bm_list); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to update bitmap extension" ); + goto fail; + } + +free_bitmap_clusters (bs, &bm->table); + +fail: +bitmap_free (bm); +bitmap_list_free (bm_list); +} +void qcow2_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp) {BdrvDirtyBitmap *bitmap;
block/qcow2.c +1 -0
@@ -3542,6 +3542,7 @@ BlockDriver bdrv_qcow2 = { .bdrv_load_autoloading_dirty_bitmaps = qcow2_load_autoloading_dirty_bitmaps, .bdrv_store_persistent_dirty_bitmaps = qcow2_store_persistent_dirty_bitmaps, .bdrv_can_store_new_dirty_bitmap = qcow2_can_store_new_dirty_bitmap, + .bdrv_remove_persistent_dirty_bitmap = qcow2_remove_persistent_dirty_bitmap, };static void bdrv_qcow2_init (void )
block/qcow2.h +3 -0
@@ -627,5 +627,8 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs,int qcow2_check_bitmaps_refcounts (BlockDriverState *bs,BdrvCheckResult *res,void **refcount_table,int64_t *refcount_table_size); +void qcow2_remove_persistent_dirty_bitmap (BlockDriverState *bs, +const char *name, +Error **errp);#endif
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 23/25] qcow2: add .bdrv_remove_persistent_dirty_bitmap
Quoted 80 lines · Vladimir Sementsov-Ogievskiy
Realize .bdrv_remove_persistent_dirty_bitmap interface.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>--- block/qcow2-bitmap.c | 41 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 1 + block/qcow2.h | 3 +++ 3 files changed, 45 insertions(+)block/qcow2-bitmap.c +41
-0@@ -1055,6 +1055,47 @@ static Qcow2Bitmap *find_bitmap_by_name(Qcow2BitmapList *bm_list,return NULL ; } +void qcow2_remove_persistent_dirty_bitmap (BlockDriverState *bs, +const char *name, +Error **errp) +{ +int ret; +BDRVQcow2State *s = bs->opaque; +Qcow2Bitmap *bm; +Qcow2BitmapList *bm_list; + +if (s->nb_bitmaps ==0 ) { +/* Absence of the bitmap is not an error: see explanation above +* bdrv_remove_persistent_dirty_bitmap() definition. */ +return ; + } + + bm_list =bitmap_list_load (bs, s->bitmap_directory_offset, + s->bitmap_directory_size, errp); +if (bm_list ==NULL ) { +return ; + } + + bm =find_bitmap_by_name (bm_list, name); +if (bm ==NULL ) { + goto fail; + } + +QSIMPLEQ_REMOVE (bm_list, bm, Qcow2Bitmap, entry); + + ret =update_ext_header_and_dir (bs, bm_list); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to update bitmap extension" ); + goto fail; + } + +free_bitmap_clusters (bs, &bm->table); + +fail: +bitmap_free (bm); +bitmap_list_free (bm_list); +} +void qcow2_store_persistent_dirty_bitmaps (BlockDriverState *bs,Error **errp) {BdrvDirtyBitmap *bitmap;block/qcow2.c +1
-0@@ -3542,6 +3542,7 @@ BlockDriver bdrv_qcow2 = { .bdrv_load_autoloading_dirty_bitmaps = qcow2_load_autoloading_dirty_bitmaps, .bdrv_store_persistent_dirty_bitmaps = qcow2_store_persistent_dirty_bitmaps, .bdrv_can_store_new_dirty_bitmap = qcow2_can_store_new_dirty_bitmap, + .bdrv_remove_persistent_dirty_bitmap = qcow2_remove_persistent_dirty_bitmap, };static void bdrv_qcow2_init (void )block/qcow2.h +0
-0
block/qcow2.h bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs, · line 634
Quoted 8 lines
block/qcow2.h +3
-0@@ -627,5 +627,8 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs,int qcow2_check_bitmaps_refcounts (BlockDriverState *bs,BdrvCheckResult *res,void **refcount_table,int64_t *refcount_table_size); +void qcow2_remove_persistent_dirty_bitmap (BlockDriverState *bs, +const char *name, +Error **errp);#endif
PATCH 24/25 qmp: block-dirty-bitmap-remove: remove persistent
In reply to Vladimir Sementsov-OgievskiyRemove persistent bitmap from the storage on block-dirty-bitmap-remove.
--- blockdev.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
blockdev.c +10 -0
@@ -2767,6 +2767,7 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,AioContext *aio_context;BlockDriverState *bs;BdrvDirtyBitmap *bitmap; +Error *local_err =NULL ; bitmap =block_dirty_bitmap_lookup (node, name, &bs, &aio_context, errp);if (!bitmap || !bs) {
@@ -2779,6 +2780,15 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name, name); goto out; } + +if (bdrv_dirty_bitmap_get_persistance (bitmap)) { +bdrv_remove_persistent_dirty_bitmap (bs, name, &local_err); +if (local_err !=NULL ) { +error_propagate (errp, local_err); + goto out; + } + } +bdrv_dirty_bitmap_make_anon (bitmap);bdrv_release_dirty_bitmap (bs, bitmap);
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 24/25] qmp: block-dirty-bitmap-remove: remove persistent
Quoted 27 lines · Vladimir Sementsov-Ogievskiy
Remove persistent bitmap from the storage on block-dirty-bitmap-remove.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Max Reitz <[email protected]>--- blockdev.c | 10 ++++++++++ 1 file changed, 10 insertions(+)blockdev.c +3
-0@@ -2767,6 +2767,7 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,AioContext *aio_context;BlockDriverState *bs;BdrvDirtyBitmap *bitmap; +Error *local_err =NULL ; bitmap =block_dirty_bitmap_lookup (node, name, &bs, &aio_context, errp);if (!bitmap || !bs) {@@ -2779,6 +2780,15 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name, name); goto out; } + +if (bdrv_dirty_bitmap_get_persistance (bitmap)) {
blockdev.c void qmp_block_dirty_bitmap_remove(const char *node, const char *name, · line 2794
Quoted 10 lines
blockdev.c +7
-0+bdrv_remove_persistent_dirty_bitmap (bs, name, &local_err); +if (local_err !=NULL ) { +error_propagate (errp, local_err); + goto out; + } + } +bdrv_dirty_bitmap_make_anon (bitmap);bdrv_release_dirty_bitmap (bs, bitmap);
PATCH 25/25 qcow2-bitmap: improve check_constraints_on_bitmap
In reply to Vladimir Sementsov-OgievskiyAdd detailed error messages.
--- block/qcow2-bitmap.c | 63 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 41 insertions(+), 22 deletions(-)
block/qcow2-bitmap.c +41 -22
@@ -160,28 +160,49 @@ static int check_table_entry(uint64_t entry, int cluster_size)static int check_constraints_on_bitmap (BlockDriverState *bs,const char *name, -uint32_t granularity) +uint32_t granularity, +Error **errp) {BDRVQcow2State *s = bs->opaque;int granularity_bits =ctz32 (granularity);int64_t len =bdrv_getlength (bs); -bool fail;assert (granularity >0 );assert ((granularity & (granularity -1 )) ==0 );if (len <0 ) { +error_setg_errno (errp, -len,"Failed to get size of '%s'" , +bdrv_get_device_or_node_name (bs));return len; } - fail = (granularity_bits >BME_MAX_GRANULARITY_BITS ) || - (granularity_bits <BME_MIN_GRANULARITY_BITS ) || - (len > (uint64_t )BME_MAX_PHYS_SIZE << granularity_bits) || - (len > (uint64_t )BME_MAX_TABLE_SIZE * s->cluster_size << - granularity_bits) || - (strlen (name) >BME_MAX_NAME_SIZE ); +if (granularity_bits >BME_MAX_GRANULARITY_BITS ) { +error_setg (errp,"Granularity exceeds maximum (%u bytes)" , +1 <<BME_MAX_GRANULARITY_BITS ); +return -EINVAL ; + } +if (granularity_bits <BME_MIN_GRANULARITY_BITS ) { +error_setg (errp,"Granularity is under minimum (%u bytes)" , +1 <<BME_MIN_GRANULARITY_BITS ); +return -EINVAL ; + } -return fail ? -EINVAL :0 ; +if ((len > (uint64_t )BME_MAX_PHYS_SIZE << granularity_bits) || + (len > (uint64_t )BME_MAX_TABLE_SIZE * s->cluster_size << + granularity_bits)) + { +error_setg (errp,"Too much space will be occupied by the bitmap. " +"Use larger granularity" ); +return -EINVAL ; + } + +if (strlen (name) >BME_MAX_NAME_SIZE ) { +error_setg (errp,"Name length exceeds maximum (%u characters)" , +BME_MAX_NAME_SIZE ); +return -EINVAL ; + } + +return 0 ; } static void clear_bitmap_table(BlockDriverState *bs, uint64_t *bitmap_table,
@@ -1142,9 +1163,9 @@ void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)continue ; } -if (check_constraints_on_bitmap (bs, name, granularity) <0 ) { -error_setg (errp,"Bitmap '%s' doesn't satisfy the constraints" , - name); +if (check_constraints_on_bitmap (bs, name, granularity, errp) <0 ) { +error_prepend (errp,"Bitmap '%s' doesn't satisfy the constraints: " , + name); goto fail; }
@@ -1230,13 +1251,11 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs, Error **errp) {BDRVQcow2State *s = bs->opaque; -const char *reason =NULL ;bool found;Qcow2BitmapList *bm_list; -if (check_constraints_on_bitmap (bs, name, granularity) !=0 ) { - reason ="it doesn't satisfy the constraints" ; - goto common_errp; +if (check_constraints_on_bitmap (bs, name, granularity, errp) !=0 ) { + goto fail; }if (s->nb_bitmaps ==0 ) {
@@ -1246,21 +1265,21 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs, bm_list =bitmap_list_load (bs, s->bitmap_directory_offset, s->bitmap_directory_size, errp);if (bm_list ==NULL ) { -return false; + goto fail; } found =find_bitmap_by_name (bm_list, name);bitmap_list_free (bm_list);if (found) { - reason ="bitmap with the same name is already stored" ; - goto common_errp; +error_setg (errp,"Bitmap with the same name is already stored" ); + goto fail; }return true; -common_errp: -error_setg (errp,"Can't make bitmap '%s' persistent in '%s', as %s." , - name,bdrv_get_device_or_node_name (bs), reason); +fail: +error_prepend (errp,"Can't make bitmap '%s' persistent in '%s': " , + name,bdrv_get_device_or_node_name (bs));return false; }
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 25/25] qcow2-bitmap: improve check_constraints_on_bitmap
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Add detailed error messages.
yay
Quoted 116 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-bitmap.c | 63 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 41 insertions(+), 22 deletions(-)block/qcow2-bitmap.c +38
-19@@ -160,28 +160,49 @@ static int check_table_entry(uint64_t entry, int cluster_size)static int check_constraints_on_bitmap (BlockDriverState *bs,const char *name, -uint32_t granularity) +uint32_t granularity, +Error **errp) {BDRVQcow2State *s = bs->opaque;int granularity_bits =ctz32 (granularity);int64_t len =bdrv_getlength (bs); -bool fail;assert (granularity >0 );assert ((granularity & (granularity -1 )) ==0 );if (len <0 ) { +error_setg_errno (errp, -len,"Failed to get size of '%s'" , +bdrv_get_device_or_node_name (bs));return len; } - fail = (granularity_bits >BME_MAX_GRANULARITY_BITS ) || - (granularity_bits <BME_MIN_GRANULARITY_BITS ) || - (len > (uint64_t )BME_MAX_PHYS_SIZE << granularity_bits) || - (len > (uint64_t )BME_MAX_TABLE_SIZE * s->cluster_size << - granularity_bits) || - (strlen (name) >BME_MAX_NAME_SIZE ); +if (granularity_bits >BME_MAX_GRANULARITY_BITS ) { +error_setg (errp,"Granularity exceeds maximum (%u bytes)" , +1 <<BME_MAX_GRANULARITY_BITS ); +return -EINVAL ; + } +if (granularity_bits <BME_MIN_GRANULARITY_BITS ) { +error_setg (errp,"Granularity is under minimum (%u bytes)" , +1 <<BME_MIN_GRANULARITY_BITS ); +return -EINVAL ; + } -return fail ? -EINVAL :0 ; +if ((len > (uint64_t )BME_MAX_PHYS_SIZE << granularity_bits) || + (len > (uint64_t )BME_MAX_TABLE_SIZE * s->cluster_size << + granularity_bits)) + { +error_setg (errp,"Too much space will be occupied by the bitmap. " +"Use larger granularity" ); +return -EINVAL ; + } + +if (strlen (name) >BME_MAX_NAME_SIZE ) { +error_setg (errp,"Name length exceeds maximum (%u characters)" , +BME_MAX_NAME_SIZE ); +return -EINVAL ; + } + +return 0 ; } static void clear_bitmap_table(BlockDriverState *bs, uint64_t *bitmap_table,@@ -1142,9 +1163,9 @@ void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)continue ; } -if (check_constraints_on_bitmap (bs, name, granularity) <0 ) { -error_setg (errp,"Bitmap '%s' doesn't satisfy the constraints" , - name); +if (check_constraints_on_bitmap (bs, name, granularity, errp) <0 ) { +error_prepend (errp,"Bitmap '%s' doesn't satisfy the constraints: " , + name); goto fail; }@@ -1230,13 +1251,11 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs, Error **errp) {BDRVQcow2State *s = bs->opaque; -const char *reason =NULL ;bool found;Qcow2BitmapList *bm_list; -if (check_constraints_on_bitmap (bs, name, granularity) !=0 ) { - reason ="it doesn't satisfy the constraints" ; - goto common_errp; +if (check_constraints_on_bitmap (bs, name, granularity, errp) !=0 ) { + goto fail; }if (s->nb_bitmaps ==0 ) {@@ -1246,21 +1265,21 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs, bm_list =bitmap_list_load (bs, s->bitmap_directory_offset, s->bitmap_directory_size, errp);if (bm_list ==NULL ) { -return false; + goto fail; } found =find_bitmap_by_name (bm_list, name);bitmap_list_free (bm_list);if (found) { - reason ="bitmap with the same name is already stored" ; - goto common_errp; +error_setg (errp,"Bitmap with the same name is already stored" ); + goto fail; }return true;
block/qcow2-bitmap.c bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs, · line 1285
Quoted 10 lines
block/qcow2-bitmap.c +3
-3-common_errp: -error_setg (errp,"Can't make bitmap '%s' persistent in '%s', as %s." , - name,bdrv_get_device_or_node_name (bs), reason); +fail: +error_prepend (errp,"Can't make bitmap '%s' persistent in '%s': " , + name,bdrv_get_device_or_node_name (bs));return false; }
Sorry I made it less efficient :)
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Hi all!
Hi! :)
Quoted 2 lines
There is a new update of qcow2-bitmap series - v14.
Having the cover letter be 00/24 but including 25 patches confuses the patch scraping tool a good deal. Also, can you include the "v14" in the patch emails themselves, too?
(Don't respin just for this, thanks!)
Quoted 311 lines
web: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=qcow2-bitmap-v14
git: https://src.openvz.org/scm/~vsementsov/qemu.git (tag qcow2-bitmap-v14)v14:
07: use '|=' to update
need_update_header
add John's r-b
add Max's r-b
09: remove unusedbitmap_table_to_cpu()
left Max's r-b, hope it's ok
add John's r-b
10: remove extra new line
add John's r-b
11: add John's r-b
12: add John's r-b
13: small fixes by John's review:
- remove weirdg_freeof NULL pointer from
if (tb == NULL) {
g_free(tb);
return-EINVAL;
}
- remove extra comment "/* errp is already set */"
- s/"Too large bitmap directory"/"Bitmap directory is too large"/
left Max's r-b, hope you don't mind
22: add Max's r-b
23: add Max's r-b
24: add Max's r-b
25: new patch to improve error message oncheck_constraints_on_bitmapfailv13: Just a fix for style checker.
13: line over 80
14: line over 80
22: s/if () \n{/if () {/v12:
07: do not update header inqcow2_read_extensions, instead do it in the
end ofqcow2_open, where it is updated also to clear unknown autoclear features.
Wrongbdrv_is_root_nodeused instead ofbdrv_is_read_onlyis fixed automatically.
08: add Max's r-b
09: contextual: s/raw_bsd/raw-format/
qcow2-bitmap.c: copyright s/2016/2017/
fix compilation: moveupdate_header_sync()to this patch
add Max's r-b
13:update_header_sync()is moved to 09
add Max's r-b
15: add Max's r-b
16: As qmp-commands.txt is removed from master, remove it here too.
Sentence "For now only Qcow2 disks support persistent bitmaps.
Default is false." moved toqapi/block-core.json.
Hope that is OK, Max's r-b is not dropped.
17: qmp-commands.txt deleted, r-b is not dropped too.
18: s/is failed/has failed/, add Max's r-b
19: copyright: s/2016/2017/
21: drop "res->corruptions> 0 => ret =-EINVAL", add Max's r-b
22: actually, patch is replaced with a new one, however some parts are the same.
instead of changingbdrv_release_dirty_bitmapbehaviour, create newbdrv_remove_persistent_dirty_bitmap
23: improve comment, about not-exists is not an error
24: new patchOld 24 (qcow2-bitmap: cache bitmap list in BDRVQcow2State) will be sent separately, to be applied after these series.
Also: about
bdrv_dirty_bitmap_set_persistance: I've decided not change its behaviour for now and just callbdrv_remove_persistent_dirty_bitmapfromqmp_block_dirty_bitmap_remove. Reasons:
1. Do not change reviewed part.
2. I'm not sure, that this complication of BdrvDirtyBitmap.persistent semantics is good (current semantics are just: .persistent means that bitmap should be saved on disk close). .persistent actually is not very related to "is there stored version of the bitmap in the image".
3. It may be done later.
4. It is not actually needed for now, as the only usage is dropping bitmap inbdrv_remove_persistent_dirty_bitmap. May be in future it will be needed, when we will have qmp interfaces for finer control of persistent dirty bitmaps.v11:
Fix automatic build test fail.
18: wording - "ASCII representation of SHA256 ..."
24: fix redeclaration error. (This is still RFC, may be not needed patch,
see description in v10 below).v10:
07: rm John's r-b
not add Max's r-b
fix subject s/dirty bitmaps/bitmaps
improve WARNING message
remove header ext if autoclear bit is unset, throughqcow2_updata_header()- r-b blocking change
08: movebdrv_load_autoloading_dirty_bitmapsunder asserts block (Max)
fix typo in commit message
movebdrv_load_autoloading_dirty_bitmapsafter asserts
John's r-b
09: rewritecheck_dir_entry
remove extra feature cluster_size=0 fromcheck_table_entry
which clears autoclear bit before trying to update bitmap directorybitmap_list_store- do not free old clusters if in_place=true
changes in comments, fix indents
add functions
bitmap_list_count
update_ext_header_and_dir_in_place(unset autoclear bit before trying to
modify bitmap directory)
(for usage inqcow2_load_autoloading_dirty_bitmaps)
11: add node name to error report
Max's r-b
13: add type Qcow2BitmapTable
move from bm.table_* to bm.table.*
rewritecheck_constraints_on_bitmap
flush(bs->file)instead of flush(bs) inupdate_ext_header_and_dirassert(DIV_ROUND_UP(bm_size, dsc) == tb_size);
andassert(write_size <= s->cluster_size); instore_bitmap_data
fix: s/tb_size/tb_size *sizeof(tb[0])instore_bitmap
inqcow2_store_persistent_dirty_bitmaps:
fail if !can_write
fix counting ofnew_nb_bitmapsandnew_dir_size
free old clusters _after_ successful directory update (aggregate old bitmap
tables intodrop_tables)
2 14: rename tocan_store_new_dirty_bitmap
Max's r-b
15: rename tocan_store_new_dirty_bitmap
rm extra !!
16: Max's r-b
rename tocan_store_new_dirty_bitmap
indenting
Since s/2.8/2.9
17: Max's r-bSince s/2.8/2.918: hashing can fail in comment for qapi
Since s/2.8/2.9
remove dead comment
19: indentation
Max's r-b
20: Max's r-b
21: fix error handlingreturn 0 if nb_bitmaps == 0new patches:
22-23: remove bitmap from file onbdrv_release_dirty_bitmap
24: experimental, RFC: cache bitmap list to bs, to not reload it from file.
I'm not sure that is needed now, but if you want I can merge it to earlier patches.v9:
rebase on master!
01-04,06,07: John's r-b
03: rewording in comment ("The concurrent resetting ..."), John's r-b 07: reword error messagescommit message: dirty bitmap -> bitmap09: fix uninitialized ret in
bitmap_list_store()(fix compilation warning) 18: fix compilation of tests/test-hbitmapalso, change constants names for qcow2:
DIRTY_BITMAP-> BITMAP (as it is not dirty bitmaps extension, but just bitmaps extension), QCOW -> QCOW2 (bitmaps are only for qcow2)v8:
Patches 01-06 are mostly unchanged, except for spelling and wording. 02 - add Eric's r-b
03,04 - add Max's r-b07-09 is refactored "bitmap-read" part of the feature. Main changes:
- introduce bitmap list - normal list, which represents bitmap directory. Functionbitmap_list_loadloads bitmap directory, checks everything and creates normal list.
- introduce .bdrv_load_autoloading_dirty_bitmaps : instead of loading bitmaps inqcow2_open, lets load them only inbdrv_open, to avoid reloading inbdrv_snapshot_goto
- do not delete bitmaps after read, but mark themIN_USE10 has contextual changes and rewording of comment. I've added Max's r-b, hope it's ok.
11: add
error_report("Persistent bitmaps are lost")in case of failed bitmap store12: add Max's r-b
13 is refactored "bitmap-store" part of the feature. see 07-09 description
- for now I just free old clusters and allocate new. This will be improved with a separate patch.patch about "delete bitmaps on truncate" is removed. This case will be handled later.
IN_USE, autoclear, check-constraints things are merged into other patches.14-15 are mew patches, to early check possibility of creating persistent bitmap with
specified name and granularity in specified BDS16-17: spelling, rewording, indenting, tiny code simplifying, check
can_store(by 14-15),
s/if (autoload && !persistent)/if (has_autoload&& !persistent)/, rebase (deleted qmp-commands.hx)18: alternative to md5 in query-block:
- separated qmp command -x-debug-block-dirty-bitmap-sha256
- as adviced by Daniel P. Berrange in my parallel thread:
- sha256 instead of md5
- use qcrypto_hash_* instead of GChecksum
- fix bug =) (size was wrong inhbitmap_md5)19: s/3999/3fff/, use x-debug-block-dirty-bitmap-sha256
20: new patch to rename and publish
inc_refcounts21: some fixes and refactoring mostyly by Max's comments.
Max, Eric, great tanks for your review!
I hope, I've covered most of your comments by this update.TODO:
- handle reopening imageRO->RWand incoming migration, setIN_USEfor existing loaded bitmaps in these cases.
- reuse old, already allocated data clusters for bitmaps storing
- truncate bitmaps in the image on truncatev7:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v7
based on block-next (https://github.com/XanClic/qemu/commits/block-next)- a lot of refactoring and reordering of patches.
- dead code removed (bdrv_dirty_bitmap_load, etc.)
- do not maintain extra data for now
- do not store dirty bitmap directory in memory (as we use it seldom, we can just reread if needed)By Kevin's review:
01 - commit message changed:fix->improvement(as it was not a bug) 03 - r-b
04 - r-b
05 - add 21 patch to fix spec, also, removed all (I hope) mentions of
"Bitmap Header", switch to one unified name for it - "Bitmap Directory Entry", to avoid misunderstanding with Qcow2 header. (also, add patch 22, to fix it in spec)
v6.06 - improvefor_each_dir_entryloop, reorder patches, other small fixes v6.07 ~> v7.09 - dead code removed, I've moved to one function
.bdrv_store_persistent_bitmaps and have wrapper and callback in one patch (with also some other staff. If it not ok, I can split them)
v6.08 - about keeping bitmap directory instead of bitmap list: no I don't keep
it at all.
v6.16 - oldbdrv_bitmap-storing related functions are removed. The new one is
bdrv_store_persistent_bitmaps.v6:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v6
based on block-next (https://github.com/XanClic/qemu/commits/block-next)There are a lot of changes, reorderings and additions in comparement with v5. One principal thing: now bitmaps are removed from image after loading instead of marking them
in_use. It is simpler and we do not need to store superfluous data. Also, we are no more interested in command line interface to dirty bitmaps. So it is dropped. If someone needs it I can add it later.Vladimir Sementsov-Ogievskiy (25):
specs/qcow2: fix bitmap granularity qemu-specific note specs/qcow2: do not use wording 'bitmap header'
hbitmap: improve dirty iter
tests: add hbitmap iter test
block: fixbdrv_dirty_bitmap_granularitysignatureblock/dirty-bitmap: adddeserialize_onesfunc
qcow2: add bitmaps extension
block: introduce auto-loading bitmaps
qcow2: add .bdrv_load_autoloading_dirty_bitmapsblock/dirty-bitmap: add autoload field to BdrvDirtyBitmap
block: introduce persistent dirty bitmapsblock/dirty-bitmap: addbdrv_dirty_bitmap_next()
qcow2: add .bdrv_store_persistent_dirty_bitmaps()
block: addbdrv_can_store_new_dirty_bitmap
qcow2: add .bdrv_can_store_new_dirty_bitmap
qmp: add persistent flag to block-dirty-bitmap-add
qmp: add autoload parameter to block-dirty-bitmap-add
qmp: add x-debug-block-dirty-bitmap-sha256
iotests: test qcow2 persistent dirty bitmap
qcow2-refcount: renameinc_refcounts()and make it public
qcow2-bitmap: refcounts
block/dirty-bitmap: addbdrv_remove_persistent_dirty_bitmap
qcow2: add .bdrv_remove_persistent_dirty_bitmap
qmp: block-dirty-bitmap-remove: remove persistent
qcow2-bitmap: improvecheck_constraints_on_bitmapblock.c | 68 +++ block/Makefile.objs | 2 +- block/dirty-bitmap.c | 80 ++- block/qcow2-bitmap.c | 1360 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2-refcount.c | 59 +- block/qcow2.c | 128 +++- block/qcow2.h | 42 ++ blockdev.c | 71 ++- docs/specs/qcow2.txt | 8 +- include/block/block.h | 5 + include/block/block_int.h | 12 + include/block/dirty-bitmap.h | 21 +- include/qemu/hbitmap.h | 49 +- qapi/block-core.json | 39 +- tests/Makefile.include | 2 +- tests/qemu-iotests/165 | 89 +++ tests/qemu-iotests/165.out | 5 + tests/qemu-iotests/group | 1 + tests/test-hbitmap.c | 19 + util/hbitmap.c | 51 +- 20 files changed, 2047 insertions(+), 64 deletions(-) create mode 100644 block/qcow2-bitmap.c create mode 100755 tests/qemu-iotests/165 create mode 100644 tests/qemu-iotests/165.out
Quoted 6 lines · John Snow
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Hi all!
Quoted 8 lines
Hi! :)
Quoted 2 lines
There is a new update of qcow2-bitmap series - v14.
Having the cover letter be 00/24 but including 25 patches confuses the patch scraping tool a good deal. Also, can you include the "v14" in the patch emails themselves, too?
If you do have a reason for v15, using 'qemu send-email -v15 ...' should automatically put the prefix on all the mails.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org