qemu-img map: Don't limit block status request size
PATCH 1/1 qemu-img map: Don't limit block status request size
Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from the beginning, though it only cut the maximum in half then because the interface a signed 32 bit byte count. These days, bdrv_block_status() supports a 64 bit byte count, so the arbitrary limit is even worse.
On file-posix, bdrv_block_status() eventually maps to SEEK_HOLE and SEEK_DATA, which don't support a limit, but always do all of the work necessary to find the start of the next hole/data. Much of this work may be repeated if we don't use this information fully, but query with an only slightly larger offset in the next loop iteration. Therefore, if bdrv_block_status() is called in a loop, it should always pass the full number of bytes that the whole loop is interested in.
This removes the arbitrary limit and speeds up 'qemu-img map' significantly on heavily fragmented images.
--- qemu-img.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
qemu-img.c +1 -4
@@ -3217,12 +3217,9 @@ static int img_map(int argc, char **argv) curr.start = start_offset;while (curr.start + curr.length < length) {int64_t offset = curr.start + curr.length; -int64_t n; +int64_t n = length - offset; -/* Probe up to 1 GiB at a time. */ - n =MIN (1 * GiB, length - offset); ret =get_block_status (bs, offset, n, &next); -if (ret <0 ) {error_report ("Could not read file metadata: %s" ,strerror (-ret)); goto out;
Signature · 1 line
2.25.4
Quoted 13 lines · Kevin Wolf
Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from the beginning, though it only cut the maximum in half then because the interface a signed 32 bit byte count. These days,
bdrv_block_status()supports a 64 bit byte count, so the arbitrary limit is even worse.On file-posix,
bdrv_block_status()eventually maps toSEEK_HOLEandSEEK_DATA, which don't support a limit, but always do all of the work necessary to find the start of the next hole/data. Much of this work may be repeated if we don't use this information fully, but query with an only slightly larger offset in the next loop iteration. Therefore, ifbdrv_block_status()is called in a loop, it should always pass the full number of bytes that the whole loop is interested in.
Quoted 7 lines
This removes the arbitrary limit and speeds up 'qemu-img map' significantly on heavily fragmented images.
Signed-off-by Kevin Wolf <[email protected]>--- qemu-img.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
Do you want this in 5.1? It seems like a nice optimization.
Quoted 10 lines
qemu-img.c +0
-0@@ -3217,12 +3217,9 @@ static int img_map(int argc, char **argv) curr.start = start_offset;while (curr.start + curr.length < length) {int64_t offset = curr.start + curr.length;
qemu-img.c static int img_map(int argc, char **argv) · line 3225
Quoted 10 lines
qemu-img.c +1
-4-int64_t n; +int64_t n = length - offset; -/* Probe up to 1 GiB at a time. */ - n =MIN (1 * GiB, length - offset); ret =get_block_status (bs, offset, n, &next); -if (ret <0 ) {error_report ("Could not read file metadata: %s" ,strerror (-ret)); goto out;
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Am 07.07.2020 um 16:54 hat Eric Blake geschrieben:
Quoted 17 lines · Kevin Wolf
Quoted 16 lines
Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from the beginning, though it only cut the maximum in half then because the interface a signed 32 bit byte count. These days,
bdrv_block_status()supports a 64 bit byte count, so the arbitrary limit is even worse.On file-posix,
bdrv_block_status()eventually maps toSEEK_HOLEandSEEK_DATA, which don't support a limit, but always do all of the work necessary to find the start of the next hole/data. Much of this work may be repeated if we don't use this information fully, but query with an only slightly larger offset in the next loop iteration. Therefore, ifbdrv_block_status()is called in a loop, it should always pass the full number of bytes that the whole loop is interested in.This removes the arbitrary limit and speeds up 'qemu-img map' significantly on heavily fragmented images.
Quoted 6 lines
Quoted 4 lines
Signed-off-by Kevin Wolf <[email protected]>--- qemu-img.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)Do you want this in 5.1? It seems like a nice optimization.
I guess now that I have your R-b, I can sneak both patches in for soft freeze. :-)
Kevin
Quoted 5 lines · Kevin Wolf
Am 07.07.2020 um 16:54 hat Eric Blake geschrieben:
Quoted 4 lines · Kevin Wolf
Quoted 3 lines
Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from the beginning, though it only cut the maximum in half then because the interface a signed 32 bit byte count. These days,
bdrv_block_status()
interface was a
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
supports a 64 bit byte count, so the arbitrary limit is even worse.
On file-posix,
bdrv_block_status()eventually maps toSEEK_HOLEandSEEK_DATA, which don't support a limit, but always do all of the work necessary to find the start of the next hole/data. Much of this work may be repeated if we don't use this information fully, but query with an only slightly larger offset in the next loop iteration. Therefore, ifbdrv_block_status()is called in a loop, it should always pass the full number of bytes that the whole loop is interested in.This removes the arbitrary limit and speeds up 'qemu-img map' significantly on heavily fragmented images.
Signed-off-by Kevin Wolf <[email protected]>--- qemu-img.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
Quoted 4 lines
Quoted 1 line
Do you want this in 5.1? It seems like a nice optimization.
I guess now that I have your R-b, I can sneak both patches in for soft freeze. :-)
Can we treat optimizations for speed problems as bug fixes if they land after soft freeze but before -rc1?
At any rate, this post reminds me that Vladimir's series to support 64-bit actions elsewhere has probably slipped into 5.2 territory, but I still need to fix the fact that nbd is sending uint32_t trim/zero values into signed int block layer functions (Vladimir's work is nicer but harder to review, so I'll probably end up writing one-off patches for 5.1 with minimal churn to just block/nbd.c rather than the whole block layer...)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org