Skip to contentSkip to conversation

qemu-img map: Don't limit block status request size

PATCH In review Kevin Wolf 2 participants 4 messages Jul 7, 2020

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

PATCH 1/1 qemu-img map: Don't limit block status request size

Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from the beginning, though it only cut the maximum in half then because the interface a signed 32 bit byte count. These days, bdrv_block_status() supports a 64 bit byte count, so the arbitrary limit is even worse.

On file-posix, bdrv_block_status() eventually maps to SEEK_HOLE and SEEK_DATA, which don't support a limit, but always do all of the work necessary to find the start of the next hole/data. Much of this work may be repeated if we don't use this information fully, but query with an only slightly larger offset in the next loop iteration. Therefore, if bdrv_block_status() is called in a loop, it should always pass the full number of bytes that the whole loop is interested in.

This removes the arbitrary limit and speeds up 'qemu-img map' significantly on heavily fragmented images.

---
 qemu-img.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
qemu-img.c +1 -4
index bdb9f6aa46..74946f81ca 100644
@@ -3217,12 +3217,9 @@ static int img_map(int argc, char **argv)
     curr.start = start_offset;
     while (curr.start + curr.length < length) {
         int64_t offset = curr.start + curr.length;
-        int64_t n;
+        int64_t n = length - offset;
 
-        /* Probe up to 1 GiB at a time.  */
-        n = MIN(1 * GiB, length - offset);
         ret = get_block_status(bs, offset, n, &next);
-
         if (ret < 0) {
             error_report("Could not read file metadata: %s", strerror(-ret));
             goto out;
Signature · 1 line
2.25.4
Reviewed-by on v1 1/1
#1 lore raw
Quoted 13 lines · Kevin Wolf

Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from the beginning, though it only cut the maximum in half then because the interface a signed 32 bit byte count. These days, bdrv_block_status() supports a 64 bit byte count, so the arbitrary limit is even worse.

On file-posix, bdrv_block_status() eventually maps to SEEK_HOLE and SEEK_DATA, which don't support a limit, but always do all of the work necessary to find the start of the next hole/data. Much of this work may be repeated if we don't use this information fully, but query with an only slightly larger offset in the next loop iteration. Therefore, if bdrv_block_status() is called in a loop, it should always pass the full number of bytes that the whole loop is interested in.

Quoted 7 lines

This removes the arbitrary limit and speeds up 'qemu-img map' significantly on heavily fragmented images.

---
  qemu-img.c | 5 +----
  1 file changed, 1 insertion(+), 4 deletions(-)

Do you want this in 5.1? It seems like a nice optimization.

Quoted 10 lines
qemu-img.c +0 -0
index bdb9f6aa46..74946f81ca 100644
@@ -3217,12 +3217,9 @@ static int img_map(int argc, char **argv)
      curr.start = start_offset;
      while (curr.start + curr.length < length) {
          int64_t offset = curr.start + curr.length;

qemu-img.c static int img_map(int argc, char **argv) · line 3225

Quoted 10 lines
qemu-img.c +1 -4
-        int64_t n;
+        int64_t n = length - offset;
 
-        /* Probe up to 1 GiB at a time.  */
-        n = MIN(1 * GiB, length - offset);
          ret = get_block_status(bs, offset, n, &next);
-
          if (ret < 0) {
              error_report("Could not read file metadata: %s", strerror(-ret));
              goto out;
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#2 lore raw

Am 07.07.2020 um 16:54 hat Eric Blake geschrieben:

Quoted 17 lines · Kevin Wolf
Quoted 16 lines

Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from the beginning, though it only cut the maximum in half then because the interface a signed 32 bit byte count. These days, bdrv_block_status() supports a 64 bit byte count, so the arbitrary limit is even worse.

On file-posix, bdrv_block_status() eventually maps to SEEK_HOLE and SEEK_DATA, which don't support a limit, but always do all of the work necessary to find the start of the next hole/data. Much of this work may be repeated if we don't use this information fully, but query with an only slightly larger offset in the next loop iteration. Therefore, if bdrv_block_status() is called in a loop, it should always pass the full number of bytes that the whole loop is interested in.

This removes the arbitrary limit and speeds up 'qemu-img map' significantly on heavily fragmented images.

Quoted 6 lines
Quoted 4 lines
---
  qemu-img.c | 5 +----
  1 file changed, 1 insertion(+), 4 deletions(-)

Do you want this in 5.1? It seems like a nice optimization.

I guess now that I have your R-b, I can sneak both patches in for soft freeze. :-)

Kevin

#3 lore raw
Quoted 5 lines · Kevin Wolf

Am 07.07.2020 um 16:54 hat Eric Blake geschrieben:

Quoted 4 lines · Kevin Wolf
Quoted 3 lines

Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from the beginning, though it only cut the maximum in half then because the interface a signed 32 bit byte count. These days, bdrv_block_status()

interface was a

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines

supports a 64 bit byte count, so the arbitrary limit is even worse.

On file-posix, bdrv_block_status() eventually maps to SEEK_HOLE and SEEK_DATA, which don't support a limit, but always do all of the work necessary to find the start of the next hole/data. Much of this work may be repeated if we don't use this information fully, but query with an only slightly larger offset in the next loop iteration. Therefore, if bdrv_block_status() is called in a loop, it should always pass the full number of bytes that the whole loop is interested in.

This removes the arbitrary limit and speeds up 'qemu-img map' significantly on heavily fragmented images.

---
   qemu-img.c | 5 +----
   1 file changed, 1 insertion(+), 4 deletions(-)
Quoted 4 lines
Quoted 1 line

Do you want this in 5.1? It seems like a nice optimization.

I guess now that I have your R-b, I can sneak both patches in for soft freeze. :-)

Can we treat optimizations for speed problems as bug fixes if they land after soft freeze but before -rc1?

At any rate, this post reminds me that Vladimir's series to support 64-bit actions elsewhere has probably slipped into 5.2 territory, but I still need to fix the fact that nbd is sending uint32_t trim/zero values into signed int block layer functions (Vladimir's work is nicer but harder to review, so I'll probably end up writing one-off patches for 5.1 with minimal churn to just block/nbd.c rather than the whole block layer...)

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org