Skip to contentSkip to conversation

A pair of memory access problems

PATCH In review Michal Privoznik 4 participants 8 messages Nov 11, 2019 – Nov 15, 2019

lore
Share & tools
lore
#3 lore raw

PATCH 2/2 vfio-helpers: Free QEMUVFIOState in qemu_vfio_close()

In reply to Michal Privoznik

The qemu_vfio_open_pci() allocates this QEMUVFIOState structure but free counterpart is missing. Since we already have qemu_vfio_close() which does cleanup of the state, it looks like a perfect place to free the structure too.

==178278== 528 (360 direct, 168 indirect) bytes in 1 blocks are definitely lost in loss record 6,605 of 6,985

==178278==    at 0x4A35476: calloc (vg_replace_malloc.c:752)
==178278==    by 0x51B1158: g_malloc0 (in /usr/lib64/libglib-2.0.so.0.6000.6)
==178278==    by 0xA68613: qemu_vfio_open_pci (vfio-helpers.c:428)
==178278==    by 0x9779EA: nvme_init (nvme.c:606)
==178278==    by 0x97830F: nvme_file_open (nvme.c:795)
==178278==    by 0x8E9439: bdrv_open_driver (block.c:1293)
==178278==    by 0x8E9E1C: bdrv_open_common (block.c:1553)
==178278==    by 0x8ED264: bdrv_open_inherit (block.c:3083)
==178278==    by 0x8ED79D: bdrv_open (block.c:3176)
==178278==    by 0x5DA5C1: bds_tree_init (blockdev.c:670)
==178278==    by 0x5E2B64: qmp_blockdev_add (blockdev.c:4354)
==178278==    by 0x5ECB1D: configure_blockdev (vl.c:1202)
---
 util/vfio-helpers.c | 1 +
 1 file changed, 1 insertion(+)
util/vfio-helpers.c +1 -0
index 813f7ec564..5ff91c1e5c 100644
@@ -721,4 +721,5 @@ void qemu_vfio_close(QEMUVFIOState *s)
     close(s->device);
     close(s->group);
     close(s->container);
+    g_free(s);
 }
Signature · 1 line
2.23.0
#1 lore raw

PATCH 1/2 hw/vfio/pci: Fix double free of migration_blocker

In reply to Michal Privoznik

When user tries to hotplug a VFIO device, but the operation fails somewhere in the middle (in my testing it failed because of RLIMIT_MEMLOCK forbidding more memory allocation), then a double free occurs. In vfio_realize() the vdev->migration_blocker is allocated, then something goes wrong which causes control to jump onto 'error' label where the error is freed. But the pointer is left pointing to invalid memory. Later, when vfio_instance_finalize() is called, the memory is freed again.

In my testing the second hunk was sufficient to fix the bug, but I figured the first hunk doesn't hurt either.

==169952== Invalid read of size 8

==169952==    at 0xA47DCD: error_free (error.c:266)
==169952==    by 0x4E0A18: vfio_instance_finalize (pci.c:3040)
==169952==    by 0x8DF74C: object_deinit (object.c:606)
==169952==    by 0x8DF7BE: object_finalize (object.c:620)
==169952==    by 0x8E0757: object_unref (object.c:1074)
==169952==    by 0x45079C: memory_region_unref (memory.c:1779)
==169952==    by 0x45376B: do_address_space_destroy (memory.c:2793)
==169952==    by 0xA5C600: call_rcu_thread (rcu.c:283)
==169952==    by 0xA427CB: qemu_thread_start (qemu-thread-posix.c:519)
==169952==    by 0x80A8457: start_thread (in /lib64/libpthread-2.29.so)
==169952==    by 0x81C96EE: clone (in /lib64/libc-2.29.so)
==169952==  Address 0x143137e0 is 0 bytes inside a block of size 48 free'd
==169952==    at 0x4A342BB: free (vg_replace_malloc.c:530)
==169952==    by 0xA47E05: error_free (error.c:270)
==169952==    by 0x4E0945: vfio_realize (pci.c:3025)
==169952==    by 0x76A4FF: pci_qdev_realize (pci.c:2099)
==169952==    by 0x689B9A: device_set_realized (qdev.c:876)
==169952==    by 0x8E2C80: property_set_bool (object.c:2080)
==169952==    by 0x8E0EF6: object_property_set (object.c:1272)
==169952==    by 0x8E3FC8: object_property_set_qobject (qom-qobject.c:26)
==169952==    by 0x8E11DB: object_property_set_bool (object.c:1338)
==169952==    by 0x5E7BDD: qdev_device_add (qdev-monitor.c:673)
==169952==    by 0x5E81E5: qmp_device_add (qdev-monitor.c:798)
==169952==    by 0x9E18A8: do_qmp_dispatch (qmp-dispatch.c:132)
==169952==  Block was alloc'd at
==169952==    at 0x4A35476: calloc (vg_replace_malloc.c:752)
==169952==    by 0x51B1158: g_malloc0 (in /usr/lib64/libglib-2.0.so.0.6000.6)
==169952==    by 0xA47357: error_setv (error.c:61)
==169952==    by 0xA475D9: error_setg_internal (error.c:97)
==169952==    by 0x4DF8C2: vfio_realize (pci.c:2737)
==169952==    by 0x76A4FF: pci_qdev_realize (pci.c:2099)
==169952==    by 0x689B9A: device_set_realized (qdev.c:876)
==169952==    by 0x8E2C80: property_set_bool (object.c:2080)
==169952==    by 0x8E0EF6: object_property_set (object.c:1272)
==169952==    by 0x8E3FC8: object_property_set_qobject (qom-qobject.c:26)
==169952==    by 0x8E11DB: object_property_set_bool (object.c:1338)
==169952==    by 0x5E7BDD: qdev_device_add (qdev-monitor.c:673)
---
 hw/vfio/pci.c | 2 ++
 1 file changed, 2 insertions(+)
hw/vfio/pci.c +2 -0
index e6569a7968..9c165995df 100644
@@ -2740,6 +2740,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp)
         if (err) {
             error_propagate(errp, err);
             error_free(vdev->migration_blocker);
+            vdev->migration_blocker = NULL;
             return;
         }
     }
@@ -3023,6 +3024,7 @@ error:
     if (vdev->migration_blocker) {
         migrate_del_blocker(vdev->migration_blocker);
         error_free(vdev->migration_blocker);
+        vdev->migration_blocker = NULL;
     }
 }
 
Signature · 1 line
2.23.0
#0 lore raw

cover A pair of memory access problems

The first patch fixes a crasher, the second fixes a memleak.

Michal Privoznik (2):
hw/vfio/pci: Fix double free of migration_blocker
vfio-helpers: Free QEMUVFIOState in qemu_vfio_close()

 hw/vfio/pci.c       | 2 ++
 util/vfio-helpers.c | 1 +
 2 files changed, 3 insertions(+)
Signature · 1 line
2.23.0
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] hw/vfio/pci: Fix double free of migration_blocker

In reply to Michal Privoznik
Quoted 11 lines · On Mon, 11 Nov 2019 11:37:41 +0100 Michal Privoznik <[email protected]> wrote:

When user tries to hotplug a VFIO device, but the operation fails somewhere in the middle (in my testing it failed because of RLIMIT_MEMLOCK forbidding more memory allocation), then a double free occurs. In vfio_realize() the vdev->migration_blocker is allocated, then something goes wrong which causes control to jump onto 'error' label where the error is freed. But the pointer is left pointing to invalid memory. Later, when vfio_instance_finalize() is called, the memory is freed again.

In my testing the second hunk was sufficient to fix the bug, but I figured the first hunk doesn't hurt either.

Better safe than sorry, I guess.

Quoted 40 lines

==169952== Invalid read of size 8

==169952==    at 0xA47DCD: error_free (error.c:266)
==169952==    by 0x4E0A18: vfio_instance_finalize (pci.c:3040)
==169952==    by 0x8DF74C: object_deinit (object.c:606)
==169952==    by 0x8DF7BE: object_finalize (object.c:620)
==169952==    by 0x8E0757: object_unref (object.c:1074)
==169952==    by 0x45079C: memory_region_unref (memory.c:1779)
==169952==    by 0x45376B: do_address_space_destroy (memory.c:2793)
==169952==    by 0xA5C600: call_rcu_thread (rcu.c:283)
==169952==    by 0xA427CB: qemu_thread_start (qemu-thread-posix.c:519)
==169952==    by 0x80A8457: start_thread (in /lib64/libpthread-2.29.so)
==169952==    by 0x81C96EE: clone (in /lib64/libc-2.29.so)
==169952==  Address 0x143137e0 is 0 bytes inside a block of size 48 free'd
==169952==    at 0x4A342BB: free (vg_replace_malloc.c:530)
==169952==    by 0xA47E05: error_free (error.c:270)
==169952==    by 0x4E0945: vfio_realize (pci.c:3025)
==169952==    by 0x76A4FF: pci_qdev_realize (pci.c:2099)
==169952==    by 0x689B9A: device_set_realized (qdev.c:876)
==169952==    by 0x8E2C80: property_set_bool (object.c:2080)
==169952==    by 0x8E0EF6: object_property_set (object.c:1272)
==169952==    by 0x8E3FC8: object_property_set_qobject (qom-qobject.c:26)
==169952==    by 0x8E11DB: object_property_set_bool (object.c:1338)
==169952==    by 0x5E7BDD: qdev_device_add (qdev-monitor.c:673)
==169952==    by 0x5E81E5: qmp_device_add (qdev-monitor.c:798)
==169952==    by 0x9E18A8: do_qmp_dispatch (qmp-dispatch.c:132)
==169952==  Block was alloc'd at
==169952==    at 0x4A35476: calloc (vg_replace_malloc.c:752)
==169952==    by 0x51B1158: g_malloc0 (in /usr/lib64/libglib-2.0.so.0.6000.6)
==169952==    by 0xA47357: error_setv (error.c:61)
==169952==    by 0xA475D9: error_setg_internal (error.c:97)
==169952==    by 0x4DF8C2: vfio_realize (pci.c:2737)
==169952==    by 0x76A4FF: pci_qdev_realize (pci.c:2099)
==169952==    by 0x689B9A: device_set_realized (qdev.c:876)
==169952==    by 0x8E2C80: property_set_bool (object.c:2080)
==169952==    by 0x8E0EF6: object_property_set (object.c:1272)
==169952==    by 0x8E3FC8: object_property_set_qobject (qom-qobject.c:26)
==169952==    by 0x8E11DB: object_property_set_bool (object.c:1338)
==169952==    by 0x5E7BDD: qdev_device_add (qdev-monitor.c:673)
Quoted 1 line
Fixes f045a0104c8c ("vfio: unplug failover primary device before migration")
Quoted 3 lines
---
 hw/vfio/pci.c | 2 ++
 1 file changed, 2 insertions(+)
#4 lore raw

Re: [PATCH 2/2] vfio-helpers: Free QEMUVFIOState in qemu_vfio_close()

In reply to Michal Privoznik
Quoted 29 lines · On Mon, 11 Nov 2019 11:37:42 +0100 Michal Privoznik <[email protected]> wrote:

The qemu_vfio_open_pci() allocates this QEMUVFIOState structure but free counterpart is missing. Since we already have qemu_vfio_close() which does cleanup of the state, it looks like a perfect place to free the structure too.

==178278== 528 (360 direct, 168 indirect) bytes in 1 blocks are definitely lost in loss record 6,605 of 6,985

==178278==    at 0x4A35476: calloc (vg_replace_malloc.c:752)
==178278==    by 0x51B1158: g_malloc0 (in /usr/lib64/libglib-2.0.so.0.6000.6)
==178278==    by 0xA68613: qemu_vfio_open_pci (vfio-helpers.c:428)
==178278==    by 0x9779EA: nvme_init (nvme.c:606)
==178278==    by 0x97830F: nvme_file_open (nvme.c:795)
==178278==    by 0x8E9439: bdrv_open_driver (block.c:1293)
==178278==    by 0x8E9E1C: bdrv_open_common (block.c:1553)
==178278==    by 0x8ED264: bdrv_open_inherit (block.c:3083)
==178278==    by 0x8ED79D: bdrv_open (block.c:3176)
==178278==    by 0x5DA5C1: bds_tree_init (blockdev.c:670)
==178278==    by 0x5E2B64: qmp_blockdev_add (blockdev.c:4354)
==178278==    by 0x5ECB1D: configure_blockdev (vl.c:1202)
---
 util/vfio-helpers.c | 1 +
 1 file changed, 1 insertion(+)
util/vfio-helpers.c +0 -0
index 813f7ec564..5ff91c1e5c 100644

util/vfio-helpers.c void qemu_vfio_close(QEMUVFIOState *s) · line 724

Quoted 4 lines
util/vfio-helpers.c +1 -0
@@ -721,4 +721,5 @@ void qemu_vfio_close(QEMUVFIOState *s)
     close(s->device);
     close(s->group);
     close(s->container);
+    g_free(s);

Not sure if freeing the parameter passed in via a function called 'close' isn't too surprising... it's not that obvious that the caller is also relinquishing its reference to the QEMUVFIOState; maybe rename the function to qemu_vfio_close_and_free() or so?

[Looking at the blockdev code, it uses the pattern of first closing the bs and then freeing it separately, which is a bit odd as the only call to bdrv_close (which will eventually end up here for nvme) is immediately followed by a g_free. Just something I noticed.]

Quoted 1 line
util/vfio-helpers.c +0 -0
 }
#6 lore raw
In reply to Michal Privoznik
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===

TEST check-unit: tests/test-thread-pool
TEST check-unit: tests/test-hbitmap
**
ERROR:/tmp/qemu-test/src/tests/migration-test.c:903:wait_for_migration_fail: assertion failed: (!strcmp(status, "setup") || !strcmp(status, "failed") || (allow_active && !strcmp(status, "active")))
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/migration-test.c:903:wait_for_migration_fail: assertion failed: (!strcmp(status, "setup") || !strcmp(status, "failed") || (allow_active && !strcmp(status, "active")))
make: *** [check-qtest-aarch64] Error 1
make: *** Waiting for unfinished jobs....
TEST check-unit: tests/test-bdrv-drain
TEST check-unit: tests/test-bdrv-graph-mod
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=e993b5a80ea14863bad17c8751205f9e', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-_b06sxmc/src/docker-src.2019-11-11-09.03.28.17585:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=e993b5a80ea14863bad17c8751205f9e
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-_b06sxmc/src'
make: *** [docker-run-test-quick@centos7] Error 2

real 10m21.635s
user 0m8.000s

The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#5 lore raw

Re: [PATCH 2/2] vfio-helpers: Free QEMUVFIOState in qemu_vfio_close()

In reply to Cornelia Huck
Quoted 27 lines · Cornelia Huck
Quoted 24 lines · On Mon, 11 Nov 2019 11:37:42 +0100 Michal Privoznik <[email protected]> wrote:

The qemu_vfio_open_pci() allocates this QEMUVFIOState structure but free counterpart is missing. Since we already have qemu_vfio_close() which does cleanup of the state, it looks like a perfect place to free the structure too.

==178278== 528 (360 direct, 168 indirect) bytes in 1 blocks are definitely lost in loss record 6,605 of 6,985

==178278==    at 0x4A35476: calloc (vg_replace_malloc.c:752)
==178278==    by 0x51B1158: g_malloc0 (in /usr/lib64/libglib-2.0.so.0.6000.6)
==178278==    by 0xA68613: qemu_vfio_open_pci (vfio-helpers.c:428)
==178278==    by 0x9779EA: nvme_init (nvme.c:606)
==178278==    by 0x97830F: nvme_file_open (nvme.c:795)
==178278==    by 0x8E9439: bdrv_open_driver (block.c:1293)
==178278==    by 0x8E9E1C: bdrv_open_common (block.c:1553)
==178278==    by 0x8ED264: bdrv_open_inherit (block.c:3083)
==178278==    by 0x8ED79D: bdrv_open (block.c:3176)
==178278==    by 0x5DA5C1: bds_tree_init (blockdev.c:670)
==178278==    by 0x5E2B64: qmp_blockdev_add (blockdev.c:4354)
==178278==    by 0x5ECB1D: configure_blockdev (vl.c:1202)
---
  util/vfio-helpers.c | 1 +
  1 file changed, 1 insertion(+)
Quoted 14 lines

util/vfio-helpers.c void qemu_vfio_close(QEMUVFIOState *s) · line 724

Quoted 9 lines
util/vfio-helpers.c +1 -0
index 813f7ec564..5ff91c1e5c 100644
@@ -721,4 +721,5 @@ void qemu_vfio_close(QEMUVFIOState *s)
      close(s->device);
      close(s->group);
      close(s->container);
+    g_free(s);

Not sure if freeing the parameter passed in via a function called 'close' isn't too surprising... it's not that obvious that the caller is also relinquishing its reference to the QEMUVFIOState; maybe rename the function to qemu_vfio_close_and_free() or so?

Alright, I'll rename the function. I worry that if free is left as an exercise to caller then it'll be always forgotten about. That's why I put the call into close function.

Michal

#7 lore raw
In reply to Michal Privoznik
Quoted 9 lines · On Mon, 11 Nov 2019 11:37:40 +0100 Michal Privoznik <[email protected]> wrote:

The first patch fixes a crasher, the second fixes a memleak.

Michal Privoznik (2):
hw/vfio/pci: Fix double free of migration_blocker
vfio-helpers: Free QEMUVFIOState in qemu_vfio_close()

 hw/vfio/pci.c       | 2 ++
 util/vfio-helpers.c | 1 +
 2 files changed, 3 insertions(+)

I'll take the first patch, looks like there's some discussion still on the vfio-helpers patch and I don't own that file anyway, so please post it separately. Thanks,

Alex