Skip to contentSkip to conversation

block/accounting: fix unlocked latency histogram reconfiguration races

PATCH Applied Denis V. Lunev 3 participants 5 messages Jul 24 – Aug 4

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover block/accounting: fix unlocked latency histogram reconfiguration races

block_latency_histogram_set() and block_latency_histograms_clear() replace BlockLatencyHistogram's nbins/boundaries/bins without taking stats->lock, while block_account_one_io() reads those same fields under that lock from whatever iothread completes the I/O. A histogram reconfiguration (block-latency-histogram-set QMP command, monitor thread) racing an in-flight completion can therefore observe those fields torn, hitting assert(pos != NULL) in block_latency_histogram_account(), or corrupting the heap outright. This showed up as a qemu-kvm SIGABRT on a customer's virtio-blk guest.

Regression test is added for illustrative purpose but I am unsure that it is viable long term. Feel free to drop.

v1 -> v2
* patch 1, 2: WITH_QEMU_LOCK_GUARD -> plain lock()/unlock(), pure additions, nothing reindented or moved
* patch 3: writer thread now also calls block_latency_histograms_clear(), the other function patch 1 fixes
* patch 2: take stats->lock for bdrv_query_blk_stats()'s whole call instead of per-section, since the interval loop's timed_average_min/max/avg() calls race timed_average_account() the same way the already-locked fields did; block_acct_queue_depth() now requires the caller to hold the lock instead of taking it itself, since bdrv_query_blk_stats() is its only caller

CC Kevin Wolf <[email protected]>
CC Hanna Reitz <[email protected]>
CC Vladimir Sementsov-Ogievskiy <[email protected]>
CC Andrey Drobyshev <[email protected]>

Denis V. Lunev (3):
block/accounting: take stats->lock in latency histogram setters block/qapi: take stats->lock when reading BlockAcctStats for
query-blockstats
tests/unit: add reproducer for BlockAcctStats histogram locking race

 block/accounting.c                 |  11 ++-
 block/qapi.c                       |   3 +
 include/block/accounting.h         |   1 +
 tests/unit/meson.build             |   1 +
 tests/unit/test-block-accounting.c | 115 +++++++++++++++++++++++++++++
 5 files changed, 129 insertions(+), 2 deletions(-)
 create mode 100644 tests/unit/test-block-accounting.c
Signature · 1 line
2.53.0
#1 lore raw

PATCH v2 1/3 block/accounting: take stats->lock in latency histogram setters

block_latency_histogram_set() and block_latency_histograms_clear() replace BlockLatencyHistogram's nbins/boundaries/bins without taking stats->lock, while block_account_one_io() reads those same fields under that lock from whatever iothread completes the I/O. The result is usual use-after-free and qemu crash.

Take stats->lock in both setters, matching the lock already held by the reader.

CC Kevin Wolf <[email protected]>
CC Hanna Reitz <[email protected]>
CC Vladimir Sementsov-Ogievskiy <[email protected]>
CC Andrey Drobyshev <[email protected]>
---
 block/accounting.c | 8 ++++++++
 1 file changed, 8 insertions(+)
block/accounting.c +8 -0
index f00fe99740..6e06c7609e 100644
@@ -194,6 +194,8 @@ int block_latency_histogram_set(BlockAcctStats *stats, enum BlockAcctType type,
         return -EINVAL;
     }
 
+    qemu_mutex_lock(&stats->lock);
+
     hist->nbins = new_nbins;
     g_free(hist->boundaries);
     hist->boundaries = g_new(uint64_t, hist->nbins - 1);
@@ -206,6 +208,8 @@ int block_latency_histogram_set(BlockAcctStats *stats, enum BlockAcctType type,
     g_free(hist->bins);
     hist->bins = g_new0(uint64_t, hist->nbins);
 
+    qemu_mutex_unlock(&stats->lock);
+
     return 0;
 }
 
@@ -213,12 +217,16 @@ void block_latency_histograms_clear(BlockAcctStats *stats)
 {
     int i;
 
+    qemu_mutex_lock(&stats->lock);
+
     for (i = 0; i < BLOCK_MAX_IOTYPE; i++) {
         BlockLatencyHistogram *hist = &stats->latency_histogram[i];
         g_free(hist->bins);
         g_free(hist->boundaries);
         memset(hist, 0, sizeof(*hist));
     }
+
+    qemu_mutex_unlock(&stats->lock);
 }
 
 static void block_account_one_io(BlockAcctStats *stats, BlockAcctCookie *cookie,
Signature · 1 line
2.53.0
#2 lore raw

PATCH v2 3/3 tests/unit: add reproducer for BlockAcctStats histogram locking race

In reply to Denis V. Lunev

block_latency_histogram_set() and block_latency_histograms_clear() replace BlockLatencyHistogram's nbins/boundaries/bins without taking stats->lock, while block_account_one_io() reads those same fields under that lock from whatever iothread completes the I/O.

Add a test that races two real threads against block_latency_histogram_set() and block_acct_start()/block_acct_done() on the same BlockAcctStats. Applied here it passes, since the previous two commits already take the lock; reverting them locally reproduces the abort this series fixes, in about a second.

CC Kevin Wolf <[email protected]>
CC Hanna Reitz <[email protected]>
CC Vladimir Sementsov-Ogievskiy <[email protected]>
CC Andrey Drobyshev <[email protected]>
---
 tests/unit/meson.build             |   1 +
 tests/unit/test-block-accounting.c | 115 +++++++++++++++++++++++++++++
 2 files changed, 116 insertions(+)
 create mode 100644 tests/unit/test-block-accounting.c
tests/unit/meson.build +1 -0
index 5ba6b1a230..dc3fb954c0 100644
@@ -75,6 +75,7 @@ if have_block
     'test-blockjob': [testblock],
     'test-blockjob-txn': [testblock],
     'test-block-backend': [testblock],
+    'test-block-accounting': [testblock],
     'test-block-iothread': [testblock],
     'test-write-threshold': [testblock],
     'test-crypto-hash': [crypto],
tests/unit/test-block-accounting.c new file +115 -0

115 lines not on this page · raw message

Signature · 1 line
2.53.0
Applied
#3 lore raw
In reply to Denis V. Lunev

Am 24.07.2026 um 13:13 hat Denis V. Lunev geschrieben:

Quoted 13 lines

block_latency_histogram_set() and block_latency_histograms_clear() replace BlockLatencyHistogram's nbins/boundaries/bins without taking stats->lock, while block_account_one_io() reads those same fields under that lock from whatever iothread completes the I/O. A histogram reconfiguration (block-latency-histogram-set QMP command, monitor thread) racing an in-flight completion can therefore observe those fields torn, hitting assert(pos != NULL) in block_latency_histogram_account(), or corrupting the heap outright. This showed up as a qemu-kvm SIGABRT on a customer's virtio-blk guest.

Regression test is added for illustrative purpose but I am unsure that it is viable long term. Feel free to drop.

Quoted 11 lines

v1 -> v2
* patch 1, 2: WITH_QEMU_LOCK_GUARD -> plain lock()/unlock(), pure additions, nothing reindented or moved
* patch 3: writer thread now also calls block_latency_histograms_clear(), the other function patch 1 fixes
* patch 2: take stats->lock for bdrv_query_blk_stats()'s whole call instead of per-section, since the interval loop's timed_average_min/max/avg() calls race timed_average_account() the same way the already-locked fields did; block_acct_queue_depth() now requires the caller to hold the lock instead of taking it itself, since bdrv_query_blk_stats() is its only caller

Thanks, applied to the block branch.

Kevin

#4 lore raw
In reply to Denis V. Lunev
Quoted 10 lines · Denis V. Lunev

block_latency_histogram_set() and block_latency_histograms_clear() replace BlockLatencyHistogram's nbins/boundaries/bins without taking stats->lock, while block_account_one_io() reads those same fields under that lock from whatever iothread completes the I/O. A histogram reconfiguration (block-latency-histogram-set QMP command, monitor thread) racing an in-flight completion can therefore observe those fields torn, hitting assert(pos != NULL) in block_latency_histogram_account(), or corrupting the heap outright. This showed up as a qemu-kvm SIGABRT on a customer's virtio-blk guest.

Quoted 2 lines

Regression test is added for illustrative purpose but I am unsure that it is viable long term. Feel free to drop.

I'm picking this up for qemu-stable too.
Please let me know if I shouldn't.

And please remember to add Cc: qemu-stable@ for future changes which are applicable for the stable series, to avoid this extra round-trip :)

Thanks,

/mjt