block/accounting: fix unlocked latency histogram reconfiguration races
Subsystems (2)
v2 cover block/accounting: fix unlocked latency histogram reconfiguration races
block_latency_histogram_set() and block_latency_histograms_clear() replace BlockLatencyHistogram's nbins/boundaries/bins without taking stats->lock, while block_account_one_io() reads those same fields under that lock from whatever iothread completes the I/O. A histogram reconfiguration (block-latency-histogram-set QMP command, monitor thread) racing an in-flight completion can therefore observe those fields torn, hitting assert(pos != NULL) in block_latency_histogram_account(), or corrupting the heap outright. This showed up as a qemu-kvm SIGABRT on a customer's virtio-blk guest.
Regression test is added for illustrative purpose but I am unsure that it is viable long term. Feel free to drop.
v1 -> v2
* patch 1, 2: WITH_QEMU_LOCK_GUARD -> plain lock()/unlock(), pure additions, nothing reindented or moved
* patch 3: writer thread now also calls block_latency_histograms_clear(), the other function patch 1 fixes
* patch 2: take stats->lock for bdrv_query_blk_stats()'s whole call instead of per-section, since the interval loop's timed_average_min/max/avg() calls race timed_average_account() the same way the already-locked fields did; block_acct_queue_depth() now requires the caller to hold the lock instead of taking it itself, since bdrv_query_blk_stats() is its only caller
Denis V. Lunev (3):
block/accounting: take stats->lock in latency histogram setters block/qapi: take stats->lock when reading BlockAcctStats for
query-blockstats
tests/unit: add reproducer for BlockAcctStats histogram locking race
block/accounting.c | 11 ++- block/qapi.c | 3 + include/block/accounting.h | 1 + tests/unit/meson.build | 1 + tests/unit/test-block-accounting.c | 115 +++++++++++++++++++++++++++++ 5 files changed, 129 insertions(+), 2 deletions(-) create mode 100644 tests/unit/test-block-accounting.c
Signature · 1 line
2.53.0
PATCH v2 1/3 block/accounting: take stats->lock in latency histogram setters
block_latency_histogram_set() and block_latency_histograms_clear() replace BlockLatencyHistogram's nbins/boundaries/bins without taking stats->lock, while block_account_one_io() reads those same fields under that lock from whatever iothread completes the I/O. The result is usual use-after-free and qemu crash.
Take stats->lock in both setters, matching the lock already held by the reader.
--- block/accounting.c | 8 ++++++++ 1 file changed, 8 insertions(+)
block/accounting.c +8 -0
@@ -194,6 +194,8 @@ int block_latency_histogram_set(BlockAcctStats *stats, enum BlockAcctType type,return -EINVAL ; } +qemu_mutex_lock (&stats->lock); + hist->nbins = new_nbins;g_free (hist->boundaries); hist->boundaries =g_new (uint64_t, hist->nbins -1 );
@@ -206,6 +208,8 @@ int block_latency_histogram_set(BlockAcctStats *stats, enum BlockAcctType type,g_free (hist->bins); hist->bins =g_new0 (uint64_t, hist->nbins); +qemu_mutex_unlock (&stats->lock); +return 0 ; }
@@ -213,12 +217,16 @@ void block_latency_histograms_clear(BlockAcctStats *stats) {int i; +qemu_mutex_lock (&stats->lock); +for (i =0 ; i <BLOCK_MAX_IOTYPE ; i++) {BlockLatencyHistogram *hist = &stats->latency_histogram[i];g_free (hist->bins);g_free (hist->boundaries);memset (hist,0 ,sizeof (*hist)); } + +qemu_mutex_unlock (&stats->lock); } static void block_account_one_io(BlockAcctStats *stats, BlockAcctCookie *cookie,
Signature · 1 line
2.53.0
PATCH v2 3/3 tests/unit: add reproducer for BlockAcctStats histogram locking race
In reply to Denis V. Lunevblock_latency_histogram_set() and block_latency_histograms_clear() replace BlockLatencyHistogram's nbins/boundaries/bins without taking stats->lock, while block_account_one_io() reads those same fields under that lock from whatever iothread completes the I/O.
Add a test that races two real threads against block_latency_histogram_set() and block_acct_start()/block_acct_done() on the same BlockAcctStats. Applied here it passes, since the previous two commits already take the lock; reverting them locally reproduces the abort this series fixes, in about a second.
--- tests/unit/meson.build | 1 + tests/unit/test-block-accounting.c | 115 +++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+) create mode 100644 tests/unit/test-block-accounting.c
tests/unit/meson.build +1 -0
@@ -75,6 +75,7 @@ if have_block 'test-blockjob': [testblock], 'test-blockjob-txn': [testblock], 'test-block-backend': [testblock], + 'test-block-accounting': [testblock], 'test-block-iothread': [testblock], 'test-write-threshold': [testblock], 'test-crypto-hash': [crypto],
tests/unit/test-block-accounting.c new file +115 -0
115 lines not on this page · raw message
Signature · 1 line
2.53.0
Kevin Wolf
AppliedAm 24.07.2026 um 13:13 hat Denis V. Lunev geschrieben:
Quoted 13 lines
block_latency_histogram_set()andblock_latency_histograms_clear()replace BlockLatencyHistogram's nbins/boundaries/bins without takingstats->lock, whileblock_account_one_io()reads those same fields under that lock from whatever iothread completes the I/O. A histogram reconfiguration (block-latency-histogram-set QMP command, monitor thread) racing an in-flight completion can therefore observe those fields torn, hitting assert(pos != NULL) inblock_latency_histogram_account(), or corrupting the heap outright. This showed up as a qemu-kvm SIGABRT on a customer's virtio-blk guest.Regression test is added for illustrative purpose but I am unsure that it is viable long term. Feel free to drop.
Quoted 11 lines
v1 -> v2
* patch 1, 2:WITH_QEMU_LOCK_GUARD-> plain lock()/unlock(), pure additions, nothing reindented or moved
* patch 3: writer thread now also callsblock_latency_histograms_clear(), the other function patch 1 fixes
* patch 2: takestats->lockfor bdrv_query_blk_stats()'s whole call instead of per-section, since the interval loop's timed_average_min/max/avg() calls racetimed_average_account()the same way the already-locked fields did;block_acct_queue_depth()now requires the caller to hold the lock instead of taking it itself, sincebdrv_query_blk_stats()is its only caller
Thanks, applied to the block branch.
Kevin
Quoted 10 lines · Denis V. Lunev
block_latency_histogram_set()andblock_latency_histograms_clear()replace BlockLatencyHistogram's nbins/boundaries/bins without takingstats->lock, whileblock_account_one_io()reads those same fields under that lock from whatever iothread completes the I/O. A histogram reconfiguration (block-latency-histogram-set QMP command, monitor thread) racing an in-flight completion can therefore observe those fields torn, hitting assert(pos != NULL) inblock_latency_histogram_account(), or corrupting the heap outright. This showed up as a qemu-kvm SIGABRT on a customer's virtio-blk guest.
Quoted 2 lines
Regression test is added for illustrative purpose but I am unsure that it is viable long term. Feel free to drop.
I'm picking this up for qemu-stable too.
Please let me know if I shouldn't.
And please remember to add Cc: qemu-stable@ for future changes which are applicable for the stable series, to avoid this extra round-trip :)
Thanks,
/mjt