Skip to contentSkip to conversation

aio-poll: improve aio-polling efficiency

RFC PATCH Superseded Jaehoon Kim 4 participants 30 messages Jan 13 – Mar 23

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover aio-poll: improve aio-polling efficiency

Dear all,

I'm submitting this patch series for review under the RFC tag.

This patch series refines the aio_poll adaptive polling logic to reduce unnecessary busy-waiting and improve CPU efficiency.

The first patch prevents redundant polling time calculation when polling is disabled. The second patch enhances the adaptive polling mechanism by dynamically adjusting the iothread's polling duration based on event intervals measured by individual AioHandlers. The third patch introduces a new 'poll-weight' parameter to adjust how much the current interval influences the next polling duration.

We evaluated the patches on an s390x host with a single guest using 16 virtio block devices backed by FCP multipath devices in a separate-disk setup, with the I/O scheduler set to 'none' in both host and guest.

The fio workload included sequential and random read/write with varying numbers of jobs (1,4,8,16) and io_depth of 8. The tests were conducted with single and dual iothreads, using the newly introduced poll-weight parameter to measure their impact on CPU cost and throughput.

Compared to the baseline, across four FIO workload patterns (sequential R/W, random R/W), and averaged over FIO job counts of 1, 4, 8, and 16, throughput decreased slightly (-3% to -8% for one iothread, -2% to -5% for two iothreads), while CPU usage on the s390x host dropped significantly (-10% to -25% and -7% to -12%, respectively).

Best regards,
Jaehoon Kim

Jaehoon Kim (3):
aio-poll: avoid unnecessary polling time computation
aio-poll: refine iothread polling using weighted handler intervals qapi/iothread: introduce poll-weight parameter for aio-poll

 include/qemu/aio.h                |   8 +-
 include/system/iothread.h         |   1 +
 iothread.c                        |  10 ++
 monitor/hmp-cmds.c                |   1 +
 qapi/misc.json                    |   6 ++
 qapi/qom.json                     |   8 +-
 qemu-options.hx                   |   7 +-
 tests/unit/test-nested-aio-poll.c |   2 +-
 util/aio-posix.c                  | 151 +++++++++++++++++++++---------
 util/aio-win32.c                  |   3 +-
 util/async.c                      |   2 +
 11 files changed, 147 insertions(+), 52 deletions(-)
Signature · 1 line
2.50.1
#1 lore raw

RFC PATCH 1/3 aio-poll: avoid unnecessary polling time computation

Nodes are no longer added to poll_aio_handlers when adaptive polling is disabled, preventing unnecessary try_poll_mode() calls. Additionally, aio_poll() skips try_poll_mode() when timeout is 0.

This avoids iterating over all nodes to compute max_ns unnecessarily when polling is disabled or timeout is 0.

---
 util/aio-posix.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
util/aio-posix.c +6 -5
index e24b955fd9..7ddf92a25f 100644
@@ -306,9 +306,8 @@ static bool aio_dispatch_handler(AioContext *ctx, AioHandler *node)
      * fdmon_supports_polling(), but only until the fd fires for the first
      * time.
      */
-    if (!QLIST_IS_INSERTED(node, node_deleted) &&
-        !QLIST_IS_INSERTED(node, node_poll) &&
-        node->io_poll) {
+    if (ctx->poll_max_ns && !QLIST_IS_INSERTED(node, node_deleted) &&
+        !QLIST_IS_INSERTED(node, node_poll) && node->io_poll) {
         trace_poll_add(ctx, node, node->pfd.fd, revents);
         if (ctx->poll_started && node->io_poll_begin) {
             node->io_poll_begin(node->opaque);
@@ -630,7 +629,7 @@ static void adjust_polling_time(AioContext *ctx, AioPolledEvent *poll,
 bool aio_poll(AioContext *ctx, bool blocking)
 {
     AioHandlerList ready_list = QLIST_HEAD_INITIALIZER(ready_list);
-    bool progress;
+    bool progress = false;
     bool use_notify_me;
     int64_t timeout;
     int64_t start = 0;
@@ -655,7 +654,9 @@ bool aio_poll(AioContext *ctx, bool blocking)
     }
 
     timeout = blocking ? aio_compute_timeout(ctx) : 0;
-    progress = try_poll_mode(ctx, &ready_list, &timeout);
+    if ((ctx->poll_max_ns != 0) && (timeout != 0)) {
+        progress = try_poll_mode(ctx, &ready_list, &timeout);
+    }
     assert(!(timeout && progress));
 
     /*
Signature · 1 line
2.50.1
#2 lore raw

Re: [PATCH RFC v1 1/3] aio-poll: avoid unnecessary polling time computation

Quoted 3 lines · Jaehoon Kim

Nodes are no longer added to poll_aio_handlers when adaptive polling is disabled, preventing unnecessary try_poll_mode() calls. Additionally, aio_poll() skips try_poll_mode() when timeout is 0.

Do these two changes need to be made together? If not, please split them into two commits. This will make the commit descriptions easier to understand.

Quoted 35 lines

This avoids iterating over all nodes to compute max_ns unnecessarily when polling is disabled or timeout is 0.

---
 util/aio-posix.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
util/aio-posix.c +3 -4
index e24b955fd9..7ddf92a25f 100644
@@ -306,9 +306,8 @@ static bool aio_dispatch_handler(AioContext *ctx, AioHandler *node)
      * fdmon_supports_polling(), but only until the fd fires for the first
      * time.
      */
-    if (!QLIST_IS_INSERTED(node, node_deleted) &&
-        !QLIST_IS_INSERTED(node, node_poll) &&
-        node->io_poll) {
+    if (ctx->poll_max_ns && !QLIST_IS_INSERTED(node, node_deleted) &&
+        !QLIST_IS_INSERTED(node, node_poll) && node->io_poll) {
         trace_poll_add(ctx, node, node->pfd.fd, revents);
         if (ctx->poll_started && node->io_poll_begin) {
             node->io_poll_begin(node->opaque);
@@ -630,7 +629,7 @@ static void adjust_polling_time(AioContext *ctx, AioPolledEvent *poll,
 bool aio_poll(AioContext *ctx, bool blocking)
 {
     AioHandlerList ready_list = QLIST_HEAD_INITIALIZER(ready_list);
-    bool progress;
+    bool progress = false;
     bool use_notify_me;
     int64_t timeout;
     int64_t start = 0;

util/aio-posix.c bool aio_poll(AioContext *ctx, bool blocking) · line 660

Quoted 8 lines
util/aio-posix.c +3 -1
@@ -655,7 +654,9 @@ bool aio_poll(AioContext *ctx, bool blocking)
     }
 
     timeout = blocking ? aio_compute_timeout(ctx) : 0;
-    progress = try_poll_mode(ctx, &ready_list, &timeout);
+    if ((ctx->poll_max_ns != 0) && (timeout != 0)) {
+        progress = try_poll_mode(ctx, &ready_list, &timeout);
+    }
     assert(!(timeout && progress));

Can you walk me through the timeout == 0 case when polling is active?

Further down in aio_poll():

/* If polling is allowed, non-blocking aio_poll does not need the
* system call---a single round of run_poll_handlers_once suffices. */
if (timeout || ctx->fdmon_ops->need_wait(ctx)) {

My concern is that aio_poll(timeout=0) could return without polling or waiting for fds when polling is active. Maybe I've missed something that prevents this?

Stefan

#3 lore raw

Re: [PATCH RFC v1 1/3] aio-poll: avoid unnecessary polling time computation

In reply to Jaehoon Kim
Quoted 6 lines · Jaehoon Kim

Nodes are no longer added to poll_aio_handlers when adaptive polling is disabled, preventing unnecessary try_poll_mode() calls. Additionally, aio_poll() skips try_poll_mode() when timeout is 0.

This avoids iterating over all nodes to compute max_ns unnecessarily when polling is disabled or timeout is 0.

Did you consider optimizing the case when polling is active and there is no way around calculating the polling time? Glib has ordered data structures that have lower operation costs than O(n).

Stefan

#4 lore raw

Re: [PATCH RFC v1 1/3] aio-poll: avoid unnecessary polling time computation

Quoted 5 lines · Stefan Hajnoczi
Quoted 4 lines · Jaehoon Kim

Nodes are no longer added to poll_aio_handlers when adaptive polling is disabled, preventing unnecessary try_poll_mode() calls. Additionally, aio_poll() skips try_poll_mode() when timeout is 0.

Quoted 7 lines
Quoted 2 lines

This avoids iterating over all nodes to compute max_ns unnecessarily when polling is disabled or timeout is 0.

Did you consider optimizing the case when polling is active and there is no way around calculating the polling time? Glib has ordered data structures that have lower operation costs than O(n).

Stefan

If I understand correctly, while optimizing the data structure to be cheaper than O(n) is one approach, I believe bypassing the function entirely efficient for cases where timeout=0 is frequently triggered.

Since the result of the calculation is not used at all in these cases, it seems to more efficient to skip the function than to optimize calculation itself.

Please let me know if I have misunderstood your point.

Thanks,
Jaehoon Kim.

#5 lore raw

Re: [PATCH RFC v1 1/3] aio-poll: avoid unnecessary polling time computation

Quoted 14 lines · JAEHOON KIM
Quoted 5 lines · Stefan Hajnoczi
Quoted 4 lines · Jaehoon Kim

Nodes are no longer added to poll_aio_handlers when adaptive polling is disabled, preventing unnecessary try_poll_mode() calls. Additionally, aio_poll() skips try_poll_mode() when timeout is 0.

Quoted 7 lines
Quoted 2 lines

This avoids iterating over all nodes to compute max_ns unnecessarily when polling is disabled or timeout is 0.

Did you consider optimizing the case when polling is active and there is no way around calculating the polling time? Glib has ordered data structures that have lower operation costs than O(n).

Stefan

Quoted 8 lines

If I understand correctly, while optimizing the data structure to be cheaper than O(n) is one approach, I believe bypassing the function entirely efficient
for cases where timeout=0 is frequently triggered.

Since the result of the calculation is not used at all in these cases, it seems
to more efficient to skip the function than to optimize calculation itself.

Yes, skipping it when not needed makes sense and I'm in favor of it.

I was thinking that, separately from this patch, the data structure might be worth optimizing since the value needs to be computed when polling is enabled.

Stefan

#6 lore raw

Re: [PATCH RFC v1 1/3] aio-poll: avoid unnecessary polling time computation

Quoted 22 lines · Stefan Hajnoczi
Quoted 13 lines · JAEHOON KIM
Quoted 5 lines · Stefan Hajnoczi
Quoted 4 lines · Jaehoon Kim

Nodes are no longer added to poll_aio_handlers when adaptive polling is disabled, preventing unnecessary try_poll_mode() calls. Additionally, aio_poll() skips try_poll_mode() when timeout is 0.

Quoted 7 lines
Quoted 2 lines

This avoids iterating over all nodes to compute max_ns unnecessarily when polling is disabled or timeout is 0.

Did you consider optimizing the case when polling is active and there is no way around calculating the polling time? Glib has ordered data structures that have lower operation costs than O(n).

Stefan

Quoted 8 lines

If I understand correctly, while optimizing the data structure to be cheaper than O(n) is one approach, I believe bypassing the function entirely efficient
for cases where timeout=0 is frequently triggered.

Since the result of the calculation is not used at all in these cases, it seems
to more efficient to skip the function than to optimize calculation itself.

Quoted 7 lines

Yes, skipping it when not needed makes sense and I'm in favor of it.

I was thinking that, separately from this patch, the data structure might be worth optimizing since the value needs to be computed when polling is enabled.

Stefan

That makes sense. I agree that when polling is enabled, optimizing the data structure could help reduce the overhead of computing the value. I'll take a closer look at potential improvements there.

Thanks,
Jaehoon

#7 lore raw

RFC PATCH 2/3 aio-poll: refine iothread polling using weighted handler intervals

In reply to Jaehoon Kim

Refine adaptive polling in aio_poll by updating iothread polling duration based on weighted AioHandler event intervals.

Each AioHandler's poll.ns is updated using a weighted factor when an event occurs. Idle handlers accumulate block_ns until poll_max_ns and then reset to 0, preventing sporadically active handlers from unnecessarily prolonging iothread polling.

The iothread polling duration is set based on the largest poll.ns among active handlers. The shrink divider defaults to 2, matching the grow rate, to reduce frequent poll_ns resets for slow devices.

The default weight factor (POLL_WEIGHT_SHIFT=2) was selected based on various FIO tests to balance mean poll_ns, reset frequency, and high poll_ns occurrences. Correlation between current block_ns and weighted value (adj_block_ns) slightly decreases as weight increases. Lower weights cause more fluctuation; higher weights maintain poll_ns once it rises.

The table below shows results for a representative randread case (bs=4k, iodepth=8, 2 IOThread), illustrating the average poll_ns, the ratio of poll_ns resets to 0, and the time spent near the maximum poll_ns for different weight values.

Weight| Mean poll_ns |  poll_ns reset rate | Time near max(%)
------| ------------ | ------------------- | -----------------
  1   |     4523     |       89.9%         |     7.41%
  2   |     8442     |       78.6%         |     15.84%
  3   |    11147     |       70.4%         |     21.38%
  4   |    11624     |       70.1%         |     23.35%

Weight=1 reacts quickly, Weight=3-4 holds poll_ns higher once it rises, and Weight=2 provides a good balance between responsiveness and CPU usage.

---
 include/qemu/aio.h |   4 +-
 util/aio-posix.c   | 135 +++++++++++++++++++++++++++++++--------------
 util/async.c       |   1 +
 3 files changed, 99 insertions(+), 41 deletions(-)
include/qemu/aio.h +3 -1
index 8cca2360d1..6c77a190e9 100644
@@ -195,7 +195,8 @@ struct BHListSlice {
 typedef QSLIST_HEAD(, AioHandler) AioHandlerSList;
 
 typedef struct AioPolledEvent {
-    int64_t ns;        /* current polling time in nanoseconds */
+    bool has_event; /* Flag to indicate if an event has occurred */
+    int64_t ns;     /* estimated block time in nanoseconds */
 } AioPolledEvent;
 
 struct AioContext {
@@ -306,6 +307,7 @@ struct AioContext {
     int poll_disable_cnt;
 
     /* Polling mode parameters */
+    int64_t poll_ns;        /* current polling time in nanoseconds */
     int64_t poll_max_ns;    /* maximum polling time in nanoseconds */
     int64_t poll_grow;      /* polling time growth factor */
     int64_t poll_shrink;    /* polling time shrink factor */
util/aio-posix.c +95 -40

193 lines not on this page · raw message

util/async.c +1 -0
index 80d6b01a8a..9d3627566f 100644
@@ -606,6 +606,7 @@ AioContext *aio_context_new(Error **errp)
     timerlistgroup_init(&ctx->tlg, aio_timerlist_notify, ctx);
 
     ctx->poll_max_ns = 0;
+    ctx->poll_ns = 0;
     ctx->poll_grow = 0;
     ctx->poll_shrink = 0;
 
Signature · 1 line
2.50.1
#8 lore raw

RFC PATCH 3/3 qapi/iothread: introduce poll-weight parameter for aio-poll

In reply to Jaehoon Kim

Introduce a new poll-weight parameter for aio-poll. This parameter controls how much the most recent event interval affects the next polling duration. When set to 0, a default value of 2 is used, meaning the current interval contributes roughly 25% to the calculation. Larger values decrease the weight of the current interval, enabling more gradual adjustments to polling duration.

---
 include/qemu/aio.h                |  4 +++-
 include/system/iothread.h         |  1 +
 iothread.c                        | 10 ++++++++++
 monitor/hmp-cmds.c                |  1 +
 qapi/misc.json                    |  6 ++++++
 qapi/qom.json                     |  8 +++++++-
 qemu-options.hx                   |  7 ++++++-
 tests/unit/test-nested-aio-poll.c |  2 +-
 util/aio-posix.c                  |  9 ++++++---
 util/aio-win32.c                  |  3 ++-
 util/async.c                      |  1 +
 11 files changed, 44 insertions(+), 8 deletions(-)
include/qemu/aio.h +3 -1
index 6c77a190e9..50b8db2712 100644
@@ -311,6 +311,7 @@ struct AioContext {
     int64_t poll_max_ns;    /* maximum polling time in nanoseconds */
     int64_t poll_grow;      /* polling time growth factor */
     int64_t poll_shrink;    /* polling time shrink factor */
+    int64_t poll_weight;    /* weight of current interval in calculation */
 
     /* AIO engine parameters */
     int64_t aio_max_batch;  /* maximum number of requests in a batch */
@@ -792,12 +793,13 @@ void aio_context_destroy(AioContext *ctx);
  * @max_ns: how long to busy poll for, in nanoseconds
  * @grow: polling time growth factor
  * @shrink: polling time shrink factor
+ * @weight: weight factor applied to the current polling interval
  *
  * Poll mode can be disabled by setting poll_max_ns to 0.
  */
 void aio_context_set_poll_params(AioContext *ctx, int64_t max_ns,
                                  int64_t grow, int64_t shrink,
-                                 Error **errp);
+                                 int64_t weight, Error **errp);
 
 /**
  * aio_context_set_aio_params:
include/system/iothread.h +1 -0
index e26d13c6c7..6ea57ed126 100644
@@ -38,6 +38,7 @@ struct IOThread {
     int64_t poll_max_ns;
     int64_t poll_grow;
     int64_t poll_shrink;
+    int64_t poll_weight;
 };
 typedef struct IOThread IOThread;
 
iothread.c +10 -0
index caf68e0764..68a944e57c 100644
@@ -164,6 +164,7 @@ static void iothread_set_aio_context_params(EventLoopBase *base, Error **errp)
                                 iothread->poll_max_ns,
                                 iothread->poll_grow,
                                 iothread->poll_shrink,
+                                iothread->poll_weight,
                                 errp);
     if (*errp) {
         return;
@@ -233,6 +234,9 @@ static IOThreadParamInfo poll_grow_info = {
 static IOThreadParamInfo poll_shrink_info = {
     "poll-shrink", offsetof(IOThread, poll_shrink),
 };
+static IOThreadParamInfo poll_weight_info = {
+    "poll-weight", offsetof(IOThread, poll_weight),
+};
 
 static void iothread_get_param(Object *obj, Visitor *v,
         const char *name, IOThreadParamInfo *info, Error **errp)
@@ -288,6 +292,7 @@ static void iothread_set_poll_param(Object *obj, Visitor *v,
                                     iothread->poll_max_ns,
                                     iothread->poll_grow,
                                     iothread->poll_shrink,
+                                    iothread->poll_weight,
                                     errp);
     }
 }
@@ -311,6 +316,10 @@ static void iothread_class_init(ObjectClass *klass, const void *class_data)
                               iothread_get_poll_param,
                               iothread_set_poll_param,
                               NULL, &poll_shrink_info);
+    object_class_property_add(klass, "poll-weight", "int",
+                              iothread_get_poll_param,
+                              iothread_set_poll_param,
+                              NULL, &poll_weight_info);
 }
 
 static const TypeInfo iothread_info = {
@@ -356,6 +365,7 @@ static int query_one_iothread(Object *object, void *opaque)
     info->poll_max_ns = iothread->poll_max_ns;
     info->poll_grow = iothread->poll_grow;
     info->poll_shrink = iothread->poll_shrink;
+    info->poll_weight = iothread->poll_weight;
     info->aio_max_batch = iothread->parent_obj.aio_max_batch;
 
     QAPI_LIST_APPEND(*tail, info);
monitor/hmp-cmds.c +1 -0
index 5a673cddb2..40e3b1da50 100644
@@ -205,6 +205,7 @@ void hmp_info_iothreads(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "  poll-max-ns=%" PRId64 "\n", value->poll_max_ns);
         monitor_printf(mon, "  poll-grow=%" PRId64 "\n", value->poll_grow);
         monitor_printf(mon, "  poll-shrink=%" PRId64 "\n", value->poll_shrink);
+        monitor_printf(mon, "  poll-weight=%" PRId64 "\n", value->poll_weight);
         monitor_printf(mon, "  aio-max-batch=%" PRId64 "\n",
                        value->aio_max_batch);
     }
qapi/misc.json +6 -0
index 28c641fe2f..b21cc48a03 100644
@@ -85,6 +85,11 @@
 # @poll-shrink: how many ns will be removed from polling time, 0 means
 #     that it's not configured (since 2.9)
 #
+# @poll-weight: the weight factor for adaptive polling.
+#     Determines how much the current event interval contributes to
+#     the next polling time calculation.  0 means that the default
+#     value is used.  (since 10.1)
+#
 # @aio-max-batch: maximum number of requests in a batch for the AIO
 #     engine, 0 means that the engine will use its default (since 6.1)
 #
@@ -96,6 +101,7 @@
            'poll-max-ns': 'int',
            'poll-grow': 'int',
            'poll-shrink': 'int',
+           'poll-weight': 'int',
            'aio-max-batch': 'int' } }
 
 ##
qapi/qom.json +7 -1
index 6f5c9de0f0..d90823478d 100644
@@ -606,6 +606,11 @@
 #     algorithm detects it is spending too long polling without
 #     encountering events.  0 selects a default behaviour (default: 0)
 #
+# @poll-weight: the weight factor for adaptive polling.
+#     Determines how much the current event interval contributes to
+#     the next polling time calculation.  0 selects a default
+#     behaviour (default: 0) since 10.1.
+#
 # The @aio-max-batch option is available since 6.1.
 #
 # Since: 2.0
@@ -614,7 +619,8 @@
   'base': 'EventLoopBaseProperties',
   'data': { '*poll-max-ns': 'int',
             '*poll-grow': 'int',
-            '*poll-shrink': 'int' } }
+            '*poll-shrink': 'int',
+            '*poll-weight': 'int' } }
 
 ##
 # @MainLoopProperties:
qemu-options.hx +6 -1
index ec92723f10..74adaf55fc 100644
@@ -6352,7 +6352,7 @@ SRST
 
             CN=laptop.example.com,O=Example Home,L=London,ST=London,C=GB
 
-    ``-object iothread,id=id,poll-max-ns=poll-max-ns,poll-grow=poll-grow,poll-shrink=poll-shrink,aio-max-batch=aio-max-batch``
+    ``-object iothread,id=id,poll-max-ns=poll-max-ns,poll-grow=poll-grow,poll-shrink=poll-shrink,poll-weight=poll-weight,aio-max-batch=aio-max-batch``
         Creates a dedicated event loop thread that devices can be
         assigned to. This is known as an IOThread. By default device
         emulation happens in vCPU threads or the main event loop thread.
@@ -6388,6 +6388,11 @@ SRST
         the polling time when the algorithm detects it is spending too
         long polling without encountering events.
 
+        The ``poll-weight`` parameter is the weight factor used in the
+        adaptive polling algorithm. It determines how much the most
+        recent event interval affects the calculation of the next
+        polling duration.
+
         The ``aio-max-batch`` parameter is the maximum number of requests
         in a batch for the AIO engine, 0 means that the engine will use
         its default.
tests/unit/test-nested-aio-poll.c +1 -1
index 9ab1ad08a7..ed791fa23b 100644
@@ -81,7 +81,7 @@ static void test(void)
     qemu_set_current_aio_context(td.ctx);
 
     /* Enable polling */
-    aio_context_set_poll_params(td.ctx, 1000000, 2, 2, &error_abort);
+    aio_context_set_poll_params(td.ctx, 1000000, 2, 2, 0, &error_abort);
 
     /* Make the event notifier active (set) right away */
     event_notifier_init(&td.poll_notifier, 1);
util/aio-posix.c +6 -3
index dd6008898b..d4f3d8ca8f 100644
@@ -630,6 +630,7 @@ static void adjust_block_ns(AioContext *ctx, int64_t block_ns)
 {
     AioHandler *node;
     int64_t adj_block_ns = -1;
+    int64_t poll_weight = ctx->poll_weight ? : POLL_WEIGHT_SHIFT;
 
     QLIST_FOREACH(node, &ctx->poll_aio_handlers, node_poll) {
         if (node->poll.has_event) {
@@ -639,8 +640,8 @@ static void adjust_block_ns(AioContext *ctx, int64_t block_ns)
              * poll.ns to smooth out polling time adjustments.
              */
             node->poll.ns = node->poll.ns
-                ? (node->poll.ns - (node->poll.ns >> POLL_WEIGHT_SHIFT))
-                + (block_ns >> POLL_WEIGHT_SHIFT) : block_ns;
+                ? (node->poll.ns - (node->poll.ns >> poll_weight))
+                + (block_ns >> poll_weight) : block_ns;
 
             if (node->poll.ns >= ctx->poll_max_ns) {
                 node->poll.ns = 0;
@@ -830,7 +831,8 @@ void aio_context_destroy(AioContext *ctx)
 }
 
 void aio_context_set_poll_params(AioContext *ctx, int64_t max_ns,
-                                 int64_t grow, int64_t shrink, Error **errp)
+                                 int64_t grow, int64_t shrink,
+                                 int64_t weight, Error **errp)
 {
     AioHandler *node;
 
@@ -847,6 +849,7 @@ void aio_context_set_poll_params(AioContext *ctx, int64_t max_ns,
     ctx->poll_max_ns = max_ns;
     ctx->poll_grow = grow;
     ctx->poll_shrink = shrink;
+    ctx->poll_weight = weight;
     ctx->poll_ns = 0;
 
     aio_notify(ctx);
util/aio-win32.c +2 -1
index 6e6f699e4b..1985843233 100644
@@ -429,7 +429,8 @@ void aio_context_destroy(AioContext *ctx)
 }
 
 void aio_context_set_poll_params(AioContext *ctx, int64_t max_ns,
-                                 int64_t grow, int64_t shrink, Error **errp)
+                                 int64_t grow, int64_t shrink,
+                                 int64_t weight, Error **errp)
 {
     if (max_ns) {
         error_setg(errp, "AioContext polling is not implemented on Windows");
util/async.c +1 -0
index 9d3627566f..741fcfd6a7 100644
@@ -609,6 +609,7 @@ AioContext *aio_context_new(Error **errp)
     ctx->poll_ns = 0;
     ctx->poll_grow = 0;
     ctx->poll_shrink = 0;
+    ctx->poll_weight = 0;
 
     ctx->aio_max_batch = 0;
 
Signature · 1 line
2.50.1
#9 lore raw

Re: [PATCH RFC v1 3/3] qapi/iothread: introduce poll-weight parameter for aio-poll

Quoted 8 lines · Jaehoon Kim

Introduce a new poll-weight parameter for aio-poll. This parameter controls how much the most recent event interval affects the next polling duration. When set to 0, a default value of 2 is used, meaning the current interval contributes roughly 25% to the calculation. Larger values decrease the weight of the current interval, enabling more gradual adjustments to polling duration.

[...]

qapi/misc.json line 91

Quoted 12 lines
qapi/misc.json +4 -0
index 28c641fe2f..b21cc48a03 100644
@@ -85,6 +85,11 @@
 # @poll-shrink: how many ns will be removed from polling time, 0 means
 #     that it's not configured (since 2.9)
 #
+# @poll-weight: the weight factor for adaptive polling.
+#     Determines how much the current event interval contributes to
+#     the next polling time calculation.  0 means that the default
+#     value is used.  (since 10.1)

When the default value is used, the actual value being used remains hidden. Why?

Quoted 17 lines
qapi/misc.json +2 -0
+#
 # @aio-max-batch: maximum number of requests in a batch for the AIO
 #     engine, 0 means that the engine will use its default (since 6.1)
 #
@@ -96,6 +101,7 @@
            'poll-max-ns': 'int',
            'poll-grow': 'int',
            'poll-shrink': 'int',
+           'poll-weight': 'int',
            'aio-max-batch': 'int' } }
 
 ##
qapi/qom.json +0 -0
index 6f5c9de0f0..d90823478d 100644

qapi/qom.json line 612

Quoted 7 lines
qapi/qom.json +4 -0
@@ -606,6 +606,11 @@
 #     algorithm detects it is spending too long polling without
 #     encountering events.  0 selects a default behaviour (default: 0)
 #
+# @poll-weight: the weight factor for adaptive polling.
+#     Determines how much the current event interval contributes to
+#     the next polling time calculation.  0 selects a default
+#     behaviour (default: 0) since 10.1.

This leaves the actual default behavior unspecified. Is this a good idea?

Quoted 14 lines
qapi/qom.json +3 -1
+#
 # The @aio-max-batch option is available since 6.1.
 #
 # Since: 2.0
@@ -614,7 +619,8 @@
   'base': 'EventLoopBaseProperties',
   'data': { '*poll-max-ns': 'int',
             '*poll-grow': 'int',
-            '*poll-shrink': 'int' } }
+            '*poll-shrink': 'int',
+            '*poll-weight': 'int' } }
 
 ##
 # @MainLoopProperties:

[...]

#10 lore raw

Re: [PATCH RFC v1 3/3] qapi/iothread: introduce poll-weight parameter for aio-poll

Quoted 12 lines · Markus Armbruster
Quoted 8 lines · Jaehoon Kim

Introduce a new poll-weight parameter for aio-poll. This parameter controls how much the most recent event interval affects the next polling duration. When set to 0, a default value of 2 is used, meaning the current interval contributes roughly 25% to the calculation. Larger values decrease the weight of the current interval, enabling more gradual adjustments to polling duration.

[...]

Quoted 14 lines

qapi/misc.json line 91

Quoted 12 lines
qapi/misc.json +4 -0
index 28c641fe2f..b21cc48a03 100644
@@ -85,6 +85,11 @@
  # @poll-shrink: how many ns will be removed from polling time, 0 means
  #     that it's not configured (since 2.9)
  #
+# @poll-weight: the weight factor for adaptive polling.
+#     Determines how much the current event interval contributes to
+#     the next polling time calculation.  0 means that the default
+#     value is used.  (since 10.1)

When the default value is used, the actual value being used remains hidden. Why?

Actually, I just followed the existing pattern of poll-grow, which also defaults to a factor of 2 when set to 0. It wasn't my intention to hide the value; I kept this because the previous API has been working fine without issues. If you think the actual value should be visible, I'll consider ways to make it explicit in the next version.

Quoted 11 lines
Quoted 11 lines
qapi/misc.json +2 -0
+#
  # @aio-max-batch: maximum number of requests in a batch for the AIO
  #     engine, 0 means that the engine will use its default (since 6.1)
  #
@@ -96,6 +101,7 @@
             'poll-max-ns': 'int',
             'poll-grow': 'int',
             'poll-shrink': 'int',
+           'poll-weight': 'int',
             'aio-max-batch': 'int' } }
 
Quoted 15 lines

qapi/qom.json line 612

Quoted 13 lines
qapi/misc.json +0 -0
  ##
qapi/qom.json +4 -0
index 6f5c9de0f0..d90823478d 100644
@@ -606,6 +606,11 @@
  #     algorithm detects it is spending too long polling without
  #     encountering events.  0 selects a default behaviour (default: 0)
  #
+# @poll-weight: the weight factor for adaptive polling.
+#     Determines how much the current event interval contributes to
+#     the next polling time calculation.  0 selects a default
+#     behaviour (default: 0) since 10.1.

This leaves the actual default behavior unspecified. Is this a good idea?

I agree that the documentation should be more explicit.
I'll update it to clarify that the default factor is 2 and explain its meaning.

Quoted 17 lines
Quoted 14 lines
qapi/qom.json +3 -1
+#
  # The @aio-max-batch option is available since 6.1.
  #
  # Since: 2.0
@@ -614,7 +619,8 @@
    'base': 'EventLoopBaseProperties',
    'data': { '*poll-max-ns': 'int',
              '*poll-grow': 'int',
-            '*poll-shrink': 'int' } }
+            '*poll-shrink': 'int',
+            '*poll-weight': 'int' } }
 
  ##
  # @MainLoopProperties:

[...]

#11 lore raw

Re: [PATCH RFC v1 3/3] qapi/iothread: introduce poll-weight parameter for aio-poll

Cc: [email protected] for a possible query-iothreads change discussed below.

Quoted 29 lines · JAEHOON KIM
Quoted 12 lines · Markus Armbruster
Quoted 8 lines · Jaehoon Kim

Introduce a new poll-weight parameter for aio-poll. This parameter controls how much the most recent event interval affects the next polling duration. When set to 0, a default value of 2 is used, meaning the current interval contributes roughly 25% to the calculation. Larger values decrease the weight of the current interval, enabling more gradual adjustments to polling duration.

[...]

Quoted 15 lines

qapi/misc.json line 91

Quoted 12 lines
qapi/misc.json +4 -0
index 28c641fe2f..b21cc48a03 100644
@@ -85,6 +85,11 @@
 # @poll-shrink: how many ns will be removed from polling time, 0 means
 #     that it's not configured (since 2.9)
 #
+# @poll-weight: the weight factor for adaptive polling.
+#     Determines how much the current event interval contributes to
+#     the next polling time calculation.  0 means that the default
+#     value is used.  (since 10.1)

When the default value is used, the actual value being used remains hidden. Why?

Quoted 2 lines

Actually, I just followed the existing pattern of poll-grow, which also defaults to a factor of 2 when set to 0.

Yes, and consistency is always desirable. But let's have a look at the new interface in isolation, to see whether it's actually good.

Quoted 4 lines

It wasn't my intention to hide the value; I kept this because the previous API has been working fine without issues.
If you think the actual value should be visible, I'll consider ways to make it explicit in the next version.

As is, query-iothreads tells us "the weight factor for adaptive polling is X, and it was set by the user", or "the weight factor for adaptive polling was not set by the user, but picked by the system."

If we returned the actual value, it would tell us "the weight factor for adaptive polling is X".

Only the former interface tells us whether the user or the system picked.

Only the latter interface tells us what the system picked.

Which one is useful in practice?

I'd argue the latter. A management application knows whether it set a value without query-iothreads' help, but it doesn't know what the system picked. The people coding it may know if a contract specifies what the system picks (see below).

If we conclude that returning the actual value is better for new @poll-weight, then it would surely be better for @poll-grow and @poll-shrink, too. Could we still improve them?

Libvirt developers, any advice?

Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
qapi/misc.json +2 -0
+#
 # @aio-max-batch: maximum number of requests in a batch for the AIO
 #     engine, 0 means that the engine will use its default (since 6.1)
 #
@@ -96,6 +101,7 @@
             'poll-max-ns': 'int',
             'poll-grow': 'int',
             'poll-shrink': 'int',
+           'poll-weight': 'int',
             'aio-max-batch': 'int' } }
 
Quoted 16 lines

qapi/qom.json line 612

Quoted 13 lines
qapi/misc.json +0 -0
 ##
qapi/qom.json +4 -0
index 6f5c9de0f0..d90823478d 100644
@@ -606,6 +606,11 @@
 #     algorithm detects it is spending too long polling without
 #     encountering events.  0 selects a default behaviour (default: 0)
 #
+# @poll-weight: the weight factor for adaptive polling.
+#     Determines how much the current event interval contributes to
+#     the next polling time calculation.  0 selects a default
+#     behaviour (default: 0) since 10.1.

This leaves the actual default behavior unspecified. Is this a good idea?

Quoted 3 lines

I agree that the documentation should be more explicit. I'll update it to clarify that the default factor is 2 and explain its meaning.

I understand that you're mirroring how @poll-grow and @poll-shrink work, but let's ignore that for a minute.

Compare four possible interfaces:

1. Optional @poll-weight defaults to 2. Values <= 0 are rejected.

2. Optional @poll-weight defaults to 2. Value 0 is replaced by the default value 2. Values < 0 are rejected.

3. Optional @poll-weight defaults to 0. Values < 0 are rejected. Value 0 makes the system pick a value, namely 2.

4. Optional @poll-weight defaults to 0. Values < 0 are rejected. Value 0 makes the system pick a value. It currently picks 2.

The difference between 3. and 4. is that 3. makes "system picks 2" part of the contract, while 4. doesn't.

1. is the simplest. Is 2.'s additional complexity worthwhile? 3.'s? 4.'s?

Quoted 16 lines
Quoted 16 lines
Quoted 14 lines
qapi/qom.json +3 -1
+#
 # The @aio-max-batch option is available since 6.1.
 #
 # Since: 2.0
@@ -614,7 +619,8 @@
    'base': 'EventLoopBaseProperties',
    'data': { '*poll-max-ns': 'int',
              '*poll-grow': 'int',
-            '*poll-shrink': 'int' } }
+            '*poll-shrink': 'int',
+            '*poll-weight': 'int' } }
 
 ##
 # @MainLoopProperties:

[...]

#12 lore raw

Re: [PATCH RFC v1 3/3] qapi/iothread: introduce poll-weight parameter for aio-poll

Quoted 13 lines · On Thu, 15 Jan 2026 08:28:51 +0100 Markus Armbruster <[email protected]> wrote:

I understand that you're mirroring how @poll-grow and @poll-shrink work, but let's ignore that for a minute.

Compare four possible interfaces:

1. Optional @poll-weight defaults to 2. Values <= 0 are rejected.

2. Optional @poll-weight defaults to 2. Value 0 is replaced by the default value 2. Values < 0 are rejected.

3. Optional @poll-weight defaults to 0. Values < 0 are rejected. Value 0 makes the system pick a value, namely 2.

Quoted 8 lines

4. Optional @poll-weight defaults to 0. Values < 0 are rejected. Value 0 makes the system pick a value. It currently picks 2.

The difference between 3. and 4. is that 3. makes "system picks 2" part of the contract, while 4. doesn't.

1. is the simplest. Is 2.'s additional complexity worthwhile? 3.'s? 4.'s?

Isn't there more options? Like

5. Optional @poll-weight defaults to system-default. Value 0 is replaced by the system pick the system default value. Currently the system default value is 2. Values < 0 are rejected.

That would mean:
* current value inspectable
* system default not part of the interface contract
* interface offers a "please go back to value not user specified:
operation

BTW I like your approach with explicitly listing and evaluating the options a lot!

Regards,
Halil

#13 lore raw

Re: [PATCH RFC v1 3/3] qapi/iothread: introduce poll-weight parameter for aio-poll

Quoted 36 lines · Halil Pasic
Quoted 13 lines · On Thu, 15 Jan 2026 08:28:51 +0100 Markus Armbruster <[email protected]> wrote:

I understand that you're mirroring how @poll-grow and @poll-shrink work, but let's ignore that for a minute.

Compare four possible interfaces:

1. Optional @poll-weight defaults to 2. Values <= 0 are rejected.

2. Optional @poll-weight defaults to 2. Value 0 is replaced by the default value 2. Values < 0 are rejected.

3. Optional @poll-weight defaults to 0. Values < 0 are rejected. Value 0 makes the system pick a value, namely 2.

Quoted 8 lines

4. Optional @poll-weight defaults to 0. Values < 0 are rejected. Value 0 makes the system pick a value. It currently picks 2.

The difference between 3. and 4. is that 3. makes "system picks 2" part of the contract, while 4. doesn't.

1. is the simplest. Is 2.'s additional complexity worthwhile? 3.'s? 4.'s?

Isn't there more options? Like

5. Optional @poll-weight defaults to system-default. Value 0 is replaced by the system pick the system default value. Currently the system default value is 2. Values < 0 are rejected.

That would mean:
* current value inspectable
* system default not part of the interface contract
* interface offers a "please go back to value not user specified:
operation

Quoted 6 lines

BTW I like your approach with explicitly listing and evaluating the options a lot!

Regards,
Halil

Thank you both for laying out options 1-5 so clearly; the detailed breakdown was very helpful.
After considering the trade-offs, I agree that "Option 1" is the simplest and most robust interface. It ensures the value exposed to users always reflects the actual effect. Option 5 is a clever way to reset values, but I'm leaning toward Option 1 to keep the interface as predictable as possible.
Avoiding special meanings for '0' makes the logic easier for users to reason about.

I will update the next revision to follow "Option 1". Thanks again for the feedback!

#14 lore raw

Re: [PATCH RFC v1 3/3] qapi/iothread: introduce poll-weight parameter for aio-poll

In reply to Halil Pasic
Quoted 19 lines · Halil Pasic
Quoted 16 lines · On Thu, 15 Jan 2026 08:28:51 +0100 Markus Armbruster <[email protected]> wrote:

I understand that you're mirroring how @poll-grow and @poll-shrink work, but let's ignore that for a minute.

Compare four possible interfaces:

1. Optional @poll-weight defaults to 2. Values <= 0 are rejected.

2. Optional @poll-weight defaults to 2. Value 0 is replaced by the default value 2. Values < 0 are rejected.

3. Optional @poll-weight defaults to 0. Values < 0 are rejected. Value 0 makes the system pick a value, namely 2.

4. Optional @poll-weight defaults to 0. Values < 0 are rejected. Value 0 makes the system pick a value. It currently picks 2.

Quoted 7 lines
Quoted 5 lines

The difference between 3. and 4. is that 3. makes "system picks 2" part of the contract, while 4. doesn't.

1. is the simplest. Is 2.'s additional complexity worthwhile? 3.'s? 4.'s?

Isn't there more options? Like

Yes :)

Quoted 4 lines

5. Optional @poll-weight defaults to system-default. Value 0 is replaced by the system pick the system default value. Currently the system default value is 2. Values < 0 are rejected.

Quoted 8 lines

That would mean:
* current value inspectable
* system default not part of the interface contract
* interface offers a "please go back to value not user specified:
operation

BTW I like your approach with explicitly listing and evaluating the options a lot!

Thanks!

#15 lore raw
In reply to Jaehoon Kim
Quoted 9 lines · Jaehoon Kim

We evaluated the patches on an s390x host with a single guest using 16 virtio block devices backed by FCP multipath devices in a separate-disk setup, with the I/O scheduler set to 'none' in both host and guest.

The fio workload included sequential and random read/write with varying numbers of jobs (1,4,8,16) and io_depth of 8. The tests were conducted with single and dual iothreads, using the newly introduced poll-weight parameter to measure their impact on CPU cost and throughput.

Quoted 5 lines

Compared to the baseline, across four FIO workload patterns (sequential R/W, random R/W), and averaged over FIO job counts of 1, 4, 8, and 16, throughput decreased slightly (-3% to -8% for one iothread, -2% to -5% for two iothreads), while CPU usage on the s390x host dropped significantly (-10% to -25% and -7% to -12%, respectively).

Hi Jaehoon,
I would like to run the same fio benchmarks on a local NVMe drive (<10us request latency) to see how that type of hardware configuration is affected. Are the scripts and fio job files available somewhere?

Thanks,
Stefan

#16 lore raw
Quoted 15 lines · Stefan Hajnoczi
Quoted 9 lines · Jaehoon Kim

We evaluated the patches on an s390x host with a single guest using 16 virtio block devices backed by FCP multipath devices in a separate-disk setup, with the I/O scheduler set to 'none' in both host and guest.

The fio workload included sequential and random read/write with varying numbers of jobs (1,4,8,16) and io_depth of 8. The tests were conducted with single and dual iothreads, using the newly introduced poll-weight parameter to measure their impact on CPU cost and throughput.

Quoted 5 lines

Compared to the baseline, across four FIO workload patterns (sequential R/W, random R/W), and averaged over FIO job counts of 1, 4, 8, and 16, throughput decreased slightly (-3% to -8% for one iothread, -2% to -5% for two iothreads), while CPU usage on the s390x host dropped significantly (-10% to -25% and -7% to -12%, respectively).

Quoted 7 lines

Hi Jaehoon,
I would like to run the same fio benchmarks on a local NVMe drive (<10us request latency) to see how that type of hardware configuration is affected. Are the scripts and fio job files available somewhere?

Thanks,
Stefan

Thank you for your reply.
The fio scripts are not available in a location you can access, but there is nothing particularly special in the settings. I’m sharing below the methodology and test setup used by our performance team.

Guest Setup
----------------------
- 12 vCPUs, 4 GiB memory
- 16 virtio disks based on the FCP multipath devices in the host

FIO test parameters
-----------------------
- FIO Version: fio-3.33
- Filesize: 2G
- Blocksize: 8K / 128K
- Direct I/O: 1
- FIO I/O Engine: libaio
- NUMJOB List: 1, 4, 8, 16
- IODEPTH: 8
- Runtime (s): 150

Two FIO samples for random read
--------------------------------
fio --direct=1 --name=test --numjobs=16 --filename=base.0.0:base.1.0:base.2.0:base.3.0:base.4.0:base.5.0:base.6.0:base.7.0:base.8.0:base.9.0:base.10.0:base.11.0:base.12.0:base.13.0:base.14.0:base.15.0 --size=32G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k
fio --direct=1 --name=test --numjobs=4 --filename=subw1/base.0.0:subw4/base.3.0:subw8/base.7.0:subw12/base.11.0:subw16/base.15.0 --size=8G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k

additional notes
----------------
- Each file is placed on a separate disk device mounted under subw<n> as specified in --filename=....
- We execute one warmup run, then two measurement runs and calculate the average

Please let me know if you need any additional information.

Regards,
Jaehoon Kim

#17 lore raw
Quoted 50 lines · JAEHOON KIM
Quoted 15 lines · Stefan Hajnoczi
Quoted 9 lines · Jaehoon Kim

We evaluated the patches on an s390x host with a single guest using 16 virtio block devices backed by FCP multipath devices in a separate-disk setup, with the I/O scheduler set to 'none' in both host and guest.

The fio workload included sequential and random read/write with varying numbers of jobs (1,4,8,16) and io_depth of 8. The tests were conducted with single and dual iothreads, using the newly introduced poll-weight parameter to measure their impact on CPU cost and throughput.

Quoted 5 lines

Compared to the baseline, across four FIO workload patterns (sequential R/W, random R/W), and averaged over FIO job counts of 1, 4, 8, and 16, throughput decreased slightly (-3% to -8% for one iothread, -2% to -5% for two iothreads), while CPU usage on the s390x host dropped significantly (-10% to -25% and -7% to -12%, respectively).

Quoted 7 lines

Hi Jaehoon,
I would like to run the same fio benchmarks on a local NVMe drive (<10us request latency) to see how that type of hardware configuration is affected. Are the scripts and fio job files available somewhere?

Thanks,
Stefan

Thank you for your reply.
The fio scripts are not available in a location you can access, but there is nothing particularly special in the settings. I’m sharing below the methodology and test setup used by our performance team.

Guest Setup
----------------------
- 12 vCPUs, 4 GiB memory
- 16 virtio disks based on the FCP multipath devices in the host

FIO test parameters
-----------------------
- FIO Version: fio-3.33
- Filesize: 2G
- Blocksize: 8K / 128K
- Direct I/O: 1
- FIO I/O Engine: libaio
- NUMJOB List: 1, 4, 8, 16
- IODEPTH: 8
- Runtime (s): 150

Two FIO samples for random read
--------------------------------
fio --direct=1 --name=test --numjobs=16 --filename=base.0.0:base.1.0:base.2.0:base.3.0:base.4.0:base.5.0:base.6.0:base.7.0:base.8.0:base.9.0:base.10.0:base.11.0:base.12.0:base.13.0:base.14.0:base.15.0 --size=32G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k
fio --direct=1 --name=test --numjobs=4 --filename=subw1/base.0.0:subw4/base.3.0:subw8/base.7.0:subw12/base.11.0:subw16/base.15.0 --size=8G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k

Quoted 4 lines

additional notes
----------------
- Each file is placed on a separate disk device mounted under subw<n> as specified in --filename=....
- We execute one warmup run, then two measurement runs and calculate the average

Thanks, I will share x86_64 with fast local NVMe results when I have collected them.

Stefan

#18 lore raw
In reply to JAEHOON KIM
Quoted 50 lines · JAEHOON KIM
Quoted 15 lines · Stefan Hajnoczi
Quoted 9 lines · Jaehoon Kim

We evaluated the patches on an s390x host with a single guest using 16 virtio block devices backed by FCP multipath devices in a separate-disk setup, with the I/O scheduler set to 'none' in both host and guest.

The fio workload included sequential and random read/write with varying numbers of jobs (1,4,8,16) and io_depth of 8. The tests were conducted with single and dual iothreads, using the newly introduced poll-weight parameter to measure their impact on CPU cost and throughput.

Quoted 5 lines

Compared to the baseline, across four FIO workload patterns (sequential R/W, random R/W), and averaged over FIO job counts of 1, 4, 8, and 16, throughput decreased slightly (-3% to -8% for one iothread, -2% to -5% for two iothreads), while CPU usage on the s390x host dropped significantly (-10% to -25% and -7% to -12%, respectively).

Quoted 7 lines

Hi Jaehoon,
I would like to run the same fio benchmarks on a local NVMe drive (<10us request latency) to see how that type of hardware configuration is affected. Are the scripts and fio job files available somewhere?

Thanks,
Stefan

Thank you for your reply.
The fio scripts are not available in a location you can access, but there is nothing particularly special in the settings. I’m sharing below the methodology and test setup used by our performance team.

Guest Setup
----------------------
- 12 vCPUs, 4 GiB memory
- 16 virtio disks based on the FCP multipath devices in the host

FIO test parameters
-----------------------
- FIO Version: fio-3.33
- Filesize: 2G
- Blocksize: 8K / 128K
- Direct I/O: 1
- FIO I/O Engine: libaio
- NUMJOB List: 1, 4, 8, 16
- IODEPTH: 8
- Runtime (s): 150

Two FIO samples for random read
--------------------------------
fio --direct=1 --name=test --numjobs=16 --filename=base.0.0:base.1.0:base.2.0:base.3.0:base.4.0:base.5.0:base.6.0:base.7.0:base.8.0:base.9.0:base.10.0:base.11.0:base.12.0:base.13.0:base.14.0:base.15.0 --size=32G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k
fio --direct=1 --name=test --numjobs=4 --filename=subw1/base.0.0:subw4/base.3.0:subw8/base.7.0:subw12/base.11.0:subw16/base.15.0 --size=8G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k

Quoted 4 lines

additional notes
----------------
- Each file is placed on a separate disk device mounted under subw<n> as specified in --filename=....
- We execute one warmup run, then two measurement runs and calculate the average

Hi Jaehoon,
I ran fio benchmarks on an Intel Optane SSD DC P4800X Series drive (<10 microsecond latency). This is with just 1 drive.

The 8 KiB block size results show something similar to what you
reported: there are IOPS (or throughput) regressions and CPU utilization improvements.

Although the CPU improvements are welcome, I think the default behavior should only be changed if the IOPS regressions can be brought below 5%.

The regressions seem to happen regardless of whether 1 or 2 IOThreads are configured. CPU utilization is different (98% vs 78%) depending on the number of IOThreads, so the regressions happen across a range of CPU utilizations.

The 128 KiB block size results are not interesting because the drive already saturates at numjobs=1. This is expected since the drive cannot go much above ~2 GiB/s throughput.

You can find the Ansible playbook, libvirt domain XML, fio command-lines, and the fio/sar data here:

https://gitlab.com/stefanha/virt-playbooks/-/tree/aio-polling-efficiency

Please let me know if you'd like me to rerun the benchmark with new patches or a configuration change.

Do you want to have a video call to discuss your work and how to get the patches merged?

Host
----
CPU: Intel Xeon Silver 4214 CPU @ 2.20GHz
RAM: 32 GiB

Guest
-----
vCPUs: 8
RAM: 4 GiB
Disk: 1 virtio-blk aio=native cache=none

IOPS
----

rw        bs   numjobs iothreads iops   diff
randread  8k   1       1         163417 -7.8%
randread  8k   1       2         165041 -2.4%
randread  8k   4       1         221508 -0.64%
randread  8k   4       2         251298 0.008%
randread  8k   8       1         222128 -0.51%
randread  8k   8       2         249489 -2.6%
randread  8k   16      1         230535 -0.18%
randread  8k   16      2         246732 -0.22%
randread  128k 1       1          17616 -0.11%
randread  128k 1       2          17678 0.027%
randread  128k 4       1          17536 -0.27%
randread  128k 4       2          17610 -0.031%
randread  128k 8       1          17369 -0.42%
randread  128k 8       2          17433 -0.071%
randread  128k 16      1          17215 -0.61%
randread  128k 16      2          17269 -0.22%
randwrite 8k   1       1         156597 -3.1%
randwrite 8k   1       2         157720 -3.8%
randwrite 8k   4       1         218448 -0.5%
randwrite 8k   4       2         247075 -5.1%
randwrite 8k   8       1         220866 -0.75%
randwrite 8k   8       2         260935 -0.011%
randwrite 8k   16      1         230913 0.23%
randwrite 8k   16      2         261125 -0.01%
randwrite 128k 1       1          16009 0.094%
randwrite 128k 1       2          16070 0.035%
randwrite 128k 4       1          16073 -0.62%
randwrite 128k 4       2          16131 0.059%
randwrite 128k 8       1          16106 0.092%
randwrite 128k 8       2          16153 0.048%
randwrite 128k 16      1          16102 -0.0091%
randwrite 128k 16      2          16160 0.048%

IOThread CPU usage
------------------

iothreads before  after
1         98.7    95.81
2         78.43   66.13

Stefan

#19 lore raw
Quoted 132 lines · Stefan Hajnoczi
Quoted 49 lines · JAEHOON KIM
Quoted 15 lines · Stefan Hajnoczi
Quoted 9 lines · Jaehoon Kim

We evaluated the patches on an s390x host with a single guest using 16 virtio block devices backed by FCP multipath devices in a separate-disk setup, with the I/O scheduler set to 'none' in both host and guest.

The fio workload included sequential and random read/write with varying numbers of jobs (1,4,8,16) and io_depth of 8. The tests were conducted with single and dual iothreads, using the newly introduced poll-weight parameter to measure their impact on CPU cost and throughput.

Quoted 5 lines

Compared to the baseline, across four FIO workload patterns (sequential R/W, random R/W), and averaged over FIO job counts of 1, 4, 8, and 16, throughput decreased slightly (-3% to -8% for one iothread, -2% to -5% for two iothreads), while CPU usage on the s390x host dropped significantly (-10% to -25% and -7% to -12%, respectively).

Quoted 7 lines

Hi Jaehoon,
I would like to run the same fio benchmarks on a local NVMe drive (<10us request latency) to see how that type of hardware configuration is affected. Are the scripts and fio job files available somewhere?

Thanks,
Stefan

Thank you for your reply.
The fio scripts are not available in a location you can access, but there is nothing particularly special in the settings. I’m sharing below the methodology and test setup used by our performance team.

Guest Setup
----------------------
- 12 vCPUs, 4 GiB memory
- 16 virtio disks based on the FCP multipath devices in the host

FIO test parameters
-----------------------
- FIO Version: fio-3.33
- Filesize: 2G
- Blocksize: 8K / 128K
- Direct I/O: 1
- FIO I/O Engine: libaio
- NUMJOB List: 1, 4, 8, 16
- IODEPTH: 8
- Runtime (s): 150

Two FIO samples for random read
--------------------------------
fio --direct=1 --name=test --numjobs=16 --filename=base.0.0:base.1.0:base.2.0:base.3.0:base.4.0:base.5.0:base.6.0:base.7.0:base.8.0:base.9.0:base.10.0:base.11.0:base.12.0:base.13.0:base.14.0:base.15.0 --size=32G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k
fio --direct=1 --name=test --numjobs=4 --filename=subw1/base.0.0:subw4/base.3.0:subw8/base.7.0:subw12/base.11.0:subw16/base.15.0 --size=8G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k

Quoted 4 lines

additional notes
----------------
- Each file is placed on a separate disk device mounted under subw<n> as specified in --filename=....
- We execute one warmup run, then two measurement runs and calculate the average

Hi Jaehoon,
I ran fio benchmarks on an Intel Optane SSD DC P4800X Series drive (<10 microsecond latency). This is with just 1 drive.

The 8 KiB block size results show something similar to what you
reported: there are IOPS (or throughput) regressions and CPU utilization improvements.

Although the CPU improvements are welcome, I think the default behavior should only be changed if the IOPS regressions can be brought below 5%.

The regressions seem to happen regardless of whether 1 or 2 IOThreads are configured. CPU utilization is different (98% vs 78%) depending on the number of IOThreads, so the regressions happen across a range of CPU utilizations.

The 128 KiB block size results are not interesting because the drive already saturates at numjobs=1. This is expected since the drive cannot go much above ~2 GiB/s throughput.

You can find the Ansible playbook, libvirt domain XML, fio command-lines, and the fio/sar data here:

https://gitlab.com/stefanha/virt-playbooks/-/tree/aio-polling-efficiency

Please let me know if you'd like me to rerun the benchmark with new patches or a configuration change.

Do you want to have a video call to discuss your work and how to get the patches merged?

Host
----
CPU: Intel Xeon Silver 4214 CPU @ 2.20GHz
RAM: 32 GiB

Guest
-----
vCPUs: 8
RAM: 4 GiB
Disk: 1 virtio-blk aio=native cache=none

IOPS
----

rw        bs   numjobs iothreads iops   diff
randread  8k   1       1         163417 -7.8%
randread  8k   1       2         165041 -2.4%
randread  8k   4       1         221508 -0.64%
randread  8k   4       2         251298 0.008%
randread  8k   8       1         222128 -0.51%
randread  8k   8       2         249489 -2.6%
randread  8k   16      1         230535 -0.18%
randread  8k   16      2         246732 -0.22%
randread  128k 1       1          17616 -0.11%
randread  128k 1       2          17678 0.027%
randread  128k 4       1          17536 -0.27%
randread  128k 4       2          17610 -0.031%
randread  128k 8       1          17369 -0.42%
randread  128k 8       2          17433 -0.071%
randread  128k 16      1          17215 -0.61%
randread  128k 16      2          17269 -0.22%
randwrite 8k   1       1         156597 -3.1%
randwrite 8k   1       2         157720 -3.8%
randwrite 8k   4       1         218448 -0.5%
randwrite 8k   4       2         247075 -5.1%
randwrite 8k   8       1         220866 -0.75%
randwrite 8k   8       2         260935 -0.011%
randwrite 8k   16      1         230913 0.23%
randwrite 8k   16      2         261125 -0.01%
randwrite 128k 1       1          16009 0.094%
randwrite 128k 1       2          16070 0.035%
randwrite 128k 4       1          16073 -0.62%
randwrite 128k 4       2          16131 0.059%
randwrite 128k 8       1          16106 0.092%
randwrite 128k 8       2          16153 0.048%
randwrite 128k 16      1          16102 -0.0091%
randwrite 128k 16      2          16160 0.048%
Quoted 7 lines

IOThread CPU usage
------------------

iothreads before  after
1         98.7    95.81
2         78.43   66.13

Stefan

Hello Stefan,

Thank you very much for your effort in running these benchmarks. The results show a pattern very similar to what our performance team observed.

I fully agree with the 5% threshold for the default behavior. However, we need an approach that balances the current performance oriented polling scheme with CPU efficiency.

I found that relying on grow/shrink parameters was too limited to achieve these results. This is why I've adjusted the process using a weight-based grow/shrink approach to ensure the polling window remains robust against jitter. Specifically, it avoids abrupt resets to zero by implementing a gradual shrink rather than an immediate reset, even when device latency exceeds the threshold.

As seen in both your results and our team's measurements, this may lead to a bit of a performance trade-off, but it provides a reasonable balance for CPU-sensitive environment.

Thank you for suggesting the video call and I am also looking forward to hearing your thoughts. I'm on US Central Time. Except for Tuesday, I can adjust my schedule to a time that works for you.

Please let me know your preferred time.

Regards,
Jaehoon Kim

#20 lore raw
Quoted 162 lines · JAEHOON KIM
Quoted 132 lines · Stefan Hajnoczi
Quoted 49 lines · JAEHOON KIM
Quoted 15 lines · Stefan Hajnoczi
Quoted 9 lines · Jaehoon Kim

We evaluated the patches on an s390x host with a single guest using 16 virtio block devices backed by FCP multipath devices in a separate-disk setup, with the I/O scheduler set to 'none' in both host and guest.

The fio workload included sequential and random read/write with varying numbers of jobs (1,4,8,16) and io_depth of 8. The tests were conducted with single and dual iothreads, using the newly introduced poll-weight parameter to measure their impact on CPU cost and throughput.

Quoted 5 lines

Compared to the baseline, across four FIO workload patterns (sequential R/W, random R/W), and averaged over FIO job counts of 1, 4, 8, and 16, throughput decreased slightly (-3% to -8% for one iothread, -2% to -5% for two iothreads), while CPU usage on the s390x host dropped significantly (-10% to -25% and -7% to -12%, respectively).

Quoted 7 lines

Hi Jaehoon,
I would like to run the same fio benchmarks on a local NVMe drive (<10us request latency) to see how that type of hardware configuration is affected. Are the scripts and fio job files available somewhere?

Thanks,
Stefan

Thank you for your reply.
The fio scripts are not available in a location you can access, but there is nothing particularly special in the settings. I’m sharing below the methodology and test setup used by our performance team.

Guest Setup
----------------------
- 12 vCPUs, 4 GiB memory
- 16 virtio disks based on the FCP multipath devices in the host

FIO test parameters
-----------------------
- FIO Version: fio-3.33
- Filesize: 2G
- Blocksize: 8K / 128K
- Direct I/O: 1
- FIO I/O Engine: libaio
- NUMJOB List: 1, 4, 8, 16
- IODEPTH: 8
- Runtime (s): 150

Two FIO samples for random read
--------------------------------
fio --direct=1 --name=test --numjobs=16 --filename=base.0.0:base.1.0:base.2.0:base.3.0:base.4.0:base.5.0:base.6.0:base.7.0:base.8.0:base.9.0:base.10.0:base.11.0:base.12.0:base.13.0:base.14.0:base.15.0 --size=32G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k
fio --direct=1 --name=test --numjobs=4 --filename=subw1/base.0.0:subw4/base.3.0:subw8/base.7.0:subw12/base.11.0:subw16/base.15.0 --size=8G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k

Quoted 4 lines

additional notes
----------------
- Each file is placed on a separate disk device mounted under subw<n> as specified in --filename=....
- We execute one warmup run, then two measurement runs and calculate the average

Hi Jaehoon,
I ran fio benchmarks on an Intel Optane SSD DC P4800X Series drive (<10 microsecond latency). This is with just 1 drive.

The 8 KiB block size results show something similar to what you
reported: there are IOPS (or throughput) regressions and CPU utilization improvements.

Although the CPU improvements are welcome, I think the default behavior should only be changed if the IOPS regressions can be brought below 5%.

The regressions seem to happen regardless of whether 1 or 2 IOThreads are configured. CPU utilization is different (98% vs 78%) depending on the number of IOThreads, so the regressions happen across a range of CPU utilizations.

The 128 KiB block size results are not interesting because the drive already saturates at numjobs=1. This is expected since the drive cannot go much above ~2 GiB/s throughput.

You can find the Ansible playbook, libvirt domain XML, fio command-lines, and the fio/sar data here:

https://gitlab.com/stefanha/virt-playbooks/-/tree/aio-polling-efficiency

Please let me know if you'd like me to rerun the benchmark with new patches or a configuration change.

Do you want to have a video call to discuss your work and how to get the patches merged?

Host
----
CPU: Intel Xeon Silver 4214 CPU @ 2.20GHz
RAM: 32 GiB

Guest
-----
vCPUs: 8
RAM: 4 GiB
Disk: 1 virtio-blk aio=native cache=none

IOPS
----

rw        bs   numjobs iothreads iops   diff
randread  8k   1       1         163417 -7.8%
randread  8k   1       2         165041 -2.4%
randread  8k   4       1         221508 -0.64%
randread  8k   4       2         251298 0.008%
randread  8k   8       1         222128 -0.51%
randread  8k   8       2         249489 -2.6%
randread  8k   16      1         230535 -0.18%
randread  8k   16      2         246732 -0.22%
randread  128k 1       1          17616 -0.11%
randread  128k 1       2          17678 0.027%
randread  128k 4       1          17536 -0.27%
randread  128k 4       2          17610 -0.031%
randread  128k 8       1          17369 -0.42%
randread  128k 8       2          17433 -0.071%
randread  128k 16      1          17215 -0.61%
randread  128k 16      2          17269 -0.22%
randwrite 8k   1       1         156597 -3.1%
randwrite 8k   1       2         157720 -3.8%
randwrite 8k   4       1         218448 -0.5%
randwrite 8k   4       2         247075 -5.1%
randwrite 8k   8       1         220866 -0.75%
randwrite 8k   8       2         260935 -0.011%
randwrite 8k   16      1         230913 0.23%
randwrite 8k   16      2         261125 -0.01%
randwrite 128k 1       1          16009 0.094%
randwrite 128k 1       2          16070 0.035%
randwrite 128k 4       1          16073 -0.62%
randwrite 128k 4       2          16131 0.059%
randwrite 128k 8       1          16106 0.092%
randwrite 128k 8       2          16153 0.048%
randwrite 128k 16      1          16102 -0.0091%
randwrite 128k 16      2          16160 0.048%
Quoted 7 lines

IOThread CPU usage
------------------

iothreads before  after
1         98.7    95.81
2         78.43   66.13

Stefan

Hello Stefan,

Thank you very much for your effort in running these benchmarks. The results show a pattern very similar to what our performance team observed.

I fully agree with the 5% threshold for the default behavior. However, we need an approach that balances the current performance oriented polling scheme with CPU efficiency.

I found that relying on grow/shrink parameters was too limited to achieve these results. This is why I've adjusted the process using a weight-based grow/shrink approach to ensure the polling window remains robust against jitter. Specifically, it avoids abrupt resets to zero by implementing a gradual shrink rather than an immediate reset, even when device latency exceeds the threshold.

As seen in both your results and our team's measurements, this may lead to a bit of a performance trade-off, but it provides a reasonable balance for CPU-sensitive environment.

Quoted 5 lines

Thank you for suggesting the video call and I am also looking forward to hearing your thoughts. I'm on US Central Time. Except for Tuesday, I can adjust my schedule to a time that works for you.

Please let me know your preferred time.

Is Monday, February 16th at 10:00am CST good for you? If not, please feel free to pick any time on Monday.

Meeting link: https://meet.jit.si/AioPollingOptimization

Anyone else interested in this topic is welcome to join.

Thanks,
Stefan

#21 lore raw
Quoted 170 lines · Stefan Hajnoczi
Quoted 161 lines · JAEHOON KIM
Quoted 132 lines · Stefan Hajnoczi
Quoted 49 lines · JAEHOON KIM
Quoted 15 lines · Stefan Hajnoczi
Quoted 9 lines · Jaehoon Kim

We evaluated the patches on an s390x host with a single guest using 16 virtio block devices backed by FCP multipath devices in a separate-disk setup, with the I/O scheduler set to 'none' in both host and guest.

The fio workload included sequential and random read/write with varying numbers of jobs (1,4,8,16) and io_depth of 8. The tests were conducted with single and dual iothreads, using the newly introduced poll-weight parameter to measure their impact on CPU cost and throughput.

Quoted 5 lines

Compared to the baseline, across four FIO workload patterns (sequential R/W, random R/W), and averaged over FIO job counts of 1, 4, 8, and 16, throughput decreased slightly (-3% to -8% for one iothread, -2% to -5% for two iothreads), while CPU usage on the s390x host dropped significantly (-10% to -25% and -7% to -12%, respectively).

Quoted 7 lines

Hi Jaehoon,
I would like to run the same fio benchmarks on a local NVMe drive (<10us request latency) to see how that type of hardware configuration is affected. Are the scripts and fio job files available somewhere?

Thanks,
Stefan

Thank you for your reply.
The fio scripts are not available in a location you can access, but there is nothing particularly special in the settings. I’m sharing below the methodology and test setup used by our performance team.

Guest Setup
----------------------
- 12 vCPUs, 4 GiB memory
- 16 virtio disks based on the FCP multipath devices in the host

FIO test parameters
-----------------------
- FIO Version: fio-3.33
- Filesize: 2G
- Blocksize: 8K / 128K
- Direct I/O: 1
- FIO I/O Engine: libaio
- NUMJOB List: 1, 4, 8, 16
- IODEPTH: 8
- Runtime (s): 150

Two FIO samples for random read
--------------------------------
fio --direct=1 --name=test --numjobs=16 --filename=base.0.0:base.1.0:base.2.0:base.3.0:base.4.0:base.5.0:base.6.0:base.7.0:base.8.0:base.9.0:base.10.0:base.11.0:base.12.0:base.13.0:base.14.0:base.15.0 --size=32G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k
fio --direct=1 --name=test --numjobs=4 --filename=subw1/base.0.0:subw4/base.3.0:subw8/base.7.0:subw12/base.11.0:subw16/base.15.0 --size=8G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k

Quoted 4 lines

additional notes
----------------
- Each file is placed on a separate disk device mounted under subw<n> as specified in --filename=....
- We execute one warmup run, then two measurement runs and calculate the average

Hi Jaehoon,
I ran fio benchmarks on an Intel Optane SSD DC P4800X Series drive (<10 microsecond latency). This is with just 1 drive.

The 8 KiB block size results show something similar to what you
reported: there are IOPS (or throughput) regressions and CPU utilization improvements.

Although the CPU improvements are welcome, I think the default behavior should only be changed if the IOPS regressions can be brought below 5%.

The regressions seem to happen regardless of whether 1 or 2 IOThreads are configured. CPU utilization is different (98% vs 78%) depending on the number of IOThreads, so the regressions happen across a range of CPU utilizations.

The 128 KiB block size results are not interesting because the drive already saturates at numjobs=1. This is expected since the drive cannot go much above ~2 GiB/s throughput.

You can find the Ansible playbook, libvirt domain XML, fio command-lines, and the fio/sar data here:

https://gitlab.com/stefanha/virt-playbooks/-/tree/aio-polling-efficiency

Please let me know if you'd like me to rerun the benchmark with new patches or a configuration change.

Do you want to have a video call to discuss your work and how to get the patches merged?

Host
----
CPU: Intel Xeon Silver 4214 CPU @ 2.20GHz
RAM: 32 GiB

Guest
-----
vCPUs: 8
RAM: 4 GiB
Disk: 1 virtio-blk aio=native cache=none

IOPS
----

rw        bs   numjobs iothreads iops   diff
randread  8k   1       1         163417 -7.8%
randread  8k   1       2         165041 -2.4%
randread  8k   4       1         221508 -0.64%
randread  8k   4       2         251298 0.008%
randread  8k   8       1         222128 -0.51%
randread  8k   8       2         249489 -2.6%
randread  8k   16      1         230535 -0.18%
randread  8k   16      2         246732 -0.22%
randread  128k 1       1          17616 -0.11%
randread  128k 1       2          17678 0.027%
randread  128k 4       1          17536 -0.27%
randread  128k 4       2          17610 -0.031%
randread  128k 8       1          17369 -0.42%
randread  128k 8       2          17433 -0.071%
randread  128k 16      1          17215 -0.61%
randread  128k 16      2          17269 -0.22%
randwrite 8k   1       1         156597 -3.1%
randwrite 8k   1       2         157720 -3.8%
randwrite 8k   4       1         218448 -0.5%
randwrite 8k   4       2         247075 -5.1%
randwrite 8k   8       1         220866 -0.75%
randwrite 8k   8       2         260935 -0.011%
randwrite 8k   16      1         230913 0.23%
randwrite 8k   16      2         261125 -0.01%
randwrite 128k 1       1          16009 0.094%
randwrite 128k 1       2          16070 0.035%
randwrite 128k 4       1          16073 -0.62%
randwrite 128k 4       2          16131 0.059%
randwrite 128k 8       1          16106 0.092%
randwrite 128k 8       2          16153 0.048%
randwrite 128k 16      1          16102 -0.0091%
randwrite 128k 16      2          16160 0.048%
Quoted 7 lines

IOThread CPU usage
------------------

iothreads before  after
1         98.7    95.81
2         78.43   66.13

Stefan

Hello Stefan,

Thank you very much for your effort in running these benchmarks. The results show a pattern very similar to what our performance team observed.

I fully agree with the 5% threshold for the default behavior. However, we need an approach that balances the current performance oriented polling scheme with CPU efficiency.

I found that relying on grow/shrink parameters was too limited to achieve these results. This is why I've adjusted the process using a weight-based grow/shrink approach to ensure the polling window remains robust against jitter. Specifically, it avoids abrupt resets to zero by implementing a gradual shrink rather than an immediate reset, even when device latency exceeds the threshold.

As seen in both your results and our team's measurements, this may lead to a bit of a performance trade-off, but it provides a reasonable balance for CPU-sensitive environment.

Quoted 5 lines

Thank you for suggesting the video call and I am also looking forward to hearing your thoughts. I'm on US Central Time. Except for Tuesday, I can adjust my schedule to a time that works for you.

Please let me know your preferred time.

Is Monday, February 16th at 10:00am CST good for you? If not, please feel free to pick any time on Monday.

Quoted 6 lines

Meeting link: https://meet.jit.si/AioPollingOptimization

Anyone else interested in this topic is welcome to join.

Thanks,
Stefan

Thank you for the invite, Stefan.
Monday at 10:00 AM CST works well for me.
I'll make sure to be there and look forward to the discussion. See you then!

Thanks,
Jaehoon

#22 lore raw
Quoted 173 lines · JAEHOON KIM
Quoted 172 lines · Stefan Hajnoczi
Quoted 161 lines · JAEHOON KIM
Quoted 132 lines · Stefan Hajnoczi
Quoted 49 lines · JAEHOON KIM
Quoted 15 lines · Stefan Hajnoczi
Quoted 9 lines · Jaehoon Kim

We evaluated the patches on an s390x host with a single guest using 16 virtio block devices backed by FCP multipath devices in a separate-disk setup, with the I/O scheduler set to 'none' in both host and guest.

The fio workload included sequential and random read/write with varying numbers of jobs (1,4,8,16) and io_depth of 8. The tests were conducted with single and dual iothreads, using the newly introduced poll-weight parameter to measure their impact on CPU cost and throughput.

Quoted 5 lines

Compared to the baseline, across four FIO workload patterns (sequential R/W, random R/W), and averaged over FIO job counts of 1, 4, 8, and 16, throughput decreased slightly (-3% to -8% for one iothread, -2% to -5% for two iothreads), while CPU usage on the s390x host dropped significantly (-10% to -25% and -7% to -12%, respectively).

Quoted 7 lines

Hi Jaehoon,
I would like to run the same fio benchmarks on a local NVMe drive (<10us request latency) to see how that type of hardware configuration is affected. Are the scripts and fio job files available somewhere?

Thanks,
Stefan

Thank you for your reply.
The fio scripts are not available in a location you can access, but there is nothing particularly special in the settings. I’m sharing below the methodology and test setup used by our performance team.

Guest Setup
----------------------
- 12 vCPUs, 4 GiB memory
- 16 virtio disks based on the FCP multipath devices in the host

FIO test parameters
-----------------------
- FIO Version: fio-3.33
- Filesize: 2G
- Blocksize: 8K / 128K
- Direct I/O: 1
- FIO I/O Engine: libaio
- NUMJOB List: 1, 4, 8, 16
- IODEPTH: 8
- Runtime (s): 150

Two FIO samples for random read
--------------------------------
fio --direct=1 --name=test --numjobs=16 --filename=base.0.0:base.1.0:base.2.0:base.3.0:base.4.0:base.5.0:base.6.0:base.7.0:base.8.0:base.9.0:base.10.0:base.11.0:base.12.0:base.13.0:base.14.0:base.15.0 --size=32G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k
fio --direct=1 --name=test --numjobs=4 --filename=subw1/base.0.0:subw4/base.3.0:subw8/base.7.0:subw12/base.11.0:subw16/base.15.0 --size=8G --time_based --runtime=4m --readwrite=randread --ioengine=libaio --iodepth=8 --bs=8k

Quoted 4 lines

additional notes
----------------
- Each file is placed on a separate disk device mounted under subw<n> as specified in --filename=....
- We execute one warmup run, then two measurement runs and calculate the average

Hi Jaehoon,
I ran fio benchmarks on an Intel Optane SSD DC P4800X Series drive (<10 microsecond latency). This is with just 1 drive.

The 8 KiB block size results show something similar to what you
reported: there are IOPS (or throughput) regressions and CPU utilization improvements.

Although the CPU improvements are welcome, I think the default behavior should only be changed if the IOPS regressions can be brought below 5%.

The regressions seem to happen regardless of whether 1 or 2 IOThreads are configured. CPU utilization is different (98% vs 78%) depending on the number of IOThreads, so the regressions happen across a range of CPU utilizations.

The 128 KiB block size results are not interesting because the drive already saturates at numjobs=1. This is expected since the drive cannot go much above ~2 GiB/s throughput.

You can find the Ansible playbook, libvirt domain XML, fio command-lines, and the fio/sar data here:

https://gitlab.com/stefanha/virt-playbooks/-/tree/aio-polling-efficiency

Please let me know if you'd like me to rerun the benchmark with new patches or a configuration change.

Do you want to have a video call to discuss your work and how to get the patches merged?

Host
----
CPU: Intel Xeon Silver 4214 CPU @ 2.20GHz
RAM: 32 GiB

Guest
-----
vCPUs: 8
RAM: 4 GiB
Disk: 1 virtio-blk aio=native cache=none

IOPS
----

rw        bs   numjobs iothreads iops   diff
randread  8k   1       1         163417 -7.8%
randread  8k   1       2         165041 -2.4%
randread  8k   4       1         221508 -0.64%
randread  8k   4       2         251298 0.008%
randread  8k   8       1         222128 -0.51%
randread  8k   8       2         249489 -2.6%
randread  8k   16      1         230535 -0.18%
randread  8k   16      2         246732 -0.22%
randread  128k 1       1          17616 -0.11%
randread  128k 1       2          17678 0.027%
randread  128k 4       1          17536 -0.27%
randread  128k 4       2          17610 -0.031%
randread  128k 8       1          17369 -0.42%
randread  128k 8       2          17433 -0.071%
randread  128k 16      1          17215 -0.61%
randread  128k 16      2          17269 -0.22%
randwrite 8k   1       1         156597 -3.1%
randwrite 8k   1       2         157720 -3.8%
randwrite 8k   4       1         218448 -0.5%
randwrite 8k   4       2         247075 -5.1%
randwrite 8k   8       1         220866 -0.75%
randwrite 8k   8       2         260935 -0.011%
randwrite 8k   16      1         230913 0.23%
randwrite 8k   16      2         261125 -0.01%
randwrite 128k 1       1          16009 0.094%
randwrite 128k 1       2          16070 0.035%
randwrite 128k 4       1          16073 -0.62%
randwrite 128k 4       2          16131 0.059%
randwrite 128k 8       1          16106 0.092%
randwrite 128k 8       2          16153 0.048%
randwrite 128k 16      1          16102 -0.0091%
randwrite 128k 16      2          16160 0.048%
Quoted 7 lines

IOThread CPU usage
------------------

iothreads before  after
1         98.7    95.81
2         78.43   66.13

Stefan

Hello Stefan,

Thank you very much for your effort in running these benchmarks. The results show a pattern very similar to what our performance team observed.

I fully agree with the 5% threshold for the default behavior. However, we need an approach that balances the current performance oriented polling scheme with CPU efficiency.

I found that relying on grow/shrink parameters was too limited to achieve these results. This is why I've adjusted the process using a weight-based grow/shrink approach to ensure the polling window remains robust against jitter. Specifically, it avoids abrupt resets to zero by implementing a gradual shrink rather than an immediate reset, even when device latency exceeds the threshold.

As seen in both your results and our team's measurements, this may lead to a bit of a performance trade-off, but it provides a reasonable balance for CPU-sensitive environment.

Quoted 5 lines

Thank you for suggesting the video call and I am also looking forward to hearing your thoughts. I'm on US Central Time. Except for Tuesday, I can adjust my schedule to a time that works for you.

Please let me know your preferred time.

Is Monday, February 16th at 10:00am CST good for you? If not, please feel free to pick any time on Monday.

Meeting link: https://meet.jit.si/AioPollingOptimization

Quoted 8 lines
Quoted 4 lines

Anyone else interested in this topic is welcome to join.

Thanks,
Stefan

Thank you for the invite, Stefan.
Monday at 10:00 AM CST works well for me.
I'll make sure to be there and look forward to the discussion. See you then!

Great, talk to you soon!

Stefan

#23 lore raw
In reply to Jaehoon Kim

Hi Jaehoon,
Following the call earlier this week I ran a single fio job to get a clearer picture of:
1. The QEMU 10.0.0 regression that prompted you to optimize AioContext polling.
2. How the poll-weight parameter affects IOPS.

run      rw        bs   numjobs iothreads iops   diff
v9.2.0   randread  8k   1       1         174944 3.6%
v10.0.0  randread  8k   1       1         174285 3.2%
baseline randread  8k   1       1         168908 0.0%
w2       randread  8k   1       1         163718 -3.1%
w3       randread  8k   1       1         165805 -1.8%
w4       randread  8k   1       1         167388 -0.9%

This time I only ran randread bs=8k iodepth=8 numjobs=1 with a single IOThread.

Observations:

- There might be an IOPS regression between v10.0.0 and the baseline (9ad7f544c696) that your patches apply on top of. This is different from the CPU utilization regression that you found in v9.2.0 -> v10.0.0. I will bisect it.

- poll-weight=3 and 4 improve IOPS to a level that is acceptable. CPU utilization looks like this:

run %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle

baseline   49.37      0.00     31.10      0.00      0.00     11.61      0.04      0.00      0.00      7.89
w2         46.24      0.00     32.61      0.00      0.00     11.84      0.10      0.00      0.00      9.21
w3         48.04      0.00     32.17      0.00      0.00     11.98      0.08      0.00      0.00      7.73
w4         48.56      0.00     31.23      0.00      0.00     11.48      0.03      0.00      0.00      8.69

poll-weight=2 is the winner at CPU utilization. I'm not sure if poll-weight=3 will produce an acceptable CPU utilization improvement for you. Do you have data or want to re-run to measure poll-weight=3?

Stefan

#24 lore raw
Quoted 34 lines · Stefan Hajnoczi

Hi Jaehoon,
Following the call earlier this week I ran a single fio job to get a clearer picture of:
1. The QEMU 10.0.0 regression that prompted you to optimize AioContext polling.
2. How the poll-weight parameter affects IOPS.

run      rw        bs   numjobs iothreads iops   diff
v9.2.0   randread  8k   1       1         174944 3.6%
v10.0.0  randread  8k   1       1         174285 3.2%
baseline randread  8k   1       1         168908 0.0%
w2       randread  8k   1       1         163718 -3.1%
w3       randread  8k   1       1         165805 -1.8%
w4       randread  8k   1       1         167388 -0.9%

This time I only ran randread bs=8k iodepth=8 numjobs=1 with a single IOThread.

Observations:

- There might be an IOPS regression between v10.0.0 and the baseline (9ad7f544c696) that your patches apply on top of. This is different from the CPU utilization regression that you found in v9.2.0 -> v10.0.0. I will bisect it.

- poll-weight=3 and 4 improve IOPS to a level that is acceptable. CPU utilization looks like this:

run %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle

baseline   49.37      0.00     31.10      0.00      0.00     11.61      0.04      0.00      0.00      7.89
w2         46.24      0.00     32.61      0.00      0.00     11.84      0.10      0.00      0.00      9.21
w3         48.04      0.00     32.17      0.00      0.00     11.98      0.08      0.00      0.00      7.73
w4         48.56      0.00     31.23      0.00      0.00     11.48      0.03      0.00      0.00      8.69
Quoted 5 lines

poll-weight=2 is the winner at CPU utilization. I'm not sure if poll-weight=3 will produce an acceptable CPU utilization improvement for you. Do you have data or want to re-run to measure poll-weight=3?

Stefan

Thank you very much for sharing the detailed measurement results. I truly appreciate the effort.

Regarding w=3, I will discuss with our performance team to see if the CPU consumption levels are acceptable within our internal test environment. I will get back to you with more definitive data as soon as possible.

Thanks again for your thorough analysis.

Regards,
Jaehoon.

#25 lore raw
Quoted 19 lines · JAEHOON KIM
Quoted 18 lines · Stefan Hajnoczi

Hi Jaehoon,
Following the call earlier this week I ran a single fio job to get a clearer picture of:
1. The QEMU 10.0.0 regression that prompted you to optimize AioContext polling.
2. How the poll-weight parameter affects IOPS.

run      rw        bs   numjobs iothreads iops   diff
v9.2.0   randread  8k   1       1         174944 3.6%
v10.0.0  randread  8k   1       1         174285 3.2%
baseline randread  8k   1       1         168908 0.0%
w2       randread  8k   1       1         163718 -3.1%
w3       randread  8k   1       1         165805 -1.8%
w4       randread  8k   1       1         167388 -0.9%

This time I only ran randread bs=8k iodepth=8 numjobs=1 with a single IOThread.

Quoted 6 lines
Quoted 6 lines

Observations:

- There might be an IOPS regression between v10.0.0 and the baseline (9ad7f544c696) that your patches apply on top of. This is different from the CPU utilization regression that you found in v9.2.0 -> v10.0.0. I will bisect it.

I reran v10.0.0, v10.1.0, and the baseline to check for a regression. The fio results came back not showing a regression. It's weird that the results I previously posted showed a change, but there doesn't seem to be a reproducible issue after all.

Stefan

#26 lore raw
In reply to JAEHOON KIM
Quoted 49 lines · JAEHOON KIM
Quoted 34 lines · Stefan Hajnoczi

Hi Jaehoon,
Following the call earlier this week I ran a single fio job to get a clearer picture of:
1. The QEMU 10.0.0 regression that prompted you to optimize AioContext polling.
2. How the poll-weight parameter affects IOPS.

run      rw        bs   numjobs iothreads iops   diff v9.2.0   randread  8k   1       1         174944 3.6% v10.0.0  randread  8k   1       1         174285 3.2% baseline randread  8k   1       1         168908 0.0% w2       randread  8k   1       1         163718 -3.1% w3       randread  8k   1       1         165805 -1.8% w4       randread  8k   1       1         167388 -0.9%

This time I only ran randread bs=8k iodepth=8 numjobs=1 with a single IOThread.

Observations:

- There might be an IOPS regression between v10.0.0 and the baseline (9ad7f544c696) that your patches apply on top of. This is different from the CPU utilization regression that you found in v9.2.0 -> v10.0.0. I will bisect it.

- poll-weight=3 and 4 improve IOPS to a level that is acceptable. CPU utilization looks like this:

run         %usr     %nice      %sys   %iowait    %steal %irq     %soft    %guest    %gnice     %idle baseline   49.37      0.00     31.10      0.00      0.00 11.61      0.04      0.00      0.00      7.89 w2         46.24      0.00     32.61      0.00      0.00 11.84      0.10      0.00      0.00      9.21 w3         48.04      0.00     32.17      0.00      0.00 11.98      0.08      0.00      0.00      7.73 w4         48.56      0.00     31.23      0.00      0.00 11.48      0.03      0.00      0.00      8.69

Quoted 5 lines

poll-weight=2 is the winner at CPU utilization. I'm not sure if poll-weight=3 will produce an acceptable CPU utilization improvement for you. Do you have data or want to re-run to measure poll-weight=3?

Stefan

Thank you very much for sharing the detailed measurement results. I truly appreciate the effort.

Regarding w=3, I will discuss with our performance team to see if the CPU consumption levels are acceptable within our internal test environment. I will get back to you with more definitive data as soon as possible.

Quoted 5 lines

Thanks again for your thorough analysis.

Regards,
Jaehoon.

Hello Stefan,

Thank you for your patience. I would like to share the observed changes in throughput and CPU consumption in our performance test environment as follows.

We observed that when using W=3, performance returns to a level comparable to QEMU v9.1, while W=2 results in slightly lower CPU consumption.

Our initial preference is to use W=2 by default.
However, we fully understand your concerns regarding the potential performance drop. Should the patch be accepted, we would provide guidance on using the weighted value as a configurable option.

For reference, we consider values between -2 and 2 as noise in our analysis. I look forward to your feedback.

The table below shows a comparison between:
*  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1, *Guest:* RHEL 9.6 GA vs. *  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:* RHEL 9.6 GA for FIO FCP and FICON with 1 iothread and 8 iothreads.
    The values shown are the averages for numjobs 1, 4, and 8.

FIO FCP -1 iothread

+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -3.00    |   -2.33    |    0.00    |  -0.33    |   -3.00    |   -3.33    |    1.33    |   -0.33    |
| CPU consumption avg |   -5.67    |   -4.33    |   -6.33    |  -5.33    |   -7.33    |   -5.33    |  -10.33    |   -8.67    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FCP -8 iothread

+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -5.00    |   -4.00    |   -3.67    |  -3.33    |   -4.67    |   -4.00    |   -0.33    |   -0.67    |
| CPU consumption avg |  -13.00    |  -10.67    |  -16.00    | -14.33    |  -14.00    |   -9.33    |  -13.67    |  -11.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FICON -1 iothread

+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -0.67    |   -0.67    |   -6.33    |  -6.67    |   -0.67    |    0.00    |    1.33    |    1.33    |
| CPU consumption avg |   -7.67    |   -7.33    |  -13.67    | -13.00    |   -9.00    |   -8.33    |   -5.00    |   -4.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FICON -8 iothread

+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -3.00    |   -2.67    |   -7.33    |  -7.00    |   -0.67    |   -1.00    |    0.67    |    0.67    |
| CPU consumption avg |  -16.33    |  -14.33    |  -25.33    | -27.00    |   -8.67    |   -7.00    |   -6.67    |   -5.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

The table below shows a comparison between:
* - Host:* RHEL 10.0 GA + qemu-9.1.0-15.el10, *Guest:* RHEL 9.6 GA vs. * - Host:* RHEL 10.1 GA + qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:* RHEL 9.6 GA.

FIO FCP -1 iothread

+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -0.67    |    0.00    |   -0.33    |  -1.00    |   -1.00    |   -0.67    |    3.33    |    2.00    |
| CPU consumption avg |    0.67    |    2.00    |    1.67    | 3.00    |   -3.33    |   -0.33    |   -2.33    |    0.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FCP -8 iothread

+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -2.00    |   -1.33    |   -1.00    | 0.00    |   -0.33    |    1.00    |    1.00    |    0.67    |
| CPU consumption avg |   -3.00    |   -1.00    |   -2.00    | 0.00    |  -10.33    |   -5.67    |   -6.67    |   -3.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FICON -1 iothread

+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -1.67    |   -1.67    |   -0.33    | 0.00    |   -1.33    |   -1.00    |   -1.67    |   -2.33    |
| CPU consumption avg |   -0.33    |    1.00    |    1.00    | 2.00    |   -2.33    |   -1.33    |   -1.33    |   -0.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FICON -8 iothread

+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -1.00    |   -0.67    |   -0.33    | 0.33    |    0.67    |    0.33    |    0.33    |    0.33    |
| CPU consumption avg |   -1.33    |    1.00    |    4.33    | 2.33    |   -2.00    |    0.00    |   -3.00    |   -1.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

Regards,
Jaehoon.

#27 lore raw
Quoted 175 lines · JAEHOON KIM
Quoted 49 lines · JAEHOON KIM
Quoted 34 lines · Stefan Hajnoczi

Hi Jaehoon,
Following the call earlier this week I ran a single fio job to get a clearer picture of:
1. The QEMU 10.0.0 regression that prompted you to optimize AioContext polling.
2. How the poll-weight parameter affects IOPS.

run      rw        bs   numjobs iothreads iops   diff v9.2.0   randread  8k   1       1         174944 3.6% v10.0.0  randread  8k   1       1         174285 3.2% baseline randread  8k   1       1         168908 0.0% w2       randread  8k   1       1         163718 -3.1% w3       randread  8k   1       1         165805 -1.8% w4       randread  8k   1       1         167388 -0.9%

This time I only ran randread bs=8k iodepth=8 numjobs=1 with a single IOThread.

Observations:

- There might be an IOPS regression between v10.0.0 and the baseline (9ad7f544c696) that your patches apply on top of. This is different from the CPU utilization regression that you found in v9.2.0 -> v10.0.0. I will bisect it.

- poll-weight=3 and 4 improve IOPS to a level that is acceptable. CPU utilization looks like this:

run         %usr     %nice      %sys   %iowait    %steal %irq     %soft    %guest    %gnice     %idle baseline   49.37      0.00     31.10      0.00      0.00 11.61      0.04      0.00      0.00      7.89 w2         46.24      0.00     32.61      0.00      0.00 11.84      0.10      0.00      0.00      9.21 w3         48.04      0.00     32.17      0.00      0.00 11.98      0.08      0.00      0.00      7.73 w4         48.56      0.00     31.23      0.00      0.00 11.48      0.03      0.00      0.00      8.69

Quoted 5 lines

poll-weight=2 is the winner at CPU utilization. I'm not sure if poll-weight=3 will produce an acceptable CPU utilization improvement for you. Do you have data or want to re-run to measure poll-weight=3?

Stefan

Thank you very much for sharing the detailed measurement results. I truly appreciate the effort.

Regarding w=3, I will discuss with our performance team to see if the CPU consumption levels are acceptable within our internal test environment. I will get back to you with more definitive data as soon as possible.

Quoted 5 lines

Thanks again for your thorough analysis.

Regards,
Jaehoon.

Hello Stefan,

Thank you for your patience. I would like to share the observed changes in throughput and CPU consumption in our performance test environment as follows.

We observed that when using W=3, performance returns to a level comparable to QEMU v9.1, while W=2 results in slightly lower CPU consumption.

Our initial preference is to use W=2 by default.
However, we fully understand your concerns regarding the potential performance drop. Should the patch be accepted, we would provide guidance on using the weighted value as a configurable option.

For reference, we consider values between -2 and 2 as noise in our analysis. I look forward to your feedback.

The table below shows a comparison between:
*  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1, *Guest:* RHEL 9.6 GA vs. *  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:* RHEL 9.6 GA for FIO FCP and FICON with 1 iothread and 8 iothreads.
    The values shown are the averages for numjobs 1, 4, and 8.

FIO FCP -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -3.00    |   -2.33    |    0.00    |  -0.33    |   -3.00    |   -3.33    |    1.33    |   -0.33    |
| CPU consumption avg |   -5.67    |   -4.33    |   -6.33    |  -5.33    |   -7.33    |   -5.33    |  -10.33    |   -8.67    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FCP -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -5.00    |   -4.00    |   -3.67    |  -3.33    |   -4.67    |   -4.00    |   -0.33    |   -0.67    |
| CPU consumption avg |  -13.00    |  -10.67    |  -16.00    | -14.33    |  -14.00    |   -9.33    |  -13.67    |  -11.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FICON -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -0.67    |   -0.67    |   -6.33    |  -6.67    |   -0.67    |    0.00    |    1.33    |    1.33    |
| CPU consumption avg |   -7.67    |   -7.33    |  -13.67    | -13.00    |   -9.00    |   -8.33    |   -5.00    |   -4.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FICON -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -3.00    |   -2.67    |   -7.33    |  -7.00    |   -0.67    |   -1.00    |    0.67    |    0.67    |
| CPU consumption avg |  -16.33    |  -14.33    |  -25.33    | -27.00    |   -8.67    |   -7.00    |   -6.67    |   -5.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

The table below shows a comparison between:
* - Host:* RHEL 10.0 GA + qemu-9.1.0-15.el10, *Guest:* RHEL 9.6 GA vs. * - Host:* RHEL 10.1 GA + qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:* RHEL 9.6 GA.

FIO FCP -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -0.67    |    0.00    |   -0.33    |  -1.00    |   -1.00    |   -0.67    |    3.33    |    2.00    |
| CPU consumption avg |    0.67    |    2.00    |    1.67    | 3.00    |   -3.33    |   -0.33    |   -2.33    |    0.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FCP -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -2.00    |   -1.33    |   -1.00    | 0.00    |   -0.33    |    1.00    |    1.00    |    0.67    |
| CPU consumption avg |   -3.00    |   -1.00    |   -2.00    | 0.00    |  -10.33    |   -5.67    |   -6.67    |   -3.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FICON -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -1.67    |   -1.67    |   -0.33    | 0.00    |   -1.33    |   -1.00    |   -1.67    |   -2.33    |
| CPU consumption avg |   -0.33    |    1.00    |    1.00    | 2.00    |   -2.33    |   -1.33    |   -1.33    |   -0.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FICON -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
Quoted 10 lines
| Throughput avg      |   -1.00    |   -0.67    |   -0.33    | 0.33    |    0.67    |    0.33    |    0.33    |    0.33    |
| CPU consumption avg |   -1.33    |    1.00    |    4.33    | 2.33    |   -2.00    |    0.00    |   -3.00    |   -1.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

Regards,
Jaehoon.

Hello Stefan,

I’m following up to see if you’ve had a chance to review the performance results I shared. As I mentioned, we are completely fine with using w=3 as the default value. It effectively restores performance to QEMU v9.1 levels, and we will provide guidance for users who need further CPU savings to use w=2.

I’ve updated the patch to set the default value to 3 and include the feedback so far. Please let me know if you have any comments or if I should post the new patch.

Regards,
Jaehoon

#28 lore raw
Quoted 196 lines · JAEHOON KIM
Quoted 178 lines · JAEHOON KIM
Quoted 49 lines · JAEHOON KIM
Quoted 34 lines · Stefan Hajnoczi

Hi Jaehoon,
Following the call earlier this week I ran a single fio job to get a clearer picture of:
1. The QEMU 10.0.0 regression that prompted you to optimize AioContext polling.
2. How the poll-weight parameter affects IOPS.

run      rw        bs   numjobs iothreads iops   diff v9.2.0   randread  8k   1       1         174944 3.6% v10.0.0  randread  8k   1       1         174285 3.2% baseline randread  8k   1       1         168908 0.0% w2       randread  8k   1       1         163718 -3.1% w3       randread  8k   1       1         165805 -1.8% w4       randread  8k   1       1         167388 -0.9%

This time I only ran randread bs=8k iodepth=8 numjobs=1 with a single IOThread.

Observations:

- There might be an IOPS regression between v10.0.0 and the baseline (9ad7f544c696) that your patches apply on top of. This is different from the CPU utilization regression that you found in v9.2.0 -> v10.0.0. I will bisect it.

- poll-weight=3 and 4 improve IOPS to a level that is acceptable. CPU utilization looks like this:

run         %usr     %nice      %sys   %iowait    %steal %irq     %soft    %guest    %gnice     %idle baseline   49.37      0.00     31.10      0.00      0.00 11.61      0.04      0.00      0.00      7.89 w2         46.24      0.00     32.61      0.00      0.00 11.84      0.10      0.00      0.00      9.21 w3         48.04      0.00     32.17      0.00      0.00 11.98      0.08      0.00      0.00      7.73 w4         48.56      0.00     31.23      0.00      0.00 11.48      0.03      0.00      0.00      8.69

Quoted 5 lines

poll-weight=2 is the winner at CPU utilization. I'm not sure if poll-weight=3 will produce an acceptable CPU utilization improvement for you. Do you have data or want to re-run to measure poll-weight=3?

Stefan

Thank you very much for sharing the detailed measurement results. I truly appreciate the effort.

Regarding w=3, I will discuss with our performance team to see if the CPU consumption levels are acceptable within our internal test environment. I will get back to you with more definitive data as soon as possible.

Quoted 5 lines

Thanks again for your thorough analysis.

Regards,
Jaehoon.

Hello Stefan,

Thank you for your patience. I would like to share the observed changes in throughput and CPU consumption in our performance test environment as follows.

We observed that when using W=3, performance returns to a level comparable to QEMU v9.1, while W=2 results in slightly lower CPU consumption.

Our initial preference is to use W=2 by default.
However, we fully understand your concerns regarding the potential performance drop. Should the patch be accepted, we would provide guidance on using the weighted value as a configurable option.

For reference, we consider values between -2 and 2 as noise in our analysis. I look forward to your feedback.

The table below shows a comparison between:
*  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1, *Guest:* RHEL 9.6 GA vs. *  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:* RHEL 9.6 GA for FIO FCP and FICON with 1 iothread and 8 iothreads.
    The values shown are the averages for numjobs 1, 4, and 8.

FIO FCP -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -3.00    |   -2.33    |    0.00    |  -0.33    |   -3.00    |   -3.33    |    1.33    |   -0.33    |
| CPU consumption avg |   -5.67    |   -4.33    |   -6.33    |  -5.33    |   -7.33    |   -5.33    |  -10.33    |   -8.67    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FCP -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -5.00    |   -4.00    |   -3.67    |  -3.33    |   -4.67    |   -4.00    |   -0.33    |   -0.67    |
| CPU consumption avg |  -13.00    |  -10.67    |  -16.00    | 
-14.33    |  -14.00    |   -9.33    |  -13.67    |  -11.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FICON -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -0.67    |   -0.67    |   -6.33    |  -6.67    |   -0.67    |    0.00    |    1.33    |    1.33    |
| CPU consumption avg |   -7.67    |   -7.33    |  -13.67    | 
-13.00    |   -9.00    |   -8.33    |   -5.00    |   -4.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FICON -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -3.00    |   -2.67    |   -7.33    |  -7.00    |   -0.67    |   -1.00    |    0.67    |    0.67    |
| CPU consumption avg |  -16.33    |  -14.33    |  -25.33    | 
-27.00    |   -8.67    |   -7.00    |   -6.67    |   -5.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

The table below shows a comparison between:
* - Host:* RHEL 10.0 GA + qemu-9.1.0-15.el10, *Guest:* RHEL 9.6 GA vs. * - Host:* RHEL 10.1 GA + qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:* RHEL 9.6 GA.

FIO FCP -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -0.67    |    0.00    |   -0.33    |  -1.00    |   -1.00    |   -0.67    |    3.33    |    2.00    |
| CPU consumption avg |    0.67    |    2.00    |    1.67    | 3.00    |   -3.33    |   -0.33    |   -2.33    |    0.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FCP -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -2.00    |   -1.33    |   -1.00    | 0.00    |   -0.33    |    1.00    |    1.00    |    0.67    |
| CPU consumption avg |   -3.00    |   -1.00    |   -2.00    | 0.00    |  -10.33    |   -5.67    |   -6.67    |   -3.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FICON -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
| Throughput avg      |   -1.67    |   -1.67    |   -0.33    | 0.00    |   -1.33    |   -1.00    |   -1.67    |   -2.33    |
| CPU consumption avg |   -0.33    |    1.00    |    1.00    | 2.00    |   -2.33    |   -1.33    |   -1.33    |   -0.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

FIO FICON -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq. Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3 (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 
Quoted 10 lines
| Throughput avg      |   -1.00    |   -0.67    |   -0.33    | 0.33    |    0.67    |    0.33    |    0.33    |    0.33    |
| CPU consumption avg |   -1.33    |    1.00    |    4.33    | 2.33    |   -2.00    |    0.00    |   -3.00    |   -1.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+ 

Regards,
Jaehoon.

Hello Stefan,

I’m following up to see if you’ve had a chance to review the performance results I shared. As I mentioned, we are completely fine with using w=3 as the default value. It effectively restores performance to QEMU v9.1 levels, and we will provide guidance for users who need further CPU savings to use w=2.

Quoted 7 lines

I’ve updated the patch to set the default value to 3 and include the feedback so far. Please let me know if you have any comments or if I should post the new patch.

Regards,
Jaehoon

Hello,

I'd like to follow up on this thread. I have posted a v2 incorporating the feedback so far. I'd really appreciate your thoughts when you have a chance.

Thanks again for your time.

Regards,
Jaehoon.

#29 lore raw
Quoted 251 lines · JAEHOON KIM
Quoted 241 lines · JAEHOON KIM
Quoted 226 lines · JAEHOON KIM
Quoted 55 lines · JAEHOON KIM
Quoted 39 lines · Stefan Hajnoczi

Hi Jaehoon,
Following the call earlier this week I ran a single fio job to get a clearer picture of:
1. The QEMU 10.0.0 regression that prompted you to optimize AioContext polling.
2. How the poll-weight parameter affects IOPS.

run      rw        bs   numjobs iothreads iops   diff v9.2.0   randread  8k   1       1         174944 3.6% v10.0.0  randread  8k   1       1         174285 3.2% baseline randread  8k   1       1         168908 0.0% w2       randread  8k   1       1         163718 -3.1% w3       randread  8k   1       1         165805 -1.8% w4       randread  8k   1       1         167388 -0.9%

This time I only ran randread bs=8k iodepth=8 numjobs=1 with a single IOThread.

Observations:

- There might be an IOPS regression between v10.0.0 and the baseline (9ad7f544c696) that your patches apply on top of. This is different from the CPU utilization regression that you found in v9.2.0 -> v10.0.0. I will bisect it.

- poll-weight=3 and 4 improve IOPS to a level that is acceptable. CPU utilization looks like this:

run         %usr     %nice      %sys   %iowait    %steal %irq     %soft    %guest    %gnice     %idle baseline   49.37      0.00     31.10      0.00      0.00 11.61      0.04      0.00      0.00      7.89 w2         46.24      0.00     32.61      0.00      0.00 11.84      0.10      0.00      0.00      9.21 w3         48.04      0.00     32.17      0.00      0.00 11.98      0.08      0.00      0.00      7.73 w4         48.56      0.00     31.23      0.00      0.00 11.48      0.03      0.00      0.00      8.69

Quoted 6 lines

poll-weight=2 is the winner at CPU utilization. I'm not sure if poll-weight=3 will produce an acceptable CPU utilization improvement for
you. Do you have data or want to re-run to measure poll-weight=3?

Stefan

Thank you very much for sharing the detailed measurement results. I truly appreciate the effort.

Regarding w=3, I will discuss with our performance team to see if the CPU consumption levels are acceptable within our internal test environment. I will get back to you with more definitive data as soon as possible.

Quoted 5 lines

Thanks again for your thorough analysis.

Regards,
Jaehoon.

Hello Stefan,

Thank you for your patience. I would like to share the observed changes in throughput
and CPU consumption in our performance test environment as follows.

We observed that when using W=3, performance returns to a level comparable to
QEMU v9.1, while W=2 results in slightly lower CPU consumption.

Our initial preference is to use W=2 by default.
However, we fully understand your concerns regarding the potential performance drop.
Should the patch be accepted, we would provide guidance on using the weighted value as
a configurable option.

For reference, we consider values between -2 and 2 as noise in our analysis.
I look forward to your feedback.

The table below shows a comparison between:
*  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1, *Guest:* RHEL 9.6 GA vs. *  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:* RHEL 9.6 GA
    for FIO FCP and FICON with 1 iothread and 8 iothreads. The values shown are the averages for numjobs 1, 4, and 8.

FIO FCP -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
(%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -3.00    |   -2.33    |    0.00    |
 -0.33    |   -3.00    |   -3.33    |    1.33    |   -0.33    |
| CPU consumption avg |   -5.67    |   -4.33    |   -6.33    |
 -5.33    |   -7.33    |   -5.33    |  -10.33    |   -8.67    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FCP -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
(%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -5.00    |   -4.00    |   -3.67    |
 -3.33    |   -4.67    |   -4.00    |   -0.33    |   -0.67    |
| CPU consumption avg |  -13.00    |  -10.67    |  -16.00    |
-14.33    |  -14.00    |   -9.33    |  -13.67    |  -11.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FICON -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
(%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -0.67    |   -0.67    |   -6.33    |
 -6.67    |   -0.67    |    0.00    |    1.33    |    1.33    |
| CPU consumption avg |   -7.67    |   -7.33    |  -13.67    |
-13.00    |   -9.00    |   -8.33    |   -5.00    |   -4.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FICON -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
(%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -3.00    |   -2.67    |   -7.33    |
 -7.00    |   -0.67    |   -1.00    |    0.67    |    0.67    |
| CPU consumption avg |  -16.33    |  -14.33    |  -25.33    |
-27.00    |   -8.67    |   -7.00    |   -6.67    |   -5.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

The table below shows a comparison between:
* - Host:* RHEL 10.0 GA + qemu-9.1.0-15.el10, *Guest:* RHEL 9.6 GA vs. * - Host:* RHEL 10.1 GA + qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:* RHEL 9.6 GA.

FIO FCP -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
(%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -0.67    |    0.00    |   -0.33    |
 -1.00    |   -1.00    |   -0.67    |    3.33    |    2.00    |
| CPU consumption avg |    0.67    |    2.00    |    1.67    | 3.00 
  |   -3.33    |   -0.33    |   -2.33    |    0.00    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FCP -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
(%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -2.00    |   -1.33    |   -1.00    | 0.00 
  |   -0.33    |    1.00    |    1.00    |    0.67    |
| CPU consumption avg |   -3.00    |   -1.00    |   -2.00    | 0.00 
  |  -10.33    |   -5.67    |   -6.67    |   -3.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FICON -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
(%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -1.67    |   -1.67    |   -0.33    | 0.00 
  |   -1.33    |   -1.00    |   -1.67    |   -2.33    |
| CPU consumption avg |   -0.33    |    1.00    |    1.00    | 2.00 
  |   -2.33    |   -1.33    |   -1.33    |   -0.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

FIO FICON -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+

|                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
|                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
(%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg      |   -1.00    |   -0.67    |   -0.33    | 0.33 
  |    0.67    |    0.33    |    0.33    |    0.33    |
| CPU consumption avg |   -1.33    |    1.00    |    4.33    | 2.33 
  |   -2.00    |    0.00    |   -3.00    |   -1.33    |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
Quoted 4 lines

Regards,
Jaehoon.

Hello Stefan,

I’m following up to see if you’ve had a chance to review the performance results I shared.
As I mentioned, we are completely fine with using w=3 as the default value.
It effectively restores performance to QEMU v9.1 levels, and we will provide guidance
for users who need further CPU savings to use w=2.

Quoted 9 lines

I’ve updated the patch to set the default value to 3 and include the feedback so far.
Please let me know if you have any comments or if I should post the new patch.

Regards,
Jaehoon

Quoted 7 lines

Hello,

I'd like to follow up on this thread. I have posted a v2 incorporating the feedback so far.
I'd really appreciate your thoughts when you have a chance.

Thanks again for your time.

Hi Jaehoon,
Sorry that I haven't replied yet. I will review your emails and the v2 patches tomorrow.

Stefan