Skip to contentSkip to conversation

crypto: fix bogus error benchmarking pbkdf on fast machines

PATCH Superseded Daniel P. Berrangé 4 participants 5 messages Jan 8, 2025 – Jan 22, 2025 Crypto API

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 crypto: fix bogus error benchmarking pbkdf on fast machines

We're seeing periodic reports of errors like:

$ qemu-img create -f luks --object secret,data=123456,id=sec0 \
                  -o key-secret=sec0 luks-info.img 1M
  Formatting 'luks-info.img', fmt=luks size=1048576 key-secret=sec0
  qemu-img: luks-info.img: Unable to get accurate CPU usage

This error message comes from a recent attempt to workaround a kernel bug with measuring rusage in long running processes:

commit c72cab5ad9f849bbcfcf4be7952b8b8946cc626e
Author: Tiago Pasqualini <[email protected]>
Date: Wed Sep 4 20:52:30 2024 -0300

crypto: run qcrypto_pbkdf2_count_iters in a new thread

Unfortunately this has a subtle bug on machines which are very fast.

On the first time around the loop, the 'iterations' value is quite small (1 << 15), and so will run quite fast. Testing has shown that some machines can complete this benchmarking task in as little as 7 milliseconds.

Unfortunately the 'getrusage' data is not updated at the time of the 'getrusage' call, it is done asynchronously by the schedular. The 7 millisecond completion time for the benchmark is short enough that 'getrusage' sometimes reports 0 accumulated execution time.

As a result the 'delay_ms == 0' sanity check in the above commit is triggering non-deterministically on such machines.

The benchmarking loop intended to run multiple times, increasing the 'iterations' value until the benchmark ran for > 500 ms, but the sanity check doesn't allow this to happen.

To fix it, we keep a loop counter and only run the sanity check after we've been around the loop more than 5 times. At that point the 'iterations' value is high enough that even with infrequent updates of 'getrusage' accounting data on fast machines, we should see a non-zero value.

---
 crypto/pbkdf.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
crypto/pbkdf.c +13 -2
index 0dd7c3aeaa..b285958319 100644
@@ -107,7 +107,7 @@ static void *threaded_qcrypto_pbkdf2_count_iters(void *data)
     size_t nsalt = iters_data->nsalt;
     size_t nout = iters_data->nout;
     Error **errp = iters_data->errp;
-
+    size_t scaled = 0;
     uint64_t ret = -1;
     g_autofree uint8_t *out = g_new(uint8_t, nout);
     uint64_t iterations = (1 << 15);
@@ -131,7 +131,17 @@ static void *threaded_qcrypto_pbkdf2_count_iters(void *data)
 
         delta_ms = end_ms - start_ms;
 
-        if (delta_ms == 0) { /* sanity check */
+        /*
+         * For very small 'iterations' values, CPU (or crypto
+         * accelerator) might be fast enough that the schedular
+         * hasn't incremented getrusage() data, or incremented
+         * it by a very small amount, resulting in delta_ms == 0.
+         * Once we've scaled 'iterations' x10, 5 times, we really
+         * should be seeing delta_ms != 0, so sanity check at
+         * that point.
+         */
+        if (scaled > 5 &&
+            delta_ms == 0) { /* sanity check */
             error_setg(errp, "Unable to get accurate CPU usage");
             goto cleanup;
         } else if (delta_ms > 500) {
@@ -141,6 +151,7 @@ static void *threaded_qcrypto_pbkdf2_count_iters(void *data)
         } else {
             iterations = (iterations * 1000 / delta_ms);
         }
+        scaled++;
     }
 
     iterations = iterations * 1000 / delta_ms;
Signature · 1 line
2.47.1
#1 lore raw
Quoted 18 lines · Daniel P. Berrangé

We're seeing periodic reports of errors like:

$ qemu-img create -f luks --object secret,data=123456,id=sec0 \
                  -o key-secret=sec0 luks-info.img 1M
  Formatting 'luks-info.img', fmt=luks size=1048576 key-secret=sec0
  qemu-img: luks-info.img: Unable to get accurate CPU usage

This error message comes from a recent attempt to workaround a kernel bug with measuring rusage in long running processes:

commit c72cab5ad9f849bbcfcf4be7952b8b8946cc626e
Author: Tiago Pasqualini <[email protected]>
Date: Wed Sep 4 20:52:30 2024 -0300

crypto: run qcrypto_pbkdf2_count_iters in a new thread

Unfortunately this has a subtle bug on machines which are very fast.

Quoted 7 lines

On the first time around the loop, the 'iterations' value is quite small (1 << 15), and so will run quite fast. Testing has shown that some machines can complete this benchmarking task in as little as 7 milliseconds.

Unfortunately the 'getrusage' data is not updated at the time of the 'getrusage' call, it is done asynchronously by the schedular.

scheduler (https://en.wiktionary.org/wiki/scheduler)

Quoted 11 lines

The 7 millisecond completion time for the benchmark is short enough that 'getrusage' sometimes reports 0 accumulated execution time.

As a result the 'delay_ms == 0' sanity check in the above commit is triggering non-deterministically on such machines.

The benchmarking loop intended to run multiple times, increasing the 'iterations' value until the benchmark ran for > 500 ms, but the sanity check doesn't allow this to happen.

Quoted 8 lines

To fix it, we keep a loop counter and only run the sanity check after we've been around the loop more than 5 times. At that point the 'iterations' value is high enough that even with infrequent updates of 'getrusage' accounting data on fast machines, we should see a non-zero value.

I think it was originally reported by a Debian automated tool in
https://lore.kernel.org/qemu-devel/[email protected]/

Quoted 19 lines
---
 crypto/pbkdf.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
crypto/pbkdf.c +1 -1
index 0dd7c3aeaa..b285958319 100644
@@ -107,7 +107,7 @@ static void *threaded_qcrypto_pbkdf2_count_iters(void *data)
     size_t nsalt = iters_data->nsalt;
     size_t nout = iters_data->nout;
     Error **errp = iters_data->errp;
-
+    size_t scaled = 0;
     uint64_t ret = -1;
     g_autofree uint8_t *out = g_new(uint8_t, nout);
     uint64_t iterations = (1 << 15);

crypto/pbkdf.c static void *threaded_qcrypto_pbkdf2_count_iters(void *data) · line 136

Quoted 7 lines
crypto/pbkdf.c +3 -1
@@ -131,7 +131,17 @@ static void *threaded_qcrypto_pbkdf2_count_iters(void *data)
 
         delta_ms = end_ms - start_ms;
 
-        if (delta_ms == 0) { /* sanity check */
+        /*
+         * For very small 'iterations' values, CPU (or crypto
+         * accelerator) might be fast enough that the schedular

scheduler as above

Quoted 21 lines
crypto/pbkdf.c +9 -0
+         * hasn't incremented getrusage() data, or incremented
+         * it by a very small amount, resulting in delta_ms == 0.
+         * Once we've scaled 'iterations' x10, 5 times, we really
+         * should be seeing delta_ms != 0, so sanity check at
+         * that point.
+         */
+        if (scaled > 5 &&
+            delta_ms == 0) { /* sanity check */
             error_setg(errp, "Unable to get accurate CPU usage");
             goto cleanup;
         } else if (delta_ms > 500) {
@@ -141,6 +151,7 @@ static void *threaded_qcrypto_pbkdf2_count_iters(void *data)
         } else {
             iterations = (iterations * 1000 / delta_ms);
         }
+        scaled++;
     }
 
     iterations = iterations * 1000 / delta_ms;
Signature · 1 line
2.47.1

I don't have a good way to test this unfortunately as we only saw it once, on s390x, in a build system.

You might want to add links to the bug reports and earlier threads about this:

https://lore.kernel.org/qemu-devel/[email protected]/
https://lore.kernel.org/qemu-devel/[email protected]/
https://bugzilla.redhat.com/show_bug.cgi?id=2336437

Rich.

Signature · 4 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
Tested-by on v1 1/1
#2 lore raw
In reply to Daniel P. Berrangé
Quoted 36 lines · Daniel P. Berrangé

We're seeing periodic reports of errors like:

$ qemu-img create -f luks --object secret,data=123456,id=sec0 \
                   -o key-secret=sec0 luks-info.img 1M
   Formatting 'luks-info.img', fmt=luks size=1048576 key-secret=sec0
   qemu-img: luks-info.img: Unable to get accurate CPU usage

This error message comes from a recent attempt to workaround a kernel bug with measuring rusage in long running processes:

commit c72cab5ad9f849bbcfcf4be7952b8b8946cc626e
Author: Tiago Pasqualini <[email protected]>
Date: Wed Sep 4 20:52:30 2024 -0300

crypto: run qcrypto_pbkdf2_count_iters in a new thread

Unfortunately this has a subtle bug on machines which are very fast.

On the first time around the loop, the 'iterations' value is quite small (1 << 15), and so will run quite fast. Testing has shown that some machines can complete this benchmarking task in as little as 7 milliseconds.

Unfortunately the 'getrusage' data is not updated at the time of the 'getrusage' call, it is done asynchronously by the schedular. The 7 millisecond completion time for the benchmark is short enough that 'getrusage' sometimes reports 0 accumulated execution time.

As a result the 'delay_ms == 0' sanity check in the above commit is triggering non-deterministically on such machines.

The benchmarking loop intended to run multiple times, increasing the 'iterations' value until the benchmark ran for > 500 ms, but the sanity check doesn't allow this to happen.

Quoted 5 lines

To fix it, we keep a loop counter and only run the sanity check after we've been around the loop more than 5 times. At that point the 'iterations' value is high enough that even with infrequent updates of 'getrusage' accounting data on fast machines, we should see a non-zero value.

Thanks, this seems to fix the issue for me:

#3 lore raw
In reply to Daniel P. Berrangé

08.01.2025 21:43, Daniel P. Berrangé пишет:

Quoted 42 lines

We're seeing periodic reports of errors like:

$ qemu-img create -f luks --object secret,data=123456,id=sec0 \
                   -o key-secret=sec0 luks-info.img 1M
   Formatting 'luks-info.img', fmt=luks size=1048576 key-secret=sec0
   qemu-img: luks-info.img: Unable to get accurate CPU usage

This error message comes from a recent attempt to workaround a kernel bug with measuring rusage in long running processes:

commit c72cab5ad9f849bbcfcf4be7952b8b8946cc626e
Author: Tiago Pasqualini <[email protected]>
Date: Wed Sep 4 20:52:30 2024 -0300

crypto: run qcrypto_pbkdf2_count_iters in a new thread

Unfortunately this has a subtle bug on machines which are very fast.

On the first time around the loop, the 'iterations' value is quite small (1 << 15), and so will run quite fast. Testing has shown that some machines can complete this benchmarking task in as little as 7 milliseconds.

Unfortunately the 'getrusage' data is not updated at the time of the 'getrusage' call, it is done asynchronously by the schedular. The 7 millisecond completion time for the benchmark is short enough that 'getrusage' sometimes reports 0 accumulated execution time.

As a result the 'delay_ms == 0' sanity check in the above commit is triggering non-deterministically on such machines.

The benchmarking loop intended to run multiple times, increasing the 'iterations' value until the benchmark ran for > 500 ms, but the sanity check doesn't allow this to happen.

To fix it, we keep a loop counter and only run the sanity check after we've been around the loop more than 5 times. At that point the 'iterations' value is high enough that even with infrequent updates of 'getrusage' accounting data on fast machines, we should see a non-zero value.

Quoted 3 lines

Is this a qemu-stable material (9.2)?

Thanks,

/mjt

#4 lore raw
Quoted 43 lines · Michael Tokarev

08.01.2025 21:43, Daniel P. Berrangé пишет:

Quoted 42 lines

We're seeing periodic reports of errors like:

$ qemu-img create -f luks --object secret,data=123456,id=sec0 \
                   -o key-secret=sec0 luks-info.img 1M
   Formatting 'luks-info.img', fmt=luks size=1048576 key-secret=sec0
   qemu-img: luks-info.img: Unable to get accurate CPU usage

This error message comes from a recent attempt to workaround a kernel bug with measuring rusage in long running processes:

commit c72cab5ad9f849bbcfcf4be7952b8b8946cc626e
Author: Tiago Pasqualini <[email protected]>
Date: Wed Sep 4 20:52:30 2024 -0300

crypto: run qcrypto_pbkdf2_count_iters in a new thread

Unfortunately this has a subtle bug on machines which are very fast.

On the first time around the loop, the 'iterations' value is quite small (1 << 15), and so will run quite fast. Testing has shown that some machines can complete this benchmarking task in as little as 7 milliseconds.

Unfortunately the 'getrusage' data is not updated at the time of the 'getrusage' call, it is done asynchronously by the schedular. The 7 millisecond completion time for the benchmark is short enough that 'getrusage' sometimes reports 0 accumulated execution time.

As a result the 'delay_ms == 0' sanity check in the above commit is triggering non-deterministically on such machines.

The benchmarking loop intended to run multiple times, increasing the 'iterations' value until the benchmark ran for > 500 ms, but the sanity check doesn't allow this to happen.

To fix it, we keep a loop counter and only run the sanity check after we've been around the loop more than 5 times. At that point the 'iterations' value is high enough that even with infrequent updates of 'getrusage' accounting data on fast machines, we should see a non-zero value.

Quoted 5 lines
Quoted 3 lines

Is this a qemu-stable material (9.2)?

Yes, please include it.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|