Skip to contentSkip to conversation

hostmem: default the amount of prealloc-threads to smp-cpus

PATCH Applied [email protected] 5 participants 11 messages May 17, 2022 – May 19, 2022 Arm STM32

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 hostmem: default the amount of prealloc-threads to smp-cpus

From: Jaroslav Jindrak <[email protected]>

Prior to the introduction of the prealloc-threads property, the amount of threads used to preallocate memory was derived from the value of smp-cpus passed to qemu, the amount of physical cpus of the host and a hardcoded maximum value. When the prealloc-threads property was introduced, it included a default of 1 in backends/hostmem.c and a default of smp-cpus using the sugar API for the property itself. The latter default is not used when the property is not specified on qemu's command line, so guests that were not adjusted for this change suddenly started to use the default of 1 thread to preallocate memory, which resulted in observable slowdowns in guest boots for guests with large memory (e.g. when using libvirt <8.2.0 or managing guests manually).

This commit restores the original behavior for these cases while not impacting guests started with the prealloc-threads property in any way.

Fixes 220c1fd864e9d ("hostmem: introduce "prealloc-threads" property")
---
 backends/hostmem.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
backends/hostmem.c +1 -1
index a7bae3d713..624bb7ecd3 100644
@@ -274,7 +274,7 @@ static void host_memory_backend_init(Object *obj)
     backend->merge = machine_mem_merge(machine);
     backend->dump = machine_dump_guest_core(machine);
     backend->reserve = true;
-    backend->prealloc_threads = 1;
+    backend->prealloc_threads = machine->smp.cpus;
 }
 
 static void host_memory_backend_post_init(Object *obj)
Signature · 1 line
2.30.2
#1 lore raw
Quoted 13 lines · On Tue, 17 May 2022 14:38:58 +0200 [email protected] wrote:

From: Jaroslav Jindrak <[email protected]>

Prior to the introduction of the prealloc-threads property, the amount of threads used to preallocate memory was derived from the value of smp-cpus passed to qemu, the amount of physical cpus of the host and a hardcoded maximum value. When the prealloc-threads property was introduced, it included a default of 1 in backends/hostmem.c and a default of smp-cpus using the sugar API for the property itself. The latter default is not used when the property is not specified on qemu's command line, so guests that were not adjusted for this change suddenly started to use the default of 1 thread to preallocate memory, which resulted in observable slowdowns in guest boots for guests with large memory (e.g. when using libvirt <8.2.0 or managing guests manually).

current behavior in QEMU is intentionally conservative. threads number is subject to host configuration and limitations management layer puts on it and it's not QEMU job to conjure magic numbers that are host/workload depended.
If user needs more prealloc threads they need to specify it explicitly for each memory backend (i.e. convince management to do it or fix your scripts to so).

CCing Michal, as he recently looked into similar topic.

To behave it the old way you need to use legacy -mem-prealloc option.

Quoted 14 lines

This commit restores the original behavior for these cases while not impacting guests started with the prealloc-threads property in any way.

Fixes 220c1fd864e9d ("hostmem: introduce "prealloc-threads" property")
---
 backends/hostmem.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
backends/hostmem.c +0 -0
index a7bae3d713..624bb7ecd3 100644

backends/hostmem.c static void host_memory_backend_init(Object *obj) · line 277

Quoted 5 lines
backends/hostmem.c +1 -1
@@ -274,7 +274,7 @@ static void host_memory_backend_init(Object *obj)
     backend->merge = machine_mem_merge(machine);
     backend->dump = machine_dump_guest_core(machine);
     backend->reserve = true;
-    backend->prealloc_threads = 1;
+    backend->prealloc_threads = machine->smp.cpus;

pls, do not add more dependencies to random external objects to memory backends.

If you have to do that, use machine compat properties instead, but then the essence of the issue stays the same (user shall define optimal threads number and provide it to qemu explicitly)

Quoted 3 lines
backends/hostmem.c +0 -0
 }
 
 static void host_memory_backend_post_init(Object *obj)
#2 lore raw
Quoted 18 lines · Igor Mammedov
Quoted 13 lines · On Tue, 17 May 2022 14:38:58 +0200 [email protected] wrote:

From: Jaroslav Jindrak <[email protected]>

Prior to the introduction of the prealloc-threads property, the amount of threads used to preallocate memory was derived from the value of smp-cpus passed to qemu, the amount of physical cpus of the host and a hardcoded maximum value. When the prealloc-threads property was introduced, it included a default of 1 in backends/hostmem.c and a default of smp-cpus using the sugar API for the property itself. The latter default is not used when the property is not specified on qemu's command line, so guests that were not adjusted for this change suddenly started to use the default of 1 thread to preallocate memory, which resulted in observable slowdowns in guest boots for guests with large memory (e.g. when using libvirt <8.2.0 or managing guests manually).

Quoted 7 lines

current behavior in QEMU is intentionally conservative. threads number is subject to host configuration and limitations management layer puts on it and it's not QEMU job to conjure magic numbers that are host/workload depended.
If user needs more prealloc threads they need to specify it explicitly for each memory backend (i.e. convince management to do it or fix your scripts to so).

I understand that, but I'd say that a sudden change of the default to 1, which can
lead to guest boot slowdowns of about 2-8 times (and boots taking several minutes) because of a change that was not very well documented (to the point that it took libvirt about 2 years [0] to add support for this property) does qualify
as a regression. After all, this patch does not conjure any new numbers, but rather
restores the original behavior of qemu if the user does _not_ use the new prealloc-threads property.

[0] https://github.com/libvirt/libvirt/commit/ba7f98126fa84d354ce72929b77cc111a9a557a9

Quoted 34 lines

CCing Michal, as he recently looked into similar topic.

To behave it the old way you need to use legacy -mem-prealloc option.

Quoted 14 lines

This commit restores the original behavior for these cases while not impacting guests started with the prealloc-threads property in any way.

Fixes 220c1fd864e9d ("hostmem: introduce "prealloc-threads" property")
---
 backends/hostmem.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
backends/hostmem.c +0 -0
index a7bae3d713..624bb7ecd3 100644

backends/hostmem.c static void host_memory_backend_init(Object *obj) · line 277

Quoted 5 lines
backends/hostmem.c +1 -1
@@ -274,7 +274,7 @@ static void host_memory_backend_init(Object *obj)
     backend->merge = machine_mem_merge(machine);
     backend->dump = machine_dump_guest_core(machine);
     backend->reserve = true;
-    backend->prealloc_threads = 1;
+    backend->prealloc_threads = machine->smp.cpus;

pls, do not add more dependencies to random external objects to memory backends.

If you have to do that, use machine compat properties instead, but then the essence of the issue stays the same (user shall define optimal threads number and provide it to qemu explicitly)

Quoted 3 lines
backends/hostmem.c +0 -0
 }
 
 static void host_memory_backend_post_init(Object *obj)
#3 lore raw
In reply to Igor Mammedov
Quoted 17 lines · Igor Mammedov
Quoted 13 lines · On Tue, 17 May 2022 14:38:58 +0200 [email protected] wrote:

From: Jaroslav Jindrak <[email protected]>

Prior to the introduction of the prealloc-threads property, the amount of threads used to preallocate memory was derived from the value of smp-cpus passed to qemu, the amount of physical cpus of the host and a hardcoded maximum value. When the prealloc-threads property was introduced, it included a default of 1 in backends/hostmem.c and a default of smp-cpus using the sugar API for the property itself. The latter default is not used when the property is not specified on qemu's command line, so guests that were not adjusted for this change suddenly started to use the default of 1 thread to preallocate memory, which resulted in observable slowdowns in guest boots for guests with large memory (e.g. when using libvirt <8.2.0 or managing guests manually).

Quoted 4 lines

current behavior in QEMU is intentionally conservative. threads number is subject to host configuration and limitations management layer puts on it and it's not QEMU job to conjure magic numbers that are host/workload depended.

I think that's missing the point. QEMU *did* historically set the prealloc threads equal to num CPUs, so we have precedent here. The referenced commit lost that behaviour because it only wired up the defaults in one particular CLI scenario. That's a clear regression on QEMU's side.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw
Quoted 23 lines · On Tue, 17 May 2022 17:33:54 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 17 lines · Igor Mammedov
Quoted 13 lines · On Tue, 17 May 2022 14:38:58 +0200 [email protected] wrote:

From: Jaroslav Jindrak <[email protected]>

Prior to the introduction of the prealloc-threads property, the amount of threads used to preallocate memory was derived from the value of smp-cpus passed to qemu, the amount of physical cpus of the host and a hardcoded maximum value. When the prealloc-threads property was introduced, it included a default of 1 in backends/hostmem.c and a default of smp-cpus using the sugar API for the property itself. The latter default is not used when the property is not specified on qemu's command line, so guests that were not adjusted for this change suddenly started to use the default of 1 thread to preallocate memory, which resulted in observable slowdowns in guest boots for guests with large memory (e.g. when using libvirt <8.2.0 or managing guests manually).

Quoted 4 lines

current behavior in QEMU is intentionally conservative. threads number is subject to host configuration and limitations management layer puts on it and it's not QEMU job to conjure magic numbers that are host/workload depended.

Quoted 5 lines

I think that's missing the point. QEMU *did* historically set the prealloc threads equal to num CPUs, so we have precedent here. The referenced commit lost that behaviour because it only wired up the defaults in one particular CLI scenario. That's a clear regression on QEMU's side.

commit preserved behavior with legacy options to reduce disturbance. Considering that users will have update QEMU which most likely includes newer libvirt as well which lets configure number of allocation threads [1]. Users should fix configuration rather then pulling back layer violation hacks (precedent is not a good excuse wrt reluctance to adapt to new behavior)

1) https://libvirt.org/formatdomain.html#elementsMemoryAllocation

Quoted 3 lines

With regards,
Daniel

#5 lore raw
In reply to [email protected]
Quoted 28 lines · On 5/17/22 14:38, [email protected] wrote:

From: Jaroslav Jindrak <[email protected]>

Prior to the introduction of the prealloc-threads property, the amount of threads used to preallocate memory was derived from the value of smp-cpus passed to qemu, the amount of physical cpus of the host and a hardcoded maximum value. When the prealloc-threads property was introduced, it included a default of 1 in backends/hostmem.c and a default of smp-cpus using the sugar API for the property itself. The latter default is not used when the property is not specified on qemu's command line, so guests that were not adjusted for this change suddenly started to use the default of 1 thread to preallocate memory, which resulted in observable slowdowns in guest boots for guests with large memory (e.g. when using libvirt <8.2.0 or managing guests manually).

This commit restores the original behavior for these cases while not impacting guests started with the prealloc-threads property in any way.

Fixes 220c1fd864e9d ("hostmem: introduce "prealloc-threads" property")
---
  backends/hostmem.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
backends/hostmem.c +0 -0
index a7bae3d713..624bb7ecd3 100644

backends/hostmem.c static void host_memory_backend_init(Object *obj) · line 280

Quoted 8 lines
backends/hostmem.c +1 -1
@@ -274,7 +274,7 @@ static void host_memory_backend_init(Object *obj)
      backend->merge = machine_mem_merge(machine);
      backend->dump = machine_dump_guest_core(machine);
      backend->reserve = true;
-    backend->prealloc_threads = 1;
+    backend->prealloc_threads = machine->smp.cpus;
  }
 
  static void host_memory_backend_post_init(Object *obj)

Queued, thanks.

Paolo

#6 lore raw
Quoted 36 lines · On Tue, 17 May 2022 20:46:50 +0200 Paolo Bonzini <[email protected]> wrote:
Quoted 35 lines · On 5/17/22 14:38, [email protected] wrote:

From: Jaroslav Jindrak <[email protected]>

Prior to the introduction of the prealloc-threads property, the amount of threads used to preallocate memory was derived from the value of smp-cpus passed to qemu, the amount of physical cpus of the host and a hardcoded maximum value. When the prealloc-threads property was introduced, it included a default of 1 in backends/hostmem.c and a default of smp-cpus using the sugar API for the property itself. The latter default is not used when the property is not specified on qemu's command line, so guests that were not adjusted for this change suddenly started to use the default of 1 thread to preallocate memory, which resulted in observable slowdowns in guest boots for guests with large memory (e.g. when using libvirt <8.2.0 or managing guests manually).

This commit restores the original behavior for these cases while not impacting guests started with the prealloc-threads property in any way.

Fixes 220c1fd864e9d ("hostmem: introduce "prealloc-threads" property")
---
  backends/hostmem.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
backends/hostmem.c +1 -1
index a7bae3d713..624bb7ecd3 100644
@@ -274,7 +274,7 @@ static void host_memory_backend_init(Object *obj)
      backend->merge = machine_mem_merge(machine);
      backend->dump = machine_dump_guest_core(machine);
      backend->reserve = true;
-    backend->prealloc_threads = 1;
+    backend->prealloc_threads = machine->smp.cpus;
  }
 
Quoted 3 lines

backends/hostmem.c static void host_memory_backend_init(Object *obj) · line 280

Quoted 1 line
backends/hostmem.c +0 -0
  static void host_memory_backend_post_init(Object *obj)  

Queued, thanks.

could you drop this patch pls (there is an more acceptable alternative, see my other replies in this thread if we decide to put management policy decisions in QEMU code).

(well unless layer violation is acceptable practice now and it's really discouraging to do cleanup work if gets discarded)

PS:
There is no good default in this case (whatever number is picked it could be good or bad depending on usecase).

Quoted 2 lines

Paolo

#7 lore raw
Quoted 19 lines · Igor Mammedov
Quoted 8 lines · On Tue, 17 May 2022 20:46:50 +0200 Paolo Bonzini <[email protected]> wrote:
Quoted 8 lines
backends/hostmem.c +0 -0
index a7bae3d713..624bb7ecd3 100644
@@ -274,7 +274,7 @@ static void host_memory_backend_init(Object
 *obj)
       backend->merge = machine_mem_merge(machine);
       backend->dump = machine_dump_guest_core(machine);
Quoted 8 lines
Quoted 6 lines
backends/hostmem.c +1 -1
       backend->reserve = true;
-    backend->prealloc_threads = 1;
+    backend->prealloc_threads = machine->smp.cpus;
   }


  static void host_memory_backend_post_init(Object *obj)

Queued, thanks.

Quoted 4 lines

PS:
There is no good default in this case (whatever number is picked it could be good or bad depending on usecase).

That is fair enough. What we observed, however, is that, with QEMU 5.2, starting a 1024G VM takes ~34s.

Then you just update QEMU to > 5.2 (and don't do/changing anything else) and the same VM now takes ~4m30s to start.

If users are managing QEMU via Libvirt *and* have _at_least_ Libvirt 8.2, they can indeed set, e.g., <allocation mode='immediate' threads='NNN'/> (provided they can understand where the problem is, and figure out that this is the solution).

If they have Libvirt < 8.2 (e.g., people/distros that have, say, QEMU 6.2 and Libvirt 8.0.0, or something like that), there's basically nothing they can do... Except perhaps command line passthrough [1], but that's really rather tricky!

So, I personally don't know where any default should be set and how, but the above situation is not nice for users to have to handle.

[1] https://libvirt.org/kbase/qemu-passthrough-security.html

Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)
#8 lore raw
Quoted 27 lines · Dario Faggioli
Quoted 19 lines · Igor Mammedov
Quoted 8 lines · On Tue, 17 May 2022 20:46:50 +0200 Paolo Bonzini <[email protected]> wrote:
Quoted 8 lines
backends/hostmem.c +0 -0
index a7bae3d713..624bb7ecd3 100644
@@ -274,7 +274,7 @@ static void host_memory_backend_init(Object
 *obj)
       backend->merge = machine_mem_merge(machine);
       backend->dump = machine_dump_guest_core(machine);
Quoted 8 lines
Quoted 6 lines
backends/hostmem.c +1 -1
       backend->reserve = true;
-    backend->prealloc_threads = 1;
+    backend->prealloc_threads = machine->smp.cpus;
   }


  static void host_memory_backend_post_init(Object *obj)

Queued, thanks.

Quoted 4 lines

PS:
There is no good default in this case (whatever number is picked it could be good or bad depending on usecase).

That is fair enough. What we observed, however, is that, with QEMU 5.2, starting a 1024G VM takes ~34s.

Quoted 7 lines

Then you just update QEMU to > 5.2 (and don't do/changing anything else) and the same VM now takes ~4m30s to start.

If users are managing QEMU via Libvirt *and* have _at_least_ Libvirt 8.2, they can indeed set, e.g., <allocation mode='immediate' threads='NNN'/> (provided they can understand where the problem is, and figure out that this is the solution).

I think you get the QEMU version numbers a bit mixed up based on what i see in git history

Originally mem prellocation was single threaded and slow.

In v2.8.1 it switched to multi threaded with nthreads==vcpus

commit 1e356fc14beaa3ece6c0e961bd479af58be3198b
Author: Jitendra Kolhe <[email protected]>
Date: Fri Feb 24 09:01:43 2017 +0530

mem-prealloc: reduce large guest start-up and migration time.

This applied to --mem-prealloc and --object memory-backend*,prealloc=on

In v5.0.0 the prealloc-threads property was introduced with

commit ffac16fab33bb42f17e47624985220c1fd864e9d
Author: Igor Mammedov <[email protected]>
Date: Wed Feb 19 11:09:50 2020 -0500

hostmem: introduce "prealloc-threads" property

This changed it so that --mem-prealloc stil uses nthreads=vcpus but --object memory-backend,prealloc=on regressed to nthreads=1

When picking defaults there is never a perfect answer, it is more a matter of the least-worst option.

It is pretty clear that nthreads=1 is terrible for any large VMs. Defaulting it to nvcpus made conceptual sense as the user has implicit said that they expect the VM to be able to consume nvcpus worth of CPU time on the host, so we might as well consume that allotted resource.

I struggle to come up with a compelling reason why it is better to only use 1 single thread for preallocation. There might be some niches where its useful but I can't see it being the common case desirable behaviour.

Having different defaults based on how you configure it is also especially unplesant experience.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#9 lore raw
Quoted 8 lines · Daniel P. Berrangé

When picking defaults there is never a perfect answer, it is more a matter of the least-worst option.

It is pretty clear that nthreads=1 is terrible for any large VMs. Defaulting it to nvcpus made conceptual sense as the user has implicit said that they expect the VM to be able to consume nvcpus worth of CPU time on the host, so we might as well consume that allotted resource.

I agree. Yes, one could argue that the regression was on the libvirt side, but it's easier to fix it in QEMU.

If we later add the ability to create a memory backend before machine creation (for example with a QMP-only binary), then it's of course okay for those backends to use only one thread and require a manual choice for the # or preallocation threads.

Paolo

#10 lore raw
Quoted 9 lines · On Wed, 18 May 2022 16:06:47 +0200 Paolo Bonzini <[email protected]> wrote:
Quoted 8 lines · Daniel P. Berrangé

When picking defaults there is never a perfect answer, it is more a matter of the least-worst option.

It is pretty clear that nthreads=1 is terrible for any large VMs. Defaulting it to nvcpus made conceptual sense as the user has implicit said that they expect the VM to be able to consume nvcpus worth of CPU time on the host, so we might as well consume that allotted resource.

that assumes that allocation threads a permitted to actually use all resources and not limited to 1 pcpu only and then it also assumes 'more vcpus' => 'large RAM'.

Quoted 2 lines

I agree. Yes, one could argue that the regression was on the libvirt side, but it's easier to fix it in QEMU.

libvirt already provides means to set threads number, what needs fixing is setting up reasonable value in config which depends on how VM is configured and constrains mgmt/host put on it.

Quoted 4 lines

If we later add the ability to create a memory backend before machine creation (for example with a QMP-only binary), then it's of course okay for those backends to use only one thread and require a manual choice for the # or preallocation threads.

What I'm vehemently against is putting back direct machine references into backend code. I'm fine with 'prealloc-threads' property set from machine code (whether it's compat property or some sugar_prop() crutch in vl.c to appease CLI users).

Quoted 3 lines

Paolo