Skip to contentSkip to conversation

tools/libxl: make default of max event channels dependant on vcpus

PATCH In review Juergen Gross 4 participants 17 messages Apr 6, 2020 – Jun 2, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 tools/libxl: make default of max event channels dependant on vcpus

Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.

Instead of requiring to specify the allowed number of event channels via the "event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.

As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements. Nevertheless it is much better than the static upper limit of today as more guests will boot just fine with the new approach.

In order not to regress current configs use 1023 as the minimum default setting.

---
V2:
- use max() instead of min()
- clarify commit message a little bit

---
 tools/libxl/libxl_create.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/libxl/libxl_create.c +1 -1
index e7cb2dbc2b..c025b21894 100644
@@ -226,7 +226,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
             b_info->iomem[i].gfn = b_info->iomem[i].start;
 
     if (!b_info->event_channels)
-        b_info->event_channels = 1023;
+        b_info->event_channels = max(1023, b_info->max_vcpus * 8 + 255);
 
     libxl__arch_domain_build_info_setdefault(gc, b_info);
     libxl_defbool_setdefault(&b_info->dm_restrict, false);
Signature · 1 line
2.16.4
#1 lore raw

Hi Jurgen,

Quoted 4 lines · Juergen Gross

Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.

Large guests on which arch? Which type of guests?

Quoted 9 lines

Instead of requiring to specify the allowed number of event channels via the "event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.

As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements.

I am not overly happy to extend the default numbers of event channels for everyone based on Linux behavior on a given setup. Yes you have more guests that would be able to run, but at the expense of allowing a guest to use more xen memory.

For instance, I don't think this limit increase is necessary on Arm.

Quoted 7 lines

Nevertheless it is
much better than the static upper limit of today as more guests will boot just fine with the new approach.

In order not to regress current configs use 1023 as the minimum default setting.

Quoted 7 lines

---
V2:
- use max() instead of min()
- clarify commit message a little bit

---
  tools/libxl/libxl_create.c | 2 +-

The documentation should be updated.

tools/libxl/libxl_create.c int libxl__domain_build_info_setdefault(libxl__gc *gc, · line 229

Quoted 12 lines
  1 file changed, 1 insertion(+), 1 deletion(-)
tools/libxl/libxl_create.c +1 -1
index e7cb2dbc2b..c025b21894 100644
@@ -226,7 +226,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
              b_info->iomem[i].gfn = b_info->iomem[i].start;
 
      if (!b_info->event_channels)
-        b_info->event_channels = 1023;
+        b_info->event_channels = max(1023, b_info->max_vcpus * 8 + 255);

What is the 255 for?

Cheers,

Signature · 1 line
Julien Grall
#2 lore raw
Quoted 9 lines · Julien Grall

Hi Jurgen,

Quoted 4 lines · Juergen Gross

Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.

Large guests on which arch? Which type of guests?

I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.

Quoted 7 lines
Quoted 7 lines

Instead of requiring to specify the allowed number of event channels via the "event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.

Quoted 5 lines
Quoted 3 lines

As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements.

I am not overly happy to extend the default numbers of event channels for everyone based on Linux behavior on a given setup. Yes you have more guests that would be able to run, but at the expense of allowing a guest to use more xen memory.

The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.

Quoted 18 lines

For instance, I don't think this limit increase is necessary on Arm.

Quoted 7 lines

Nevertheless it is
much better than the static upper limit of today as more guests will boot just fine with the new approach.

In order not to regress current configs use 1023 as the minimum default setting.

Quoted 7 lines

---
V2:
- use max() instead of min()
- clarify commit message a little bit
---
  tools/libxl/libxl_create.c | 2 +-

Quoted 1 line

The documentation should be updated.

Oh, indeed.

Quoted 15 lines

tools/libxl/libxl_create.c int libxl__domain_build_info_setdefault(libxl__gc · line 229

Quoted 13 lines
  1 file changed, 1 insertion(+), 1 deletion(-)
tools/libxl/libxl_create.c +1 -1
index e7cb2dbc2b..c025b21894 100644
@@ -226,7 +226,7 @@ int libxl__domain_build_info_setdefault(libxl__gc 
 *gc,
               b_info->iomem[i].gfn = b_info->iomem[i].start;
       if (!b_info->event_channels)
-        b_info->event_channels = 1023;
+        b_info->event_channels = max(1023, b_info->max_vcpus * 8 + 255);

What is the 255 for?

Just some headroom for e.g. pv devices.

Juergen

#3 lore raw

Hi Juergen,

Quoted 9 lines · Jürgen Groß
Quoted 8 lines · Julien Grall

Hi Jurgen,

Quoted 4 lines · Juergen Gross

Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.

Quoted 4 lines
Quoted 1 line

Large guests on which arch? Which type of guests?

I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.

How about the guest types?

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

Instead of requiring to specify the allowed number of event channels via the "event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.

As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements.

Quoted 5 lines
Quoted 1 line

I am not overly happy to extend the default numbers of event channels for everyone based on Linux behavior on a given setup. Yes you have more guests that would be able to run, but at the expense of allowing a guest to use more xen memory.

The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.

This is not a very forward looking argument. For Arm, we limit to 128 vCPUs at the moment but it would be possible to support many more (I think our vGIC implementation support up to 4096 vCPUs).

So even if your change impacts a small subset, each architectures should be able to make the decision on the limit and not imposed by x86 Linux PV.

Quoted 40 lines
Quoted 19 lines

For instance, I don't think this limit increase is necessary on Arm.

Quoted 7 lines

Nevertheless it is
much better than the static upper limit of today as more guests will boot just fine with the new approach.

In order not to regress current configs use 1023 as the minimum default setting.

Quoted 7 lines

---
V2:
- use max() instead of min()
- clarify commit message a little bit
---
  tools/libxl/libxl_create.c | 2 +-

Quoted 1 line

The documentation should be updated.

Oh, indeed.

Quoted 16 lines
Quoted 8 lines
  1 file changed, 1 insertion(+), 1 deletion(-)
tools/libxl/libxl_create.c +0 -0
index e7cb2dbc2b..c025b21894 100644

tools/libxl/libxl_create.c int libxl__domain_build_info_setdefault(libxl__gc · line 230

Quoted 6 lines
tools/libxl/libxl_create.c +1 -1
@@ -226,7 +226,7 @@ int libxl__domain_build_info_setdefault(libxl__gc 
 *gc,
               b_info->iomem[i].gfn = b_info->iomem[i].start;
       if (!b_info->event_channels)
-        b_info->event_channels = 1023;
+        b_info->event_channels = max(1023, b_info->max_vcpus * 8 + 
 255);

What is the 255 for?

Quoted 1 line

Just some headroom for e.g. pv devices.

That should really be explained in the commit message and a comment.

Cheers,

Signature · 1 line
Julien Grall
#4 lore raw
Quoted 12 lines · Julien Grall

Hi Juergen,

Quoted 9 lines · Jürgen Groß
Quoted 8 lines · Julien Grall

Hi Jurgen,

Quoted 4 lines · Juergen Gross

Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Large guests on which arch? Which type of guests?

I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.

How about the guest types?

PV and HVM with PV enhancements.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

Instead of requiring to specify the allowed number of event channels via the "event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.

As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I am not overly happy to extend the default numbers of event channels for everyone based on Linux behavior on a given setup. Yes you have more guests that would be able to run, but at the expense of allowing a guest to use more xen memory.

The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.

This is not a very forward looking argument. For Arm, we limit to 128 vCPUs at the moment but it would be possible to support many more (I think our vGIC implementation support up to 4096 vCPUs).

So even if your change impacts a small subset, each architectures should be able to make the decision on the limit and not imposed by x86 Linux PV.

Okay, what about moving the default setting of b_info->event_channels into libxl__arch_domain_build_info_setdefault() then?

Quoted 41 lines
Quoted 41 lines
Quoted 20 lines

For instance, I don't think this limit increase is necessary on Arm.

Quoted 7 lines

Nevertheless it is
much better than the static upper limit of today as more guests will boot just fine with the new approach.

In order not to regress current configs use 1023 as the minimum default setting.

Quoted 7 lines

---
V2:
- use max() instead of min()
- clarify commit message a little bit
---
  tools/libxl/libxl_create.c | 2 +-

Quoted 1 line

The documentation should be updated.

Oh, indeed.

Quoted 16 lines
Quoted 8 lines
  1 file changed, 1 insertion(+), 1 deletion(-)
tools/libxl/libxl_create.c +0 -0
index e7cb2dbc2b..c025b21894 100644

tools/libxl/libxl_create.c int · line 230

Quoted 6 lines
tools/libxl/libxl_create.c +1 -1
@@ -226,7 +226,7 @@ int 
 libxl__domain_build_info_setdefault(libxl__gc *gc,
               b_info->iomem[i].gfn = b_info->iomem[i].start;
       if (!b_info->event_channels)
-        b_info->event_channels = 1023;
+        b_info->event_channels = max(1023, b_info->max_vcpus * 8 + 
 255);

What is the 255 for?

Quoted 3 lines
Quoted 1 line

Just some headroom for e.g. pv devices.

That should really be explained in the commit message and a comment.

Okay.

Juergen

#5 lore raw

Jürgen Groß writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 2 lines

Okay, what about moving the default setting of b_info->event_channels into libxl__arch_domain_build_info_setdefault() then?

FTAOD, if this satisfies ARM maintainers then I am content with it, even though I doubt the utility.

I guess you should make two patches 1. duplicate the existing formula (no functional change) 2. change the x86 formula.

I would ack the first and be guided by x86 folks for the 2nd.

Ian.

#6 lore raw

Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]

Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 6 lines

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Hi Ian,

Quoted 7 lines
Quoted 3 lines · Ian Jackson

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

... 1023 event channels is going to be plenty enough for most of the use cases.

OK, thanks for the quick reply.

So, Jürgen, I think everyone will be happy with this:

Ian Jackson writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 4 lines

I guess you should make two patches 1. duplicate the existing formula (no functional change) 2. change the x86 formula.

I would ack the first and be guided by x86 folks for the 2nd.

Ian.

#7 lore raw

Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]

Quoted 12 lines · Ian Jackson

Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 11 lines

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Hi Ian,

Quoted 3 lines · Ian Jackson

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

Quoted 12 lines
Quoted 2 lines

... 1023 event channels is going to be plenty enough for most of the use cases.

OK, thanks for the quick reply.

So, Jürgen, I think everyone will be happy with this:

Ian Jackson writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 4 lines

I guess you should make two patches 1. duplicate the existing formula (no functional change) 2. change the x86 formula.

I would ack the first and be guided by x86 folks for the 2nd.

Okay, thanks. Will do that.

Juergen

#8 lore raw

Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]

In reply to Ian Jackson
Quoted 12 lines · Ian Jackson

Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 11 lines

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Hi Ian,

Quoted 3 lines · Ian Jackson

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

Quoted 6 lines
Quoted 2 lines

... 1023 event channels is going to be plenty enough for most of the use cases.

OK, thanks for the quick reply.

So, Jürgen, I think everyone will be happy with this:

I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).

Jan

Quoted 8 lines

Ian Jackson writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 4 lines

I guess you should make two patches 1. duplicate the existing formula (no functional change) 2. change the x86 formula.

I would ack the first and be guided by x86 folks for the 2nd.

Ian.

#9 lore raw

Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]

Quoted 18 lines · Jan Beulich
Quoted 17 lines · Ian Jackson

Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 6 lines

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Hi Ian,

Quoted 7 lines
Quoted 3 lines · Ian Jackson

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

... 1023 event channels is going to be plenty enough for most of the use cases.

OK, thanks for the quick reply.

Quoted 7 lines
Quoted 1 line

So, Jürgen, I think everyone will be happy with this:

I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).

IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.

The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.

And how should Xen know the OS kernel needs exactly after all?

And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.

The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.

Juergen

#10 lore raw

Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]

Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson

Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 6 lines

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Hi Ian,

Quoted 7 lines
Quoted 3 lines · Ian Jackson

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

... 1023 event channels is going to be plenty enough for most of the use cases.

OK, thanks for the quick reply.

So, Jürgen, I think everyone will be happy with this:

Quoted 8 lines
Quoted 5 lines

I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).

IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.

This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...

Quoted 5 lines

The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.

And how should Xen know the OS kernel needs exactly after all?

... the answer of "It can#t" to this question.

Quoted 5 lines

And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.

Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.

Quoted 4 lines

The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.

I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.

Jan

#11 lore raw

Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]

Quoted 58 lines · Jan Beulich
Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson

Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 6 lines

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Hi Ian,

Quoted 7 lines
Quoted 3 lines · Ian Jackson

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

... 1023 event channels is going to be plenty enough for most of the use cases.

OK, thanks for the quick reply.

So, Jürgen, I think everyone will be happy with this:

Quoted 8 lines
Quoted 5 lines

I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).

IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.

This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...

Quoted 5 lines

The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.

And how should Xen know the OS kernel needs exactly after all?

... the answer of "It can#t" to this question.

Quoted 5 lines

And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.

Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.

Quoted 4 lines

The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.

Quoted 4 lines

I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.

Reviving this discussion.

I strongly disagree with your reasoning.

Rejecting to modify tools defaults for large guests to make them boot is a bad move IMO. We are driving more people away from Xen this way.

The fear of a misbehaving guest of that size to use a few additional pages on a machine with at least 100 cpus is fine from the academical point of view, but should not be weighed higher than the usability aspect in this case IMO.

Juergen

#12 lore raw

Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]

Quoted 68 lines · Jürgen Groß
Quoted 58 lines · Jan Beulich
Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson

Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 6 lines

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Hi Ian,

Quoted 7 lines
Quoted 3 lines · Ian Jackson

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

... 1023 event channels is going to be plenty enough for most of the use cases.

OK, thanks for the quick reply.

So, Jürgen, I think everyone will be happy with this:

Quoted 8 lines
Quoted 5 lines

I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).

IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.

This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...

Quoted 5 lines

The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.

And how should Xen know the OS kernel needs exactly after all?

... the answer of "It can#t" to this question.

Quoted 5 lines

And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.

Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.

Quoted 4 lines

The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.

Quoted 4 lines

I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.

Reviving this discussion.

I strongly disagree with your reasoning.

Quoted 7 lines

Rejecting to modify tools defaults for large guests to make them boot is a bad move IMO. We are driving more people away from Xen this way.

The fear of a misbehaving guest of that size to use a few additional pages on a machine with at least 100 cpus is fine from the academical point of view, but should not be weighed higher than the usability aspect in this case IMO.

Very simple question then: Where do you draw the boundary if you don't want this to be a pure "is permitted" or "is not permitted" underlying rule? If we had a model where _all_ resources consumed by a guest were accounted against its tool stack requested allocation, things would be easier.

Jan

#13 lore raw

Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]

Quoted 77 lines · Jan Beulich
Quoted 68 lines · Jürgen Groß
Quoted 58 lines · Jan Beulich
Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson

Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 6 lines

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Hi Ian,

Quoted 7 lines
Quoted 3 lines · Ian Jackson

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

... 1023 event channels is going to be plenty enough for most of the use cases.

OK, thanks for the quick reply.

So, Jürgen, I think everyone will be happy with this:

Quoted 8 lines
Quoted 5 lines

I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).

IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.

This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...

Quoted 5 lines

The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.

And how should Xen know the OS kernel needs exactly after all?

... the answer of "It can#t" to this question.

Quoted 5 lines

And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.

Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.

Quoted 4 lines

The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.

Quoted 4 lines

I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.

Reviving this discussion.

I strongly disagree with your reasoning.

Quoted 7 lines

Rejecting to modify tools defaults for large guests to make them boot is a bad move IMO. We are driving more people away from Xen this way.

The fear of a misbehaving guest of that size to use a few additional pages on a machine with at least 100 cpus is fine from the academical point of view, but should not be weighed higher than the usability aspect in this case IMO.

Quoted 5 lines

Very simple question then: Where do you draw the boundary if you don't want this to be a pure "is permitted" or "is not permitted" underlying rule? If we had a model where _all_ resources consumed by a guest were accounted against its tool stack requested allocation, things would be easier.

I'd say it should be allowed in case the additional resource use is much smaller than the already used implicit resources for such a guest (e.g. less than an additional 1% of implicitly used memory).

In cases like this, where a very small subset of guests is affected, and the additional need of resources will apply only in very extreme cases (I'm considering this case as extreme, as only non-Linux guests with huge numbers of vcpus _and_ which are misbehaving will need additional resources) I'd even accept higher margins like 5%.

Juergen

#14 lore raw

Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]

Quoted 88 lines · Jürgen Groß
Quoted 77 lines · Jan Beulich
Quoted 68 lines · Jürgen Groß
Quoted 58 lines · Jan Beulich
Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson

Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 6 lines

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Hi Ian,

Quoted 7 lines
Quoted 3 lines · Ian Jackson

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

... 1023 event channels is going to be plenty enough for most of the use cases.

OK, thanks for the quick reply.

So, Jürgen, I think everyone will be happy with this:

Quoted 8 lines
Quoted 5 lines

I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).

IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.

This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...

Quoted 5 lines

The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.

And how should Xen know the OS kernel needs exactly after all?

... the answer of "It can#t" to this question.

Quoted 5 lines

And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.

Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.

Quoted 4 lines

The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.

Quoted 4 lines

I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.

Reviving this discussion.

I strongly disagree with your reasoning.

Quoted 7 lines

Rejecting to modify tools defaults for large guests to make them boot is a bad move IMO. We are driving more people away from Xen this way.

The fear of a misbehaving guest of that size to use a few additional pages on a machine with at least 100 cpus is fine from the academical point of view, but should not be weighed higher than the usability aspect in this case IMO.

Quoted 5 lines

Very simple question then: Where do you draw the boundary if you don't want this to be a pure "is permitted" or "is not permitted" underlying rule? If we had a model where _all_ resources consumed by a guest were accounted against its tool stack requested allocation, things would be easier.

I'd say it should be allowed in case the additional resource use is much smaller than the already used implicit resources for such a guest (e.g. less than an additional 1% of implicitly used memory).

Quoted 5 lines

In cases like this, where a very small subset of guests is affected, and the additional need of resources will apply only in very extreme cases (I'm considering this case as extreme, as only non-Linux guests with huge numbers of vcpus _and_ which are misbehaving will need additional resources) I'd even accept higher margins like 5%.

IOW if we had 20 such cases, doubling resource consumption would be okay to you? Not to me...

FAOD: If I'm the (almost?) only one to object here, I'll be okay to be outvoted. But I'd like people agreeing with the change to explicitly ack that they're fine with the unwanted (as I'd call it) side effects.

Jan

#15 lore raw
In reply to Jürgen Groß

Jürgen Groß writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 5 lines · Julien Grall
Quoted 1 line

Large guests on which arch? Which type of guests?

I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.

Should this be arch-dependent then ? It seems like the figure is just a heuristic anyway, and ...

Quoted 3 lines

The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.

Julien, are there likely to be any ARM guests now which have anywhere near that number of vcpus ? If not do we know now what such guests are likely to be like ?

If this is all hypothetical on ARM it would seem silly to make this arch-specific for the benefit of ARM given that the ARM implementation would be entirely guesswork. Maybe we should postpone that specialisation until we know better what the ARM function should be like for these large numbers of vcpus.

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

Ian.

#16 lore raw

Hi Ian,

Quoted 10 lines · Ian Jackson

Jürgen Groß writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):

Quoted 5 lines · Julien Grall
Quoted 1 line

Large guests on which arch? Which type of guests?

I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.

Should this be arch-dependent then ? It seems like the figure is just a heuristic anyway, and ...

Quoted 7 lines
Quoted 3 lines

The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.

Julien, are there likely to be any ARM guests now which have anywhere near that number of vcpus ? If not do we know now what such guests are likely to be like ?

We are meant to support up to 128 vCPUs. But our implementation can support up to 4096 vCPUs on vGICv3.

Quoted 6 lines

If this is all hypothetical on ARM it would seem silly to make this arch-specific for the benefit of ARM given that the ARM implementation would be entirely guesswork. Maybe we should postpone that specialisation until we know better what the ARM function should be like for these large numbers of vcpus.

There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...

Quoted 3 lines

If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.

... 1023 event channels is going to be plenty enough for most of the use cases.

Cheers,

Signature · 1 line
Julien Grall