tools/libxl: make default of max event channels dependant on vcpus
PATCH v2 1/1 tools/libxl: make default of max event channels dependant on vcpus
Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.
Instead of requiring to specify the allowed number of event channels via the "event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.
As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements. Nevertheless it is much better than the static upper limit of today as more guests will boot just fine with the new approach.
In order not to regress current configs use 1023 as the minimum default setting.
---
V2:
- use max() instead of min()
- clarify commit message a little bit
--- tools/libxl/libxl_create.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/libxl/libxl_create.c +1 -1
@@ -226,7 +226,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc, b_info->iomem[i].gfn = b_info->iomem[i].start;if (!b_info->event_channels) - b_info->event_channels =1023 ; + b_info->event_channels =max (1023 , b_info->max_vcpus *8 +255 );libxl__arch_domain_build_info_setdefault (gc, b_info);libxl_defbool_setdefault (&b_info->dm_restrict, false);
Signature · 1 line
2.16.4
Hi Jurgen,
Quoted 4 lines · Juergen Gross
Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.
Large guests on which arch? Which type of guests?
Quoted 9 lines
Instead of requiring to specify the allowed number of event channels via the "
event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements.
I am not overly happy to extend the default numbers of event channels for everyone based on Linux behavior on a given setup. Yes you have more guests that would be able to run, but at the expense of allowing a guest to use more xen memory.
For instance, I don't think this limit increase is necessary on Arm.
Quoted 7 lines
Nevertheless it is
much better than the static upper limit of today as more guests will boot just fine with the new approach.In order not to regress current configs use 1023 as the minimum default setting.
Quoted 7 lines
Signed-off-by Juergen Gross <[email protected]>---
V2:
- usemax()instead ofmin()
- clarify commit message a little bit--- tools/libxl/libxl_create.c | 2 +-
The documentation should be updated.
tools/libxl/libxl_create.c int libxl__domain_build_info_setdefault(libxl__gc *gc, · line 229
Quoted 12 lines
1 file changed, 1 insertion(+), 1 deletion(-)tools/libxl/libxl_create.c +1
-1@@ -226,7 +226,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc, b_info->iomem[i].gfn = b_info->iomem[i].start;if (!b_info->event_channels) - b_info->event_channels =1023 ; + b_info->event_channels =max (1023 , b_info->max_vcpus *8 +255 );
What is the 255 for?
Cheers,
Signature · 1 line
Julien Grall
Quoted 9 lines · Julien Grall
Hi Jurgen,
Quoted 4 lines · Juergen Gross
Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.
Large guests on which arch? Which type of guests?
I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.
Quoted 7 lines
Quoted 7 lines
Instead of requiring to specify the allowed number of event channels via the "
event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.
Quoted 5 lines
Quoted 3 lines
As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements.
I am not overly happy to extend the default numbers of event channels for everyone based on Linux behavior on a given setup. Yes you have more guests that would be able to run, but at the expense of allowing a guest to use more xen memory.
The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.
Quoted 18 lines
For instance, I don't think this limit increase is necessary on Arm.
Quoted 7 lines
Nevertheless it is
much better than the static upper limit of today as more guests will boot just fine with the new approach.In order not to regress current configs use 1023 as the minimum default setting.
Quoted 7 lines
Signed-off-by Juergen Gross <[email protected]>---
V2:
- usemax()instead ofmin()
- clarify commit message a little bit
---
tools/libxl/libxl_create.c| 2 +-
Quoted 1 line
The documentation should be updated.
Oh, indeed.
Quoted 15 lines
tools/libxl/libxl_create.c int libxl__domain_build_info_setdefault(libxl__gc · line 229
Quoted 13 lines
1 file changed, 1 insertion(+), 1 deletion(-)tools/libxl/libxl_create.c +1
-1@@ -226,7 +226,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc, b_info->iomem[i].gfn = b_info->iomem[i].start; if (!b_info->event_channels) - b_info->event_channels = 1023; + b_info->event_channels = max(1023, b_info->max_vcpus * 8 + 255);What is the 255 for?
Just some headroom for e.g. pv devices.
Juergen
Hi Juergen,
Quoted 9 lines · Jürgen Groß
Quoted 8 lines · Julien Grall
Hi Jurgen,
Quoted 4 lines · Juergen Gross
Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.
Quoted 4 lines
Quoted 1 line
Large guests on which arch? Which type of guests?
I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.
How about the guest types?
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Instead of requiring to specify the allowed number of event channels via the "
event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements.
Quoted 5 lines
Quoted 1 line
I am not overly happy to extend the default numbers of event channels for everyone based on Linux behavior on a given setup. Yes you have more guests that would be able to run, but at the expense of allowing a guest to use more xen memory.
The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.
This is not a very forward looking argument. For Arm, we limit to 128 vCPUs at the moment but it would be possible to support many more (I think our vGIC implementation support up to 4096 vCPUs).
So even if your change impacts a small subset, each architectures should be able to make the decision on the limit and not imposed by x86 Linux PV.
Quoted 40 lines
Quoted 19 lines
For instance, I don't think this limit increase is necessary on Arm.
Quoted 7 lines
Nevertheless it is
much better than the static upper limit of today as more guests will boot just fine with the new approach.In order not to regress current configs use 1023 as the minimum default setting.
Quoted 7 lines
Signed-off-by Juergen Gross <[email protected]>---
V2:
- usemax()instead ofmin()
- clarify commit message a little bit
---
tools/libxl/libxl_create.c| 2 +-Quoted 1 line
The documentation should be updated.
Oh, indeed.
Quoted 16 lines
Quoted 8 lines
1 file changed, 1 insertion(+), 1 deletion(-)tools/libxl/libxl_create.c +0
-0tools/libxl/libxl_create.c int libxl__domain_build_info_setdefault(libxl__gc · line 230
Quoted 6 lines
tools/libxl/libxl_create.c +1
-1@@ -226,7 +226,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc, b_info->iomem[i].gfn = b_info->iomem[i].start; if (!b_info->event_channels) - b_info->event_channels = 1023; + b_info->event_channels = max(1023, b_info->max_vcpus * 8 +255 );What is the 255 for?
Quoted 1 line
Just some headroom for e.g. pv devices.
That should really be explained in the commit message and a comment.
Cheers,
Signature · 1 line
Julien Grall
Quoted 12 lines · Julien Grall
Hi Juergen,
Quoted 9 lines · Jürgen Groß
Quoted 8 lines · Julien Grall
Hi Jurgen,
Quoted 4 lines · Juergen Gross
Since Xen 4.4 the maximum number of event channels for a guest is defaulting to 1023. For large guests with lots of vcpus this is not enough, as e.g. the Linux kernel uses 7 event channels per vcpu, limiting the guest to about 140 vcpus.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Large guests on which arch? Which type of guests?
I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.
How about the guest types?
PV and HVM with PV enhancements.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Instead of requiring to specify the allowed number of event channels via the "
event_channels" domain config option, make the default depend on the maximum number of vcpus of the guest. This will require to use the "event_channels" domain config option in fewer cases as before.As different guests will have differing needs the calculation of the maximum number of event channels can be a rough estimate only, currently based on the Linux kernel requirements.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I am not overly happy to extend the default numbers of event channels for everyone based on Linux behavior on a given setup. Yes you have more guests that would be able to run, but at the expense of allowing a guest to use more xen memory.
The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.
This is not a very forward looking argument. For Arm, we limit to 128 vCPUs at the moment but it would be possible to support many more (I think our vGIC implementation support up to 4096 vCPUs).
So even if your change impacts a small subset, each architectures should be able to make the decision on the limit and not imposed by x86 Linux PV.
Okay, what about moving the default setting of b_info->event_channels into libxl__arch_domain_build_info_setdefault() then?
Quoted 41 lines
Quoted 41 lines
Quoted 20 lines
For instance, I don't think this limit increase is necessary on Arm.
Quoted 7 lines
Nevertheless it is
much better than the static upper limit of today as more guests will boot just fine with the new approach.In order not to regress current configs use 1023 as the minimum default setting.
Quoted 7 lines
Signed-off-by Juergen Gross <[email protected]>---
V2:
- usemax()instead ofmin()
- clarify commit message a little bit
---
tools/libxl/libxl_create.c| 2 +-Quoted 1 line
The documentation should be updated.
Oh, indeed.
Quoted 16 lines
Quoted 8 lines
1 file changed, 1 insertion(+), 1 deletion(-)tools/libxl/libxl_create.c +0
-0tools/libxl/libxl_create.c int · line 230
Quoted 6 lines
tools/libxl/libxl_create.c +1
-1@@ -226,7 +226,7 @@ intlibxl__domain_build_info_setdefault (libxl__gc *gc, b_info->iomem[i].gfn = b_info->iomem[i].start; if (!b_info->event_channels) - b_info->event_channels = 1023; + b_info->event_channels = max(1023, b_info->max_vcpus * 8 + 255);What is the 255 for?
Quoted 3 lines
Quoted 1 line
Just some headroom for e.g. pv devices.
That should really be explained in the commit message and a comment.
Okay.
Juergen
Jürgen Groß writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):
Quoted 2 lines
Okay, what about moving the default setting of
b_info->event_channelsintolibxl__arch_domain_build_info_setdefault()then?
FTAOD, if this satisfies ARM maintainers then I am content with it, even though I doubt the utility.
I guess you should make two patches 1. duplicate the existing formula (no functional change) 2. change the x86 formula.
I would ack the first and be guided by x86 folks for the 2nd.
Ian.
Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]
Julien Grall writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):
Quoted 6 lines
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Hi Ian,
Quoted 7 lines
Quoted 3 lines · Ian Jackson
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
... 1023 event channels is going to be plenty enough for most of the use cases.
OK, thanks for the quick reply.
So, Jürgen, I think everyone will be happy with this:
Ian Jackson writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):
Quoted 4 lines
I guess you should make two patches 1. duplicate the existing formula (no functional change) 2. change the x86 formula.
I would ack the first and be guided by x86 folks for the 2nd.
Ian.
Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]
Quoted 12 lines · Ian Jackson
Julien Grall writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 11 lines
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Hi Ian,
Quoted 3 lines · Ian Jackson
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
Quoted 12 lines
Quoted 2 lines
... 1023 event channels is going to be plenty enough for most of the use cases.
OK, thanks for the quick reply.
So, Jürgen, I think everyone will be happy with this:
Ian Jackson writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 4 lines
I guess you should make two patches 1. duplicate the existing formula (no functional change) 2. change the x86 formula.
I would ack the first and be guided by x86 folks for the 2nd.
Okay, thanks. Will do that.
Juergen
Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]
In reply to Ian JacksonQuoted 12 lines · Ian Jackson
Julien Grall writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 11 lines
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Hi Ian,
Quoted 3 lines · Ian Jackson
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
Quoted 6 lines
Quoted 2 lines
... 1023 event channels is going to be plenty enough for most of the use cases.
OK, thanks for the quick reply.
So, Jürgen, I think everyone will be happy with this:
I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).
Jan
Quoted 8 lines
Ian Jackson writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 4 lines
I guess you should make two patches 1. duplicate the existing formula (no functional change) 2. change the x86 formula.
I would ack the first and be guided by x86 folks for the 2nd.
Ian.
Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Ian Jackson
Julien Grall writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 6 lines
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Hi Ian,
Quoted 7 lines
Quoted 3 lines · Ian Jackson
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
... 1023 event channels is going to be plenty enough for most of the use cases.
OK, thanks for the quick reply.
Quoted 7 lines
Quoted 1 line
So, Jürgen, I think everyone will be happy with this:
I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).
IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.
The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.
And how should Xen know the OS kernel needs exactly after all?
And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.
The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.
Juergen
Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]
Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson
Julien Grall writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 6 lines
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Hi Ian,
Quoted 7 lines
Quoted 3 lines · Ian Jackson
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
... 1023 event channels is going to be plenty enough for most of the use cases.
OK, thanks for the quick reply.
So, Jürgen, I think everyone will be happy with this:
Quoted 8 lines
Quoted 5 lines
I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).
IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.
This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...
Quoted 5 lines
The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.
And how should Xen know the OS kernel needs exactly after all?
... the answer of "It can#t" to this question.
Quoted 5 lines
And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.
Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.
Quoted 4 lines
The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.
I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.
Jan
Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]
Quoted 58 lines · Jan Beulich
Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson
Julien Grall writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 6 lines
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Hi Ian,
Quoted 7 lines
Quoted 3 lines · Ian Jackson
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
... 1023 event channels is going to be plenty enough for most of the use cases.
OK, thanks for the quick reply.
So, Jürgen, I think everyone will be happy with this:
Quoted 8 lines
Quoted 5 lines
I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).
IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.
This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...
Quoted 5 lines
The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.
And how should Xen know the OS kernel needs exactly after all?
... the answer of "It can#t" to this question.
Quoted 5 lines
And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.
Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.
Quoted 4 lines
The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.
Quoted 4 lines
I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.
Reviving this discussion.
I strongly disagree with your reasoning.
Rejecting to modify tools defaults for large guests to make them boot is a bad move IMO. We are driving more people away from Xen this way.
The fear of a misbehaving guest of that size to use a few additional pages on a machine with at least 100 cpus is fine from the academical point of view, but should not be weighed higher than the usability aspect in this case IMO.
Juergen
Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]
Quoted 68 lines · Jürgen Groß
Quoted 58 lines · Jan Beulich
Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson
Julien Grall writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 6 lines
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Hi Ian,
Quoted 7 lines
Quoted 3 lines · Ian Jackson
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
... 1023 event channels is going to be plenty enough for most of the use cases.
OK, thanks for the quick reply.
So, Jürgen, I think everyone will be happy with this:
Quoted 8 lines
Quoted 5 lines
I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).
IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.
This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...
Quoted 5 lines
The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.
And how should Xen know the OS kernel needs exactly after all?
... the answer of "It can#t" to this question.
Quoted 5 lines
And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.
Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.
Quoted 4 lines
The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.
Quoted 4 lines
I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.
Reviving this discussion.
I strongly disagree with your reasoning.
Quoted 7 lines
Rejecting to modify tools defaults for large guests to make them boot is a bad move IMO. We are driving more people away from Xen this way.
The fear of a misbehaving guest of that size to use a few additional pages on a machine with at least 100 cpus is fine from the academical point of view, but should not be weighed higher than the usability aspect in this case IMO.
Very simple question then: Where do you draw the boundary if you don't want this to be a pure "is permitted" or "is not permitted" underlying rule? If we had a model where _all_ resources consumed by a guest were accounted against its tool stack requested allocation, things would be easier.
Jan
Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]
Quoted 77 lines · Jan Beulich
Quoted 68 lines · Jürgen Groß
Quoted 58 lines · Jan Beulich
Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson
Julien Grall writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 6 lines
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Hi Ian,
Quoted 7 lines
Quoted 3 lines · Ian Jackson
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
... 1023 event channels is going to be plenty enough for most of the use cases.
OK, thanks for the quick reply.
So, Jürgen, I think everyone will be happy with this:
Quoted 8 lines
Quoted 5 lines
I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).
IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.
This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...
Quoted 5 lines
The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.
And how should Xen know the OS kernel needs exactly after all?
... the answer of "It can#t" to this question.
Quoted 5 lines
And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.
Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.
Quoted 4 lines
The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.
Quoted 4 lines
I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.
Reviving this discussion.
I strongly disagree with your reasoning.
Quoted 7 lines
Rejecting to modify tools defaults for large guests to make them boot is a bad move IMO. We are driving more people away from Xen this way.
The fear of a misbehaving guest of that size to use a few additional pages on a machine with at least 100 cpus is fine from the academical point of view, but should not be weighed higher than the usability aspect in this case IMO.
Quoted 5 lines
Very simple question then: Where do you draw the boundary if you don't want this to be a pure "is permitted" or "is not permitted" underlying rule? If we had a model where _all_ resources consumed by a guest were accounted against its tool stack requested allocation, things would be easier.
I'd say it should be allowed in case the additional resource use is much smaller than the already used implicit resources for such a guest (e.g. less than an additional 1% of implicitly used memory).
In cases like this, where a very small subset of guests is affected, and the additional need of resources will apply only in very extreme cases (I'm considering this case as extreme, as only non-Linux guests with huge numbers of vcpus _and_ which are misbehaving will need additional resources) I'd even accept higher margins like 5%.
Juergen
Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus [and 1 more messages]
Quoted 88 lines · Jürgen Groß
Quoted 77 lines · Jan Beulich
Quoted 68 lines · Jürgen Groß
Quoted 58 lines · Jan Beulich
Quoted 21 lines · Jürgen Groß
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Ian Jackson
Julien Grall writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 6 lines
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Hi Ian,
Quoted 7 lines
Quoted 3 lines · Ian Jackson
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
... 1023 event channels is going to be plenty enough for most of the use cases.
OK, thanks for the quick reply.
So, Jürgen, I think everyone will be happy with this:
Quoted 8 lines
Quoted 5 lines
I don't think I will be - my prior comment still holds on there not being any grounds to use a specific OS kernel's (and to be precise a specific OS kernel version's) requirements for determining defaults. If there was to be such a dependency, then OS kernel [variant] should be part of the inputs to such a (set of) formula(s).
IMO this kind of trying to be perfect will completely block a sane heuristic for being able to boot large guests at all.
This isn't about being perfect - I'm suggesting to leave the default alone, not to improve the calculation, not the least because I've been implying ...
Quoted 5 lines
The patch isn't about to find an as stringent as possible upper boundary for huge guests, but a sane value being able to boot most of those.
And how should Xen know the OS kernel needs exactly after all?
... the answer of "It can#t" to this question.
Quoted 5 lines
And it is not that we talking about megabytes of additional memory. A guest with 256 vcpus will just be able to use additional 36 memory pages. The maximum non-PV domain (the probably only relevant case of another OS than Linux being used) with 128 vcpus would "waste" 32 kB. In case the guest misbehaves.
Any extra page counts, or else - where do you draw the line? Any single page may decide between Xen (not) being out of memory, and hence also not being able to fulfill certain other requests.
Quoted 4 lines
The alternative would be to do nothing and having to let the user experience a somewhat cryptic guest crash. He could google for a possible solution which would probably end in a rather high static limit resulting in wasting even more memory.
Quoted 4 lines
I realize this. Otoh more people running into this will improve the chances of later ones finding useful suggestions. Of course there's also nothing wrong with trying to make the error less cryptic.
Reviving this discussion.
I strongly disagree with your reasoning.
Quoted 7 lines
Rejecting to modify tools defaults for large guests to make them boot is a bad move IMO. We are driving more people away from Xen this way.
The fear of a misbehaving guest of that size to use a few additional pages on a machine with at least 100 cpus is fine from the academical point of view, but should not be weighed higher than the usability aspect in this case IMO.
Quoted 5 lines
Very simple question then: Where do you draw the boundary if you don't want this to be a pure "is permitted" or "is not permitted" underlying rule? If we had a model where _all_ resources consumed by a guest were accounted against its tool stack requested allocation, things would be easier.
I'd say it should be allowed in case the additional resource use is much smaller than the already used implicit resources for such a guest (e.g. less than an additional 1% of implicitly used memory).
Quoted 5 lines
In cases like this, where a very small subset of guests is affected, and the additional need of resources will apply only in very extreme cases (I'm considering this case as extreme, as only non-Linux guests with huge numbers of vcpus _and_ which are misbehaving will need additional resources) I'd even accept higher margins like 5%.
IOW if we had 20 such cases, doubling resource consumption would be okay to you? Not to me...
FAOD: If I'm the (almost?) only one to object here, I'll be okay to be outvoted. But I'd like people agreeing with the change to explicitly ack that they're fine with the unwanted (as I'd call it) side effects.
Jan
Jürgen Groß writes ("Re: [PATCH v2] tools/libxl: make default of max event channels dependant on vcpus"):
Quoted 5 lines · Julien Grall
Quoted 1 line
Large guests on which arch? Which type of guests?
I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.
Should this be arch-dependent then ? It seems like the figure is just a heuristic anyway, and ...
Quoted 3 lines
The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.
Julien, are there likely to be any ARM guests now which have anywhere near that number of vcpus ? If not do we know now what such guests are likely to be like ?
If this is all hypothetical on ARM it would seem silly to make this arch-specific for the benefit of ARM given that the ARM implementation would be entirely guesswork. Maybe we should postpone that specialisation until we know better what the ARM function should be like for these large numbers of vcpus.
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
Ian.
Hi Ian,
Quoted 10 lines · Ian Jackson
Jürgen Groß writes ("Re: [PATCH v2]
tools/libxl: make default of max event channels dependant on vcpus"):Quoted 5 lines · Julien Grall
Quoted 1 line
Large guests on which arch? Which type of guests?
I'm pretty sure this applies to x86 only. I'm not aware of event channels being used on ARM for IPIs.
Should this be arch-dependent then ? It seems like the figure is just a heuristic anyway, and ...
Quoted 7 lines
Quoted 3 lines
The resulting number would be larger than today only for guests with more than 96 vcpus. So I don't think the additional amount of memory is really that problematic.
Julien, are there likely to be any ARM guests now which have anywhere near that number of vcpus ? If not do we know now what such guests are likely to be like ?
We are meant to support up to 128 vCPUs. But our implementation can support up to 4096 vCPUs on vGICv3.
Quoted 6 lines
If this is all hypothetical on ARM it would seem silly to make this arch-specific for the benefit of ARM given that the ARM implementation would be entirely guesswork. Maybe we should postpone that specialisation until we know better what the ARM function should be like for these large numbers of vcpus.
There are no correlation between event channels and vCPUs. The number of event channels only depends on the number of frontend you have in your guest. So...
Quoted 3 lines
If ARM folks want to have a different formula for the default then that is of course fine but I wonder whether this might do ARMk more harm than good in this case.
... 1023 event channels is going to be plenty enough for most of the use cases.
Cheers,
Signature · 1 line
Julien Grall