Make credit2 the default scheduler
PATCH 1/1 Make credit2 the default scheduler
Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).
Why change the default?
The code is better: more predictable, less jitter, easier to determine how modifications will affect overall behavior, easier in the future to make load-balancing behavior more subtle (e.g., taking into account the cost of powering up extra cores, &c).
Overall performance compared to Credit1 is somewhat of a mixed bag. Unfortunately most of what I have are tests using XenServer's internal perf testing system, so I can't share the raw data (via links anyway).
Here is a summary of data from an internal e-mail Dario sent in the past:
* DVDbench: On underloaded systems, credit2 outperformed credit1 by about 4%. On overloaded systems, credit2 underperformed by about 3%.
* On a range of tests (unixbench, lmbench, &c), credit and credit2 perform within 5% of each other (up and down).
* Credit2 fairly consistently beats credit for TCP-style workloads.
* Credit2 is sometimes equal to, sometimes 5-15% worse than, credit for synthetic CPU workloads (e.g., Dhrystone).
* On LoginVSI, credit2 fairly consistently outperforms credit by about 10%.
Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.
---
CC: Ian Jackson <[email protected]>
CC: Wei Liu <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Tim Deegan <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Konrad Wilk <[email protected]>
CC: Dario Faggioli <[email protected]>
--- xen/common/Kconfig | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
xen/common/Kconfig +2 -2
@@ -215,7 +215,7 @@ config SCHED_NULL choice prompt "Default Scheduler?" - default SCHED_CREDIT_DEFAULT + default SCHED_CREDIT2_DEFAULT config SCHED_CREDIT_DEFAULT bool "Credit Scheduler" if SCHED_CREDIT
@@ -236,7 +236,7 @@ config SCHED_DEFAULT default "rtds" if SCHED_RTDS_DEFAULT default "arinc653" if SCHED_ARINC653_DEFAULT default "null" if SCHED_NULL_DEFAULT - default "credit" + default "credit2" endmenu
Signature · 7 lines
2.18.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 5 lines · George Dunlap
Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).
[..]
Quoted 9 lines
Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.
Signed-off-by George Dunlap <[email protected]>
After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)
I fully agree with and second George's reasoning, and feel 100% like providing my:
Of course, although we thing the scheduler is ready, the more testing/profiling/benchmarking/... we get, the better.
So, everyone, don't be shy at throwing any kind of things against it, and report what happens, so we can fix the problem (or revert to Credit!) for 4.12
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Software Engineer @ SUSE https://www.suse.com/
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 16 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
Quoted 5 lines · George Dunlap
Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).
[..]
Quoted 9 lines
Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.
Signed-off-by George Dunlap <[email protected]>
Quoted 7 lines
After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)
I fully agree with and second George's reasoning, and feel 100% like providing my:
Acked-by Dario Faggioli <[email protected]>
And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
Quoted 5 lines · George Dunlap
Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).
[..]
Quoted 9 lines
Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.
Signed-off-by George Dunlap <[email protected]>After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)
Quoted 7 lines
Quoted 4 lines
I fully agree with and second George's reasoning, and feel 100% like providing my:
Acked-by Dario Faggioli <[email protected]>And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for
Urgh, this is a really nasty bug.
Quoted 2 lines
much longer, in particular with there already being a fix available from you I think that one should go in before the one here.
+1.
Wei.
Quoted 4 lines
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 28 lines · Wei Liu
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
Quoted 5 lines · George Dunlap
Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).
[..]
Quoted 9 lines
Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.
Signed-off-by George Dunlap <[email protected]>After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)
Quoted 7 lines
Quoted 4 lines
I fully agree with and second George's reasoning, and feel 100% like providing my:
Acked-by Dario Faggioli <[email protected]>And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for
Quoted 6 lines
Urgh, this is a really nasty bug.
Quoted 2 lines
much longer, in particular with there already being a fix available from you I think that one should go in before the one here.
+1.
The other fix is in. I'll wait a few days and push this change, if there are no objections.
-George
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 22 lines · AEST Jan Beulich
Quoted 22 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
Quoted 5 lines · George Dunlap
Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).
[..]
Quoted 8 lines
Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.
Signed-off-by George Dunlap <[email protected]>After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)
I fully agree with and second George's reasoning, and feel 100% like providing my:
Quoted 6 lines
Quoted 1 line
Acked-by Dario Faggioli <[email protected]>And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.
Even though my opinion probably isn't very heavy on this matter, I've used credit2 exclusively for a considerable time. If you're talking the issue I think you're talking about, then I discovered it when doing stuff that most people probably wouldn't bother with - evidenced that I hadn't done it before either.
I take peoples word on the performance +/- of a few percent here and there - so if its easier to maintain and better code, then yeah - it makes sense to move on with it. I certainly haven't found any normal use cases that would lead me to object to this.
Signature · 4 lines
Steven Haigh 📧 [email protected] 💻 https://www.crc.id.au 📞 +61 (3) 9001 6090 📱 0412 935 897
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 11 lines · Steven Haigh
Quoted 5 lines · AEST Jan Beulich
Quoted 1 line
Quoted 1 line
Quoted 1 line
And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.
Even though my opinion probably isn't very heavy on this matter, I've used
credit2 exclusively for a considerable time.
Well, this is really interesting and useful to know. Can I ask what your typical workload is (if any), and how are things going?
Quoted 7 lines
If you're talking the issue I
think you're talking about, then I discovered it when doing stuff that most
people probably wouldn't bother with - evidenced that I hadn't done it before
either.
Actually, we do expect the default scheduler not to crash if one creates a cpupool.
Not that there hasn't been similar bug in Credit1, while it was the default (check `git log' :-/). But what this all means is that we need to do better at testing these things, e.g., finally adding cpupool and CPU online/offline testing to OSSTest.
Anyway, the bugfix is in now. :-)
Quoted 8 lines
I take peoples word on the performance +/- of a few percent here and there -
so if its easier to maintain and better code, then yeah - it makes sense to
move on with it. I certainly haven't found any normal use cases that would
lead me to object to this.
Great, and thanks again for the feedback!
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Software Engineer @ SUSE https://www.suse.com/
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 8 lines · Dario Faggioli
Quoted 7 lines · Steven Haigh
Quoted 5 lines · AEST Jan Beulich
Quoted 1 line
Quoted 1 line
Quoted 1 line
And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.
Quoted 6 lines
Quoted 4 lines
Even though my opinion probably isn't very heavy on this matter, I've used
credit2 exclusively for a considerable time.Well, this is really interesting and useful to know. Can I ask what your typical workload is (if any), and how are things going?
I don't really have a 'typical' workload. There's mail servers, web servers, DNS, shell boxes, all kinds of varied stuff. I haven't had any noticeable performance issues that I could look at and say "X is different".
Quoted 10 lines
Quoted 7 lines
If you're talking the issue I
think you're talking about, then I discovered it when doing stuff that most
people probably wouldn't bother with - evidenced that I hadn't done it before
either.Actually, we do expect the default scheduler not to crash if one creates a cpupool.
Quoted 6 lines
Not that there hasn't been similar bug in Credit1, while it was the default (check `git log' :-/). But what this all means is that we need to do better at testing these things, e.g., finally adding cpupool and CPU online/offline testing to OSSTest.
Anyway, the bugfix is in now. :-)
Agreed. I haven't had a chance to test that patch of yet - as I need to reconfigure the IPMI BMC to recover from a hard crash. Its tucked away in a rack out of sight and mind - so this isn't quite straight forward - but possible if I know I'm going to try and crash it :)
Quoted 12 lines
Quoted 8 lines
I take peoples word on the performance +/- of a few percent here and there -
so if its easier to maintain and better code, then yeah - it makes sense to
move on with it. I certainly haven't found any normal use cases that would
lead me to object to this.Great, and thanks again for the feedback!
Regards,
Dario
Signature · 9 lines
Steven Haigh ? [email protected] ? http://www.crc.id.au ? +61 (3) 9001 6090 ? 0412 935 897 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
I fully agree with and second George's reasoning, and feel 100% like
providing my:Acked-by Dario Faggioli <[email protected]>And that's despite "sched=credit2 crashes system when using cpupools"?
Well, considering that, as you say, there's a fix on the mailing list already, yes.
And I'm guessing George must feel kind of the same, as he sent this patch, before reviewing/committing the other.
In fact, the sooner we switch, the earlier OSSTest will start use Credit2 for everything, and show us whether or not there are other issues. And, speaking about that specific bug, OSSTest does not [sic!] test cpupools, so it's not even like it'll block pushes.
That being said...
Quoted 4 lines
While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.
... I'm fine if maintainers/committers feels like waiting for the other patch to go in. The order in which they're committed, is something I have no control on. I don't think it's strictly necessary for this patch and the fix to hit the repo in a particular order. But if others do, no problem.
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Software Engineer @ SUSE https://www.suse.com/
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
A message that is not in the archive (sent off-list or not yet received): 448fbac9df715f260587016?=
[email protected]
Quoted 8 lines · Dario Faggioli
Quoted 7 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
I fully agree with and second George's reasoning, and feel 100% like
providing my:Acked-by Dario Faggioli <[email protected]>
Quoted 8 lines
Quoted 3 lines
And that's despite "sched=credit2 crashes system when using cpupools"?
Well, considering that, as you say, there's a fix on the mailing list already, yes.
And I'm guessing George must feel kind of the same, as he sent this patch, before reviewing/committing the other.
Well, also because our team had just done our retrospective and noted the state of credit2 (and I think Ian brought it up on IRC). It was either "do it now" or "put it on a list to be done soon"; and my "list of things to be done soon" is already too long. :-)
Quoted 11 lines
In fact, the sooner we switch, the earlier OSSTest will start use Credit2 for everything, and show us whether or not there are other issues. And, speaking about that specific bug, OSSTest does not [sic!] test cpupools, so it's not even like it'll block pushes.
That being said...
Quoted 4 lines
While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.
Quoted 5 lines
... I'm fine if maintainers/committers feels like waiting for the other patch to go in. The order in which they're committed, is something I have no control on. I don't think it's strictly necessary for this patch and the fix to hit the repo in a particular order. But if others do, no problem.
I feel the same way: I don't see any reason to delay, but I don't think it's worth pressing the point when it's just a few days.
-George
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 31 lines · George Dunlap
Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).
Why change the default?
The code is better: more predictable, less jitter, easier to determine how modifications will affect overall behavior, easier in the future to make load-balancing behavior more subtle (e.g., taking into account the cost of powering up extra cores, &c).
Overall performance compared to Credit1 is somewhat of a mixed bag. Unfortunately most of what I have are tests using XenServer's internal perf testing system, so I can't share the raw data (via links anyway).
Here is a summary of data from an internal e-mail Dario sent in the past:
* DVDbench: On underloaded systems, credit2 outperformed credit1 by about 4%. On overloaded systems, credit2 underperformed by about 3%.
* On a range of tests (unixbench, lmbench, &c), credit and credit2 perform within 5% of each other (up and down).
* Credit2 fairly consistently beats credit for TCP-style workloads.
* Credit2 is sometimes equal to, sometimes 5-15% worse than, credit for synthetic CPU workloads (e.g., Dhrystone).
* On LoginVSI, credit2 fairly consistently outperforms credit by about 10%.
Quoted 8 lines
Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.
Signed-off-by George Dunlap <[email protected]>
The fix for the migration issue has passed the push gate; I plan on re-applying this change again (un-reverting the revert) Monday unless I hear any objections.
-George
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel