Skip to contentSkip to conversation

Make credit2 the default scheduler

PATCH In review George Dunlap 6 participants 11 messages Sep 13, 2018 – Oct 19, 2018 3 messages not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Make credit2 the default scheduler

Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).

Why change the default?

The code is better: more predictable, less jitter, easier to determine how modifications will affect overall behavior, easier in the future to make load-balancing behavior more subtle (e.g., taking into account the cost of powering up extra cores, &c).

Overall performance compared to Credit1 is somewhat of a mixed bag. Unfortunately most of what I have are tests using XenServer's internal perf testing system, so I can't share the raw data (via links anyway).

Here is a summary of data from an internal e-mail Dario sent in the past:

* DVDbench: On underloaded systems, credit2 outperformed credit1 by about 4%. On overloaded systems, credit2 underperformed by about 3%.

* On a range of tests (unixbench, lmbench, &c), credit and credit2 perform within 5% of each other (up and down).

* Credit2 fairly consistently beats credit for TCP-style workloads.

* Credit2 is sometimes equal to, sometimes 5-15% worse than, credit for synthetic CPU workloads (e.g., Dhrystone).

* On LoginVSI, credit2 fairly consistently outperforms credit by about 10%.

Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.

---
CC: Ian Jackson <[email protected]>
CC: Wei Liu <[email protected]>
CC: Andrew Cooper <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Tim Deegan <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Konrad Wilk <[email protected]>
CC: Dario Faggioli <[email protected]>

---
 xen/common/Kconfig | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
xen/common/Kconfig +2 -2
index 1a6d6281c1..3a1c585d25 100644
@@ -215,7 +215,7 @@ config SCHED_NULL
 
 choice
 	prompt "Default Scheduler?"
-	default SCHED_CREDIT_DEFAULT
+	default SCHED_CREDIT2_DEFAULT
 
 	config SCHED_CREDIT_DEFAULT
 		bool "Credit Scheduler" if SCHED_CREDIT
@@ -236,7 +236,7 @@ config SCHED_DEFAULT
 	default "rtds" if SCHED_RTDS_DEFAULT
 	default "arinc653" if SCHED_ARINC653_DEFAULT
 	default "null" if SCHED_NULL_DEFAULT
-	default "credit"
+	default "credit2"
 
 endmenu
 
Signature · 7 lines
2.18.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Acked-by on v1 1/1
#1 lore raw
Quoted 5 lines · George Dunlap

Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).

[..]

Quoted 9 lines

Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.

After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)

I fully agree with and second George's reasoning, and feel 100% like providing my:

Of course, although we thing the scheduler is ready, the more testing/profiling/benchmarking/... we get, the better.

So, everyone, don't be shy at throwing any kind of things against it, and report what happens, so we can fix the problem (or revert to Credit!) for 4.12

Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 16 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
Quoted 5 lines · George Dunlap

Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).

[..]

Quoted 9 lines

Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.

Quoted 7 lines

After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)

I fully agree with and second George's reasoning, and feel 100% like providing my:

And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
Quoted 5 lines · George Dunlap

Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).

[..]

Quoted 9 lines

Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.

After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)

Quoted 7 lines
Quoted 4 lines

I fully agree with and second George's reasoning, and feel 100% like providing my:

And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for

Urgh, this is a really nasty bug.

Quoted 2 lines

much longer, in particular with there already being a fix available from you I think that one should go in before the one here.

+1.

Wei.

Quoted 4 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
Quoted 28 lines · Wei Liu
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
Quoted 5 lines · George Dunlap

Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).

[..]

Quoted 9 lines

Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.

After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)

Quoted 7 lines
Quoted 4 lines

I fully agree with and second George's reasoning, and feel 100% like providing my:

And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for

Quoted 6 lines

Urgh, this is a really nasty bug.

Quoted 2 lines

much longer, in particular with there already being a fix available from you I think that one should go in before the one here.

+1.

The other fix is in. I'll wait a few days and push this change, if there are no objections.

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
In reply to Jan Beulich
Quoted 22 lines · AEST Jan Beulich
Quoted 22 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:
Quoted 5 lines · George Dunlap

Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).

[..]

Quoted 8 lines

Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.

After all the effort we've spent on this, I'm really, really happy to see this (trying to) happen. Thanks for sending the patch. :-)

I fully agree with and second George's reasoning, and feel 100% like providing my:

Quoted 6 lines
Quoted 1 line

And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.

Even though my opinion probably isn't very heavy on this matter, I've used credit2 exclusively for a considerable time. If you're talking the issue I think you're talking about, then I discovered it when doing stuff that most people probably wouldn't bother with - evidenced that I hadn't done it before either.

I take peoples word on the performance +/- of a few percent here and there - so if its easier to maintain and better code, then yeah - it makes sense to move on with it. I certainly haven't found any normal use cases that would lead me to object to this.

Signature · 4 lines
Steven Haigh

📧 [email protected]       💻 https://www.crc.id.au
📞 +61 (3) 9001 6090    📱 0412 935 897

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
Quoted 11 lines · Steven Haigh
Quoted 5 lines · AEST Jan Beulich
Quoted 1 line
Quoted 1 line
Quoted 1 line

And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.

Even though my opinion probably isn't very heavy on this matter, I've used
credit2 exclusively for a considerable time.

Well, this is really interesting and useful to know. Can I ask what your typical workload is (if any), and how are things going?

Quoted 7 lines

If you're talking the issue I
think you're talking about, then I discovered it when doing stuff that most
people probably wouldn't bother with - evidenced that I hadn't done it before
either.

Actually, we do expect the default scheduler not to crash if one creates a cpupool.

Not that there hasn't been similar bug in Credit1, while it was the default (check `git log' :-/). But what this all means is that we need to do better at testing these things, e.g., finally adding cpupool and CPU online/offline testing to OSSTest.

Anyway, the bugfix is in now. :-)

Quoted 8 lines

I take peoples word on the performance +/- of a few percent here and there -
so if its easier to maintain and better code, then yeah - it makes sense to
move on with it. I certainly haven't found any normal use cases that would
lead me to object to this.

Great, and thanks again for the feedback!

Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw
Quoted 8 lines · Dario Faggioli
Quoted 7 lines · Steven Haigh
Quoted 5 lines · AEST Jan Beulich
Quoted 1 line
Quoted 1 line
Quoted 1 line

And that's despite "sched=credit2 crashes system when using cpupools"? While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.

Quoted 6 lines
Quoted 4 lines

Even though my opinion probably isn't very heavy on this matter, I've used
credit2 exclusively for a considerable time.

Well, this is really interesting and useful to know. Can I ask what your typical workload is (if any), and how are things going?

I don't really have a 'typical' workload. There's mail servers, web servers, DNS, shell boxes, all kinds of varied stuff. I haven't had any noticeable performance issues that I could look at and say "X is different".

Quoted 10 lines
Quoted 7 lines

If you're talking the issue I
think you're talking about, then I discovered it when doing stuff that most
people probably wouldn't bother with - evidenced that I hadn't done it before
either.

Actually, we do expect the default scheduler not to crash if one creates a cpupool.

Quoted 6 lines

Not that there hasn't been similar bug in Credit1, while it was the default (check `git log' :-/). But what this all means is that we need to do better at testing these things, e.g., finally adding cpupool and CPU online/offline testing to OSSTest.

Anyway, the bugfix is in now. :-)

Agreed. I haven't had a chance to test that patch of yet - as I need to reconfigure the IPMI BMC to recover from a hard crash. Its tucked away in a rack out of sight and mind - so this isn't quite straight forward - but possible if I know I'm going to try and crash it :)

Quoted 12 lines
Quoted 8 lines

I take peoples word on the performance +/- of a few percent here and there -
so if its easier to maintain and better code, then yeah - it makes sense to
move on with it. I certainly haven't found any normal use cases that would
lead me to object to this.

Great, and thanks again for the feedback!

Regards,
Dario

Signature · 9 lines
Steven Haigh

? [email protected]     ? http://www.crc.id.au
? +61 (3) 9001 6090    ? 0412 935 897

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 10 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:

I fully agree with and second George's reasoning, and feel 100% like
providing my:

And that's despite "sched=credit2 crashes system when using cpupools"?

Well, considering that, as you say, there's a fix on the mailing list already, yes.

And I'm guessing George must feel kind of the same, as he sent this patch, before reviewing/committing the other.

In fact, the sooner we switch, the earlier OSSTest will start use Credit2 for everything, and show us whether or not there are other issues. And, speaking about that specific bug, OSSTest does not [sic!] test cpupools, so it's not even like it'll block pushes.

That being said...

Quoted 4 lines

While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.

... I'm fine if maintainers/committers feels like waiting for the other patch to go in. The order in which they're committed, is something I have no control on. I don't think it's strictly necessary for this patch and the fix to hit the repo in a particular order. But if others do, no problem.

Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

A message that is not in the archive (sent off-list or not yet received): 448fbac9df715f260587016?= [email protected]

#11 lore raw
Quoted 8 lines · Dario Faggioli
Quoted 7 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 13.09.18 at 18:51, <[email protected]> wrote:

I fully agree with and second George's reasoning, and feel 100% like
providing my:

Quoted 8 lines
Quoted 3 lines

And that's despite "sched=credit2 crashes system when using cpupools"?

Well, considering that, as you say, there's a fix on the mailing list already, yes.

And I'm guessing George must feel kind of the same, as he sent this patch, before reviewing/committing the other.

Well, also because our team had just done our retrospective and noted the state of credit2 (and I think Ian brought it up on IRC). It was either "do it now" or "put it on a list to be done soon"; and my "list of things to be done soon" is already too long. :-)

Quoted 11 lines

In fact, the sooner we switch, the earlier OSSTest will start use Credit2 for everything, and show us whether or not there are other issues. And, speaking about that specific bug, OSSTest does not [sic!] test cpupools, so it's not even like it'll block pushes.

That being said...

Quoted 4 lines

While I agree that we shouldn't delay the switch for much longer, in particular with there already being a fix available from you I think that one should go in before the one here.

Quoted 5 lines

... I'm fine if maintainers/committers feels like waiting for the other patch to go in. The order in which they're committed, is something I have no control on. I don't think it's strictly necessary for this patch and the fix to hit the repo in a particular order. But if others do, no problem.

I feel the same way: I don't see any reason to delay, but I don't think it's worth pressing the point when it's just a few days.

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw
In reply to George Dunlap
Quoted 31 lines · George Dunlap

Credit2 was declared "supported" in 4.8, and as of 4.10 had two other critical features implemented (soft affinity / NUMA and caps).

Why change the default?

The code is better: more predictable, less jitter, easier to determine how modifications will affect overall behavior, easier in the future to make load-balancing behavior more subtle (e.g., taking into account the cost of powering up extra cores, &c).

Overall performance compared to Credit1 is somewhat of a mixed bag. Unfortunately most of what I have are tests using XenServer's internal perf testing system, so I can't share the raw data (via links anyway).

Here is a summary of data from an internal e-mail Dario sent in the past:

* DVDbench: On underloaded systems, credit2 outperformed credit1 by about 4%. On overloaded systems, credit2 underperformed by about 3%.

* On a range of tests (unixbench, lmbench, &c), credit and credit2 perform within 5% of each other (up and down).

* Credit2 fairly consistently beats credit for TCP-style workloads.

* Credit2 is sometimes equal to, sometimes 5-15% worse than, credit for synthetic CPU workloads (e.g., Dhrystone).

* On LoginVSI, credit2 fairly consistently outperforms credit by about 10%.

Quoted 8 lines

Credit2, like credit, has a number of workloads / setups for which performance could be improved. Personally I think networking and partially-loaded systems is going to be more representative of what Xen is actually used for; so I think credit2 is on the whole the better scheduler to use by default. And in any case, making those improvements on credit2 will be easier than on credit.

The fix for the migration issue has passed the push gate; I plan on re-applying this change again (un-reverting the revert) Monday unless I hear any objections.

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel