Skip to contentSkip to conversation

xen: credit2: fix vcpu starvation due to too few credits

PATCH Superseded Dario Faggioli 6 participants 21 messages Mar 12, 2020 – Mar 13, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover xen: credit2: fix vcpu starvation due to too few credits

Hello everyone,

There have been reports of a Credit2 issue due to which vCPUs where being starved, to the point that guest kernel would complain or even crash.

See the following xen-users and xen-devel threads:
https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00018.html
https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00015.html
https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg01158.html

I did some investigations, and figured out that the vCPUs in question are not scheduled for long time intervals because they somehow manage to be given an amount of credits which is less than the credit the idle vCPU has.

An example of this situation is shown here. In fact, we can see d0v1 sitting in the runqueue while all the CPUs are idle, as it has -1254238270 credits, which is smaller than -2^30 = −1073741824:

    (XEN) Runqueue 0:
    (XEN)   ncpus              = 28
    (XEN)   cpus               = 0-27
    (XEN)   max_weight         = 256
    (XEN)   pick_bias          = 22
    (XEN)   instload           = 1
    (XEN)   aveload            = 293391 (~111%)
    (XEN)   idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
    (XEN)   tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
    (XEN)   fully idle cores: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
    [...]
    (XEN) Runqueue 0:
    (XEN) CPU[00] runq=0, sibling=00,..., core=00,...
    (XEN) CPU[01] runq=0, sibling=00,..., core=00,...
    [...]
    (XEN) CPU[26] runq=0, sibling=00,..., core=00,...
    (XEN) CPU[27] runq=0, sibling=00,..., core=00,...
    (XEN) RUNQ:
    (XEN)     0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 (~100%)

This happens bacause --although very rarely-- vCPUs are allowed to execute for much more than the scheduler would want them to.

For example, I have a trace showing that csched2_schedule() is invoked at t=57970746155ns. At t=57970747658ns (+1503ns) the s_timer is set to fire at t=57979485083ns, i.e., 8738928ns in future. That's because credit of snext is exactly that 8738928ns. Then, what I see is that the next call to burn_credits(), coming from csched2_schedule() for the same vCPU happens at t=60083283617ns. That is *a lot* (2103798534ns) later than when we expected and asked. Of course, that also means that delta is 2112537462ns, and therefore credits will sink to -2103798534!

Also, to the best of my current knowledge, this does not look like Credit2 related, as I've observed it when running with Credit1 as well. I personally don't think it would be scheduling related, in general, but I need to do more investigation to be sure about that (and/or to figure out what the real root cause is).

The reason why Credit2 is affected much more than Credit1 is because of how time accounting is done. Basically, there's very rudimental time accounting in Credit1, which is a very bad thing, IMO, but indeed that is also what prevented for this issue to cause severe stalls.

One more thing is that Credit2 gives -2^30 credits to the idle vCPU, which was considered to be low enough, which is true. But it's not a robust choice, should an issue like the one we're discussing occur, which is happening. :-) Therefore, I think we should lower the credits of the idle vCPU to the minimum possible value, so that even under whatever unusual or weird or buggy situations like this one, we will never pick idle instead of an actual vCPU that is ready to run.

This is what is done in the first patch of this series. This is a robustness improvement and a fix (or at least the best way we can deal with the it within the scheduler) for the issue at hand. It therefore should be backported.

While looking into this, I also have found out that there is an actual bug in Credit2 code. It is something I introduced myself with commit 5e4b4199667b9 ("xen: credit2: only reset credit on reset condition"). In fact, while it was and still is a good idea to avoid resetting credits too often, the implementation of this was just wrong.

A fix for this bug is what is contained in patch 2. And it also should be backported.

Note that patch 2 alone was also already mitigating the stall/starvation issue quite substantially. Nevertheless, the proper fix for the issue itself is making Credit2 more robust against similar problem, as done in patch 1, while this other bug just happens to be something which interact with the sympthoms.

This to say that, although both patches will be bugported, asboth are actual bugfixes, if there is the need to apply something "in emergency" to fix the starvation problem, applying only patch 1 is enough.

Thanks and Regards
---
Dario Faggioli (2):
xen: credit2: avoid vCPUs to ever reach lower credits than idle
xen: credit2: fix credit reset happening too few times

 tools/xentrace/formats     |    2 +-
 tools/xentrace/xenalyze.c  |    8 +++-----
 xen/common/sched/credit2.c |   32 ++++++++++++++------------------
 3 files changed, 18 insertions(+), 24 deletions(-)

--
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#1 lore raw

PATCH 1/2 xen: credit2: avoid vCPUs to ever reach lower credits than idle

There have been report of stalls of guest vCPUs, when Credit2 was used. It seemed like these vCPUs were not getting scheduled for very long time, even under light load conditions (e.g., during dom0 boot).

Investigations led to the discovery that --although rarely-- it can happen that a vCPU manages to run for very long timeslices. In Credit2, this means that, when runtime accounting happens, the vCPU will lose a large quantity of credits. This in turn may lead to the vCPU having less credits than the idle vCPUs (-2^30). At this point, the scheduler will pick the idle vCPU, instead of the ready to run vCPU, for a few "epochs", which often times is enough for the guest kernel to think the vCPU is not responding and crashing.

An example of this situation is shown here. In fact, we can see d0v1 sitting in the runqueue while all the CPUs are idle, as it has -1254238270 credits, which is smaller than -2^30 = −1073741824:

    (XEN) Runqueue 0:
    (XEN)   ncpus              = 28
    (XEN)   cpus               = 0-27
    (XEN)   max_weight         = 256
    (XEN)   pick_bias          = 22
    (XEN)   instload           = 1
    (XEN)   aveload            = 293391 (~111%)
    (XEN)   idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
    (XEN)   tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
    (XEN)   fully idle cores: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
    [...]
    (XEN) Runqueue 0:
    (XEN) CPU[00] runq=0, sibling=00,..., core=00,...
    (XEN) CPU[01] runq=0, sibling=00,..., core=00,...
    [...]
    (XEN) CPU[26] runq=0, sibling=00,..., core=00,...
    (XEN) CPU[27] runq=0, sibling=00,..., core=00,...
    (XEN) RUNQ:
    (XEN)     0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 (~100%)

We certainly don't want, under any circumstance, this to happen. Therefore, let's use INT_MIN for the credits of the idle vCPU, in Credit2, to be sure that no vCPU can get below that value.

NOTE: investigations have been done about _how_ it is possible for a vCPU to execute for so long that its credits becomes so low. While still not completely clear, there are evidence that:
- it only happens very rarely
- it appears to be both machine and workload specific
- it does not look to be a Credit2 (e.g., as it happens when running with Credit1 as well) issue, or a scheduler issue

This patch makes Credit2 more robust to events like this, whatever the cause is, and should hence be backported (as far as possible).

---
Cc: George Dunlap <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Charles Arnold <[email protected]>
Cc: Sarah Newman <[email protected]>
---
I will provide the backports myself, at least for 4.13 and 4.12.x (and feel free to ask for more).
---
For Sarah, looking back at the various threads, I am not quite sure whether you also experienced the issue and reported it. If yes, I'm happy to add a "Reported-by:" line about you too (or, if this is fine to go in, for this to be done while committing, if possible).

---
 xen/common/sched/credit2.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/sched/credit2.c +1 -1
index c7241944a8..5c0ab9cd05 100644
@@ -234,7 +234,7 @@
  * units does not consume credits, and it must be lower than whatever
  * amount of credit 'regular' unit would end up with.
  */
-#define CSCHED2_IDLE_CREDIT          (-(1U<<30))
+#define CSCHED2_IDLE_CREDIT          INT_MIN
 /*
  * Carryover: How much "extra" credit may be carried over after
  * a reset.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

Quoted 41 lines · Dario Faggioli

There have been report of stalls of guest vCPUs, when Credit2 was used. It seemed like these vCPUs were not getting scheduled for very long time, even under light load conditions (e.g., during dom0 boot).

Investigations led to the discovery that --although rarely-- it can happen that a vCPU manages to run for very long timeslices. In Credit2, this means that, when runtime accounting happens, the vCPU will lose a large quantity of credits. This in turn may lead to the vCPU having less credits than the idle vCPUs (-2^30). At this point, the scheduler will pick the idle vCPU, instead of the ready to run vCPU, for a few "epochs", which often times is enough for the guest kernel to think the vCPU is not responding and crashing.

An example of this situation is shown here. In fact, we can see d0v1 sitting in the runqueue while all the CPUs are idle, as it has -1254238270 credits, which is smaller than -2^30 = −1073741824:

    (XEN) Runqueue 0:
    (XEN)   ncpus              = 28
    (XEN)   cpus               = 0-27
    (XEN)   max_weight         = 256
    (XEN)   pick_bias          = 22
    (XEN)   instload           = 1
    (XEN)   aveload            = 293391 (~111%)
    (XEN)   idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
    (XEN)   tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
    (XEN)   fully idle cores: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
    [...]
    (XEN) Runqueue 0:
    (XEN) CPU[00] runq=0, sibling=00,..., core=00,...
    (XEN) CPU[01] runq=0, sibling=00,..., core=00,...
    [...]
    (XEN) CPU[26] runq=0, sibling=00,..., core=00,...
    (XEN) CPU[27] runq=0, sibling=00,..., core=00,...
    (XEN) RUNQ:
    (XEN)     0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 (~100%)

We certainly don't want, under any circumstance, this to happen. Therefore, let's use INT_MIN for the credits of the idle vCPU, in Credit2, to be sure that no vCPU can get below that value.

Quoted 7 lines

NOTE: investigations have been done about _how_ it is possible for a vCPU to execute for so long that its credits becomes so low. While still not completely clear, there are evidence that:
- it only happens very rarely
- it appears to be both machine and workload specific
- it does not look to be a Credit2 (e.g., as it happens when running with Credit1 as well) issue, or a scheduler issue

On what basis?

Everything reported to xen-devel appears to suggests it is a credit2 problem.  It doesn't manifest on versions of Xen before credit2 became the default, and switching back to credit1 appears to mitigate the problem.

Certainly as far as XenServer is concerned, we haven't seen symptoms like this in a decade of running credit1.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

Quoted 54 lines · Andrew Cooper
Quoted 41 lines · Dario Faggioli

There have been report of stalls of guest vCPUs, when Credit2 was used. It seemed like these vCPUs were not getting scheduled for very long time, even under light load conditions (e.g., during dom0 boot).

Investigations led to the discovery that --although rarely-- it can happen that a vCPU manages to run for very long timeslices. In Credit2, this means that, when runtime accounting happens, the vCPU will lose a large quantity of credits. This in turn may lead to the vCPU having less credits than the idle vCPUs (-2^30). At this point, the scheduler will pick the idle vCPU, instead of the ready to run vCPU, for a few "epochs", which often times is enough for the guest kernel to think the vCPU is not responding and crashing.

An example of this situation is shown here. In fact, we can see d0v1 sitting in the runqueue while all the CPUs are idle, as it has -1254238270 credits, which is smaller than -2^30 = −1073741824:

   (XEN) Runqueue 0:
   (XEN)   ncpus              = 28
   (XEN)   cpus               = 0-27
   (XEN)   max_weight         = 256
   (XEN)   pick_bias          = 22
   (XEN)   instload           = 1
   (XEN)   aveload            = 293391 (~111%)
   (XEN)   idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
   (XEN)   tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
   (XEN)   fully idle cores: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
   [...]
   (XEN) Runqueue 0:
   (XEN) CPU[00] runq=0, sibling=00,..., core=00,...
   (XEN) CPU[01] runq=0, sibling=00,..., core=00,...
   [...]
   (XEN) CPU[26] runq=0, sibling=00,..., core=00,...
   (XEN) CPU[27] runq=0, sibling=00,..., core=00,...
   (XEN) RUNQ:
   (XEN)     0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 (~100%)

We certainly don't want, under any circumstance, this to happen. Therefore, let's use INT_MIN for the credits of the idle vCPU, in Credit2, to be sure that no vCPU can get below that value.

Quoted 7 lines

NOTE: investigations have been done about _how_ it is possible for a vCPU to execute for so long that its credits becomes so low. While still not completely clear, there are evidence that:
- it only happens very rarely
- it appears to be both machine and workload specific
- it does not look to be a Credit2 (e.g., as it happens when running with Credit1 as well) issue, or a scheduler issue

Quoted 8 lines

On what basis?

Everything reported to xen-devel appears to suggests it is a credit2 problem. It doesn't manifest on versions of Xen before credit2 became the default, and switching back to credit1 appears to mitigate the problem.

Certainly as far as XenServer is concerned, we haven't seen symptoms like this in a decade of running credit1.

One reason could be because the symptoms are different. On credit1, credits and “priority” are separated; it’s not possible in credit1 for a vcpu to end up with a lower priority than the idle domain, and no matter how low the credits become, a vcpu will always end up with some “peers” at the same priority level, meaning it always has a chance at some cpu.

What Dario is saying (if I understand him correctly) is that the *proximate* cause (allowing a vcpu to have an effective priority of less than idle) is certainly credit2-only; but the *deeper* cause (vcpus racking up massive amounts of negative credit) is not.

-George
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

Quoted 7 lines · George Dunlap
Quoted 6 lines

On Mar 12, 2020, at 1:55 PM, Andrew Cooper <
[email protected]> wrote:

Certainly as far as XenServer is concerned, we haven't seen symptoms
like this in a decade of running credit1.

Quoted 7 lines

One reason could be because the symptoms are different. On credit1, credits and “priority” are separated; it’s not possible in credit1 for a vcpu to end up with a lower priority than the idle domain, and no matter how low the credits become, a vcpu will always end up with some “peers” at the same priority level, meaning it always has a chance at some cpu.

Indeed. Under many respects, Credit1 is basically a slightly more powerful variant of Round-Robin.

The actual time vCPUs spend executing influences the scheduling decisions a lot less than in Credit2 (well, it influences the scheduling decisions a lot less than in any scheduler of any OS or hypervisor conceived and implemented in the last decade, or maybe even more).

As a matter of fact, this property happens to have shield it from the nastiest effects of the issue at hand.

Quoted 5 lines

What Dario is saying (if I understand him correctly) is that the *proximate* cause (allowing a vcpu to have an effective priority of less than idle) is certainly credit2-only; but the *deeper* cause (vcpus racking up massive amounts of negative credit) is not.

Yes, that's exactly what I meant. :-)

Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

In reply to Andrew Cooper
Quoted 13 lines · Andrew Cooper
Quoted 11 lines · Dario Faggioli

NOTE: investigations have been done about _how_ it is possible for a
vCPU to execute for so long that its credits becomes so low. While still
not completely clear, there are evidence that:
- it only happens very rarely
- it appears to be both machine and workload specific
- it does not look to be a Credit2 (e.g., as it happens when running
with Credit1 as well) issue, or a scheduler issue

Quoted 2 lines

On what basis?

On the basis that I have traced execution times of vCPUs, when running on both Credit1 and Credit2, on a system where I can sort-of reproduce the issue, and I see them being able to execute for ginormous timeslices, with both scheduler.

That's what this paragraph above, plus similar ones in the cover letter ,were supposed to mean. If it is not clear enough, sorry.

Quoted 6 lines

Everything reported to xen-devel appears to suggests it is a credit2 problem. It doesn't manifest on versions of Xen before credit2 became
the default, and switching back to credit1 appears to mitigate the problem.

Yes, because even if the issue is there in both cases, when you use Credit2 it manifests as vCPUs being starved (because of the "they end up having less priority than the idle vCPU" thing). When you use Credit1, this does not happen, because time accounting and prioritization are done differently there.

Switching to Credit1, at least on the box where I could reproduce the problem, did not make the issue that the vCPUs run for way too long disappear. It's the symptoms that this issue cause that are different for the two schedulers.

Quoted 3 lines

Certainly as far as XenServer is concerned, we haven't seen symptoms like this in a decade of running credit1.

I am only able to reproduce this issue on one box, and not really 100% deterministically. I have never noticed anything like this in the past few years, on Credit2. At least, nothing as severe as this (or we would have noticed, which is what, in fact, has happened!).

With Credit1, even if the issue is there, you won't notice it in this form, se we really can't know.

Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Acked-by on v1 1/2
#6 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

In reply to Dario Faggioli
Quoted 53 lines · Dario Faggioli

There have been report of stalls of guest vCPUs, when Credit2 was used. It seemed like these vCPUs were not getting scheduled for very long time, even under light load conditions (e.g., during dom0 boot).

Investigations led to the discovery that --although rarely-- it can happen that a vCPU manages to run for very long timeslices. In Credit2, this means that, when runtime accounting happens, the vCPU will lose a large quantity of credits. This in turn may lead to the vCPU having less credits than the idle vCPUs (-2^30). At this point, the scheduler will pick the idle vCPU, instead of the ready to run vCPU, for a few "epochs", which often times is enough for the guest kernel to think the vCPU is not responding and crashing.

An example of this situation is shown here. In fact, we can see d0v1 sitting in the runqueue while all the CPUs are idle, as it has -1254238270 credits, which is smaller than -2^30 = −1073741824:

   (XEN) Runqueue 0:
   (XEN)   ncpus              = 28
   (XEN)   cpus               = 0-27
   (XEN)   max_weight         = 256
   (XEN)   pick_bias          = 22
   (XEN)   instload           = 1
   (XEN)   aveload            = 293391 (~111%)
   (XEN)   idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
   (XEN)   tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
   (XEN)   fully idle cores: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
   [...]
   (XEN) Runqueue 0:
   (XEN) CPU[00] runq=0, sibling=00,..., core=00,...
   (XEN) CPU[01] runq=0, sibling=00,..., core=00,...
   [...]
   (XEN) CPU[26] runq=0, sibling=00,..., core=00,...
   (XEN) CPU[27] runq=0, sibling=00,..., core=00,...
   (XEN) RUNQ:
   (XEN)     0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 (~100%)

We certainly don't want, under any circumstance, this to happen. Therefore, let's use INT_MIN for the credits of the idle vCPU, in Credit2, to be sure that no vCPU can get below that value.

NOTE: investigations have been done about _how_ it is possible for a vCPU to execute for so long that its credits becomes so low. While still not completely clear, there are evidence that:
- it only happens very rarely
- it appears to be both machine and workload specific
- it does not look to be a Credit2 (e.g., as it happens when running with Credit1 as well) issue, or a scheduler issue

Quoted 6 lines

This patch makes Credit2 more robust to events like this, whatever the cause is, and should hence be backported (as far as possible).

Nit: The reported-by’s should be before the SoB (i.e., tags roughly in time order).

I think this is a good change to make the algorithm more robust, so:

But it seems like allowing a guest to rack up -2^63 credits is still a bad thing, and it would be nice to have some other backstop / reset mechanism. But I guess to have an effective mechanism of that sort we’d want to understand how it happened in the first place.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

Quoted 6 lines · George Dunlap
Quoted 6 lines

On Mar 12, 2020, at 1:44 PM, Dario Faggioli <[email protected]> wrote:

This patch makes Credit2 more robust to events like this, whatever the cause is, and should hence be backported (as far as possible).

Quoted 7 lines
Quoted 3 lines

Nit: The reported-by’s should be before the SoB (i.e., tags roughly in time order).

Ah, right! :-(

Quoted 8 lines

I think this is a good change to make the algorithm more robust, so:

But it seems like allowing a guest to rack up -2^63 credits is still a bad thing, and it would be nice to have some other backstop / reset mechanism.

I agree. FWIW, this is way it took me a while to get to the bottom of this. I was assuming it was *entirely* a Credit2 specific issue (caused by, e.g., something like I found and fixed with patch 2).

When I noticed that things were not exactly like that, I also realized that we at least need to prevent --under any circumstance-- that idle vCPUs are preferred over "regular" vCPUs, and even if I did consider approaches like "compacting" the credits dynamic, I went straight for the INT_MIN approach.

Considering how this thread is going, I guess we should actually push this further, and limit the "credit swing".

Thanks and Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

In reply to Dario Faggioli

xen/common/sched/credit2.c line 237

Quoted 8 lines · Dario Faggioli
xen/common/sched/credit2.c +1 -1
@@ -234,7 +234,7 @@
  * units does not consume credits, and it must be lower than whatever
  * amount of credit 'regular' unit would end up with.
  */
-#define CSCHED2_IDLE_CREDIT          (-(1U<<30))
+#define CSCHED2_IDLE_CREDIT          INT_MIN

The title saying "lower than", is "equal" actually fine? Looking at e.g. runq_insert() I'm getting the impression it's not.

Looking at t2c_update() I'm also getting the impression that there's UB when the subtraction underflows. After all, if -1 << 30 wasn't small enough a value, I don't see why -1 << 31 would be.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

Quoted 13 lines · Jan Beulich

xen/common/sched/credit2.c line 237

Quoted 8 lines · Dario Faggioli
xen/common/sched/credit2.c +1 -1
@@ -234,7 +234,7 @@
   * units does not consume credits, and it must be lower than whatever
   * amount of credit 'regular' unit would end up with.
   */
-#define CSCHED2_IDLE_CREDIT          (-(1U<<30))
+#define CSCHED2_IDLE_CREDIT          INT_MIN

The title saying "lower than", is "equal" actually fine? Looking at e.g. runq_insert() I'm getting the impression it's not.

Quoted 4 lines

Looking at t2c_update() I'm also getting the impression that there's UB when the subtraction underflows. After all, if -1 << 30 wasn't small enough a value, I don't see why -1 << 31 would be.

Yes, I'd limit svc->credit in t2c_update() to CSCHED2_IDLE_CREDIT+1

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

Quoted 15 lines · Jürgen Groß
Quoted 14 lines · Jan Beulich

xen/common/sched/credit2.c line 237

Quoted 9 lines · Dario Faggioli
xen/common/sched/credit2.c +0 -0
@@ -234,7 +234,7 @@
   * units does not consume credits, and it must be lower than

whatever
* amount of credit 'regular' unit would end up with. */

xen/common/sched/credit2.c +1 -1
-#define CSCHED2_IDLE_CREDIT          (-(1U<<30))
+#define CSCHED2_IDLE_CREDIT          INT_MIN

The title saying "lower than", is "equal" actually fine? Looking at e.g. runq_insert() I'm getting the impression it's not.

Quoted 7 lines
Quoted 4 lines

Looking at t2c_update() I'm also getting the impression that there's UB when the subtraction underflows. After all, if -1 << 30 wasn't small enough a value, I don't see why -1 << 31 would be.

Yes, I'd limit svc->credit in t2c_update() to CSCHED2_IDLE_CREDIT+1

Right, after having spoken with George as well, I guess I'll go for something like that (by imposing some boundaries to val, in that function, I think).

Thanks and Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#11 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

In reply to Jan Beulich
Quoted 14 lines · Jan Beulich

xen/common/sched/credit2.c line 237

Quoted 9 lines · Dario Faggioli
xen/common/sched/credit2.c +0 -0
@@ -234,7 +234,7 @@
  * units does not consume credits, and it must be lower than

whatever
* amount of credit 'regular' unit would end up with. */

xen/common/sched/credit2.c +1 -1
-#define CSCHED2_IDLE_CREDIT          (-(1U<<30))
+#define CSCHED2_IDLE_CREDIT          INT_MIN

The title saying "lower than", is "equal" actually fine? Looking at e.g. runq_insert() I'm getting the impression it's not.

In Credit2, we don't have the idle vCPUs in the runqueue. So we will never compare non-idle with idle while inserting.

But this is a good point, in general, and I think we may need to turn the ">"
in a ">=" in runq_candidate().

Quoted 5 lines

Looking at t2c_update() I'm also getting the impression that there's UB when the subtraction underflows. After all, if -1 << 30 wasn't small enough a value, I don't see why -1 << 31 would be.

Mmm... not sure I am getting. Are you suggesting we should apply a cap to val?

If yes, this looks like an issue independent from what the value of CSCHED2_IDLE_CREDIT is, but yeah, we can do that. Or am I missing something?

Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw

Re: [Xen-devel] [PATCH 1/2] xen: credit2: avoid vCPUs to ever reach lower credits than idle

Quoted 6 lines · Dario Faggioli
Quoted 5 lines · Jan Beulich

Looking at t2c_update() I'm also getting the impression that there's UB when the subtraction underflows. After all, if -1 << 30 wasn't small enough a value, I don't see why -1 << 31 would be.

Quoted 6 lines

Mmm... not sure I am getting. Are you suggesting we should apply a cap to val?

If yes, this looks like an issue independent from what the value of CSCHED2_IDLE_CREDIT is, but yeah, we can do that. Or am I missing something?

Well, at the very least UB needs to be avoided, i.e. we shouldn't subtract when the result is not representable in "credit"'s type. But even beyond that there may be a need for the subtraction to actually act as if it was saturating at INT_MIN. (Similarly additions may need to saturate at INT_MAX, if there's any possibility for those values to grow large.)

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

PATCH 2/2 xen: credit2: fix credit reset happening too few times

In reply to Dario Faggioli

There is a bug in commit 5e4b4199667b9 ("xen: credit2: only reset credit on reset condition"). In fact, the aim of that commit was to make sure that we do not perform too many credit reset operations (which are not super cheap, and in an hot-path). But the check used to determine whether a reset is necessary was the wrong one.

In fact, knowing just that some vCPUs have been skipped, while traversing the runqueue (in runq_candidate()), is not enough. We need to check explicitly whether the first vCPU in the runqueue has a negative amount of credit.

Since a trace record is changed, this patch updates xentrace format file and xenalyze as well

This should be backported.

---
Cc: George Dunlap <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Charles Arnold <[email protected]>
Cc: Glen <[email protected]>
Cc: Tomas Mozes <[email protected]>
Cc: Sarah Newman <[email protected]>
---
About the Credit2 stall issue reported recently, and mentioned in patch 1 of this series. This second patch, alone, was already mitigating the issue quite substantially.

Still, the proper fix for the issue itself is patch 1, while this is a fix for a bug in the code, introduced with a previous change, which happens to help to cure the sympthoms of the problem at hand.

---
 tools/xentrace/formats     |    2 +-
 tools/xentrace/xenalyze.c  |    8 +++-----
 xen/common/sched/credit2.c |   30 +++++++++++++-----------------
 3 files changed, 17 insertions(+), 23 deletions(-)
tools/xentrace/formats +1 -1
index d6e7e3f800..8142d8889b 100644
@@ -67,7 +67,7 @@
 0x00022210  CPU%(cpu)d  %(tsc)d (+%(reltsc)8d)  csched2:load_check     [ lrq_id[16]:orq_id[16] = 0x%(1)08x, delta = %(2)d ]
 0x00022211  CPU%(cpu)d  %(tsc)d (+%(reltsc)8d)  csched2:load_balance   [ l_bavgload = 0x%(2)08x%(1)08x, o_bavgload = 0x%(4)08x%(3)08x, lrq_id[16]:orq_id[16] = 0x%(5)08x ]
 0x00022212  CPU%(cpu)d  %(tsc)d (+%(reltsc)8d)  csched2:pick_cpu       [ b_avgload = 0x%(2)08x%(1)08x, dom:vcpu = 0x%(3)08x, rq_id[16]:new_cpu[16] = %(4)d ]
-0x00022213  CPU%(cpu)d  %(tsc)d (+%(reltsc)8d)  csched2:runq_candidate [ dom:vcpu = 0x%(1)08x, credit = %(4)d, skipped_vcpus = %(3)d, tickled_cpu = %(2)d ]
+0x00022213  CPU%(cpu)d  %(tsc)d (+%(reltsc)8d)  csched2:runq_candidate [ dom:vcpu = 0x%(1)08x, credit = %(3)d, tickled_cpu = %(2)d ]
 0x00022214  CPU%(cpu)d  %(tsc)d (+%(reltsc)8d)  csched2:schedule       [ rq:cpu = 0x%(1)08x, tasklet[8]:idle[8]:smt_idle[8]:tickled[8] = %(2)08x ]
 0x00022215  CPU%(cpu)d  %(tsc)d (+%(reltsc)8d)  csched2:ratelimit      [ dom:vcpu = 0x%(1)08x, runtime = %(2)d ]
 0x00022216  CPU%(cpu)d  %(tsc)d (+%(reltsc)8d)  csched2:runq_cand_chk  [ dom:vcpu = 0x%(1)08x ]
tools/xentrace/xenalyze.c +3 -5
index aa894673ad..e555c572f7 100644
@@ -7856,14 +7856,12 @@ void sched_process(struct pcpu_info *p)
             if (opt.dump_all) {
                 struct {
                     unsigned vcpuid:16, domid:16;
-                    unsigned tickled_cpu, skipped;
+                    unsigned tickled_cpu;
                     int credit;
                 } *r = (typeof(r))ri->d;
 
-                printf(" %s csched2:runq_candidate d%uv%u, credit = %d, "
-                       "%u vcpus skipped, ",
-                       ri->dump_header, r->domid, r->vcpuid,
-                       r->credit, r->skipped);
+                printf(" %s csched2:runq_candidate d%uv%u, credit = %d, ",
+                       ri->dump_header, r->domid, r->vcpuid, r->credit);
                 if (r->tickled_cpu == (unsigned)-1)
                     printf("no cpu was tickled\n");
                 else
xen/common/sched/credit2.c +13 -17
index 5c0ab9cd05..20692cf8ef 100644
@@ -3221,8 +3221,7 @@ csched2_runtime(const struct scheduler *ops, int cpu,
 static struct csched2_unit *
 runq_candidate(struct csched2_runqueue_data *rqd,
                struct csched2_unit *scurr,
-               int cpu, s_time_t now,
-               unsigned int *skipped)
+               int cpu, s_time_t now)
 {
     struct list_head *iter, *temp;
     const struct sched_resource *sr = get_sched_res(cpu);
@@ -3230,8 +3229,6 @@ runq_candidate(struct csched2_runqueue_data *rqd,
     struct csched2_private *prv = csched2_priv(sr->scheduler);
     bool yield = false, soft_aff_preempt = false;
 
-    *skipped = 0;
-
     if ( unlikely(is_idle_unit(scurr->unit)) )
     {
         snext = scurr;
@@ -3325,12 +3322,9 @@ runq_candidate(struct csched2_runqueue_data *rqd,
                         (unsigned char *)&d);
         }
 
-        /* Only consider units that are allowed to run on this processor. */
+        /* Only consider vcpus that are allowed to run on this processor. */
         if ( !cpumask_test_cpu(cpu, svc->unit->cpu_hard_affinity) )
-        {
-            (*skipped)++;
             continue;
-        }
 
         /*
          * If an unit is meant to be picked up by another processor, and such
@@ -3339,7 +3333,6 @@ runq_candidate(struct csched2_runqueue_data *rqd,
         if ( svc->tickled_cpu != -1 && svc->tickled_cpu != cpu &&
              cpumask_test_cpu(svc->tickled_cpu, &rqd->tickled) )
         {
-            (*skipped)++;
             SCHED_STAT_CRANK(deferred_to_tickled_cpu);
             continue;
         }
@@ -3351,7 +3344,6 @@ runq_candidate(struct csched2_runqueue_data *rqd,
         if ( sched_unit_master(svc->unit) != cpu
              && snext->credit + CSCHED2_MIGRATE_RESIST > svc->credit )
         {
-            (*skipped)++;
             SCHED_STAT_CRANK(migrate_resisted);
             continue;
         }
@@ -3375,14 +3367,13 @@ runq_candidate(struct csched2_runqueue_data *rqd,
     {
         struct {
             unsigned unit:16, dom:16;
-            unsigned tickled_cpu, skipped;
+            unsigned tickled_cpu;
             int credit;
         } d;
         d.dom = snext->unit->domain->domain_id;
         d.unit = snext->unit->unit_id;
         d.credit = snext->credit;
         d.tickled_cpu = snext->tickled_cpu;
-        d.skipped = *skipped;
         __trace_var(TRC_CSCHED2_RUNQ_CANDIDATE, 1,
                     sizeof(d),
                     (unsigned char *)&d);
@@ -3414,7 +3405,6 @@ static void csched2_schedule(
     struct csched2_runqueue_data *rqd;
     struct csched2_unit * const scurr = csched2_unit(currunit);
     struct csched2_unit *snext = NULL;
-    unsigned int skipped_units = 0;
     bool tickled;
     bool migrated = false;
 
@@ -3492,7 +3482,7 @@ static void csched2_schedule(
         snext = csched2_unit(sched_idle_unit(sched_cpu));
     }
     else
-        snext = runq_candidate(rqd, scurr, sched_cpu, now, &skipped_units);
+        snext = runq_candidate(rqd, scurr, sched_cpu, now);
 
     /* If switching from a non-idle runnable unit, put it
      * back on the runqueue. */
@@ -3504,6 +3494,8 @@ static void csched2_schedule(
     /* Accounting for non-idle tasks */
     if ( !is_idle_unit(snext->unit) )
     {
+        int top_credit;
+
         /* If switching, remove this from the runqueue and mark it scheduled */
         if ( snext != scurr )
         {
@@ -3531,11 +3523,15 @@ static void csched2_schedule(
          *  2) no other unit with higher credits wants to run.
          *
          * Here, where we want to check for reset, we need to make sure the
-         * proper unit is being used. In fact, runqueue_candidate() may have
-         * not returned the first unit in the runqueue, for various reasons
+         * proper unit is being used. In fact, runq_candidate() may have not
+         * returned the first unit in the runqueue, for various reasons
          * (e.g., affinity). Only trigger a reset when it does.
          */
-        if ( skipped_units == 0 && snext->credit <= CSCHED2_CREDIT_RESET )
+        if ( list_empty(&rqd->runq) )
+            top_credit = snext->credit;
+        else
+            top_credit = max(snext->credit, runq_elem(rqd->runq.next)->credit);
+        if ( top_credit <= CSCHED2_CREDIT_RESET )
         {
             reset_credit(sched_cpu, now, snext);
             balance_load(ops, sched_cpu, now);

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#14 lore raw
In reply to Dario Faggioli

Thanks for looking into this, seems like a specially tricky issue to tackle!

On Thu, Mar 12, 2020 at 02:44:07PM +0100, Dario Faggioli wrote:
[...]

Quoted 8 lines

For example, I have a trace showing that csched2_schedule() is invoked at t=57970746155ns. At t=57970747658ns (+1503ns) the s_timer is set to fire at t=57979485083ns, i.e., 8738928ns in future. That's because credit of snext is exactly that 8738928ns. Then, what I see is that the next call to burn_credits(), coming from csched2_schedule() for the same vCPU happens at t=60083283617ns. That is *a lot* (2103798534ns) later than when we expected and asked. Of course, that also means that delta is 2112537462ns, and therefore credits will sink to -2103798534!

Which timer does this hardware use? DYK if there's some relation between the timer hardware used and the issue?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw
Quoted 3 lines · Roger Pau Monné

Thanks for looking into this, seems like a specially tricky issue to tackle!

It was tricky indeed! :-)

Quoted 17 lines · Dario Faggioli

[...]

Quoted 6 lines

For example, I have a trace showing that csched2_schedule() is invoked at
t=57970746155ns. At t=57970747658ns (+1503ns) the s_timer is set to fire at t=57979485083ns, i.e., 8738928ns in future. That's because credit
of snext is exactly that 8738928ns. Then, what I see is that the

Quoted 8 lines

next
call to burn_credits(), coming from csched2_schedule() for the same vCPU
happens at t=60083283617ns. That is *a lot* (2103798534ns) later than
when we expected and asked. Of course, that also means that delta is
2112537462ns, and therefore credits will sink to -2103798534!

Quoted 3 lines

Which timer does this hardware use? DYK if there's some relation between the timer hardware used and the issue?

Timers came to mind but I haven't checked yet.

FWIW, one thing I saw is that, without patches, my machine times out around...

[    2.364819] NET: Registered protocol family 16
[    2.368018] xen:grant_table: Grant tables using version 1 layout
[    2.372033] Grant table initialized
[    2.377115] ACPI: bus type PCI registered
[    2.380011] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    2.384660] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
[    2.388033] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
[    2.499080] PCI: Using configuration type 1 for base access
[    2.516768] ACPI: Added _OSI(Module Device)
[    2.524006] ACPI: Added _OSI(Processor Device)
[    2.536004] ACPI: Added _OSI(3.0 _SCP Extensions)
[    2.544003] ACPI: Added _OSI(Processor Aggregator Device)
[    2.816022] ACPI: 4 ACPI AML tables successfully acquired and loaded
[    2.852011] xen: registering gsi 9 triggering 0 polarity 0
[    2.856021] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored

... here, during dom0 boot.

[    2.871615] ACPI: Dynamic OEM Table Load:
[    2.941945] ACPI: Interpreter enabled
[    2.952021] ACPI: (supports S0 S3 S4 S5)
[    2.960004] ACPI: Using IOAPIC for interrupt routing
[    2.972031] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    2.993032] ACPI: Enabled 6 GPEs in block 00 to 3F
[    3.042478] ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff])
[    3.056010] acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    3.079707] acpi PNP0A03:02: _OSC: platform does not support [SHPCHotplug LTR]
[    3.098999] acpi PNP0A03:02: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]

What do you mean with "Which timer does this hardware use" ?

It's a Intel(R) Xeon(R) CPU E5-2697 and, when booted bare metal, this is what I see:

# dmesg |grep -i time
[    0.000000] ACPI: PM-Timer IO Port: 0x408
[    0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 5188.26 BogoMIPS (lpj=10376536)
[    0.022602] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.062298] TSC deadline timer enabled
[    0.436174] RTC time: 16:56:42, date: 03/12/20
[    2.117580] RAPL PMU: API unit is 2^-32 Joules, 4 fixed counters, 655360 ms ovfl timer
[    2.123351] workingset: timestamp_bits=36 max_order=23 bucket_order=0
[    2.288144] intel_idle: lapic_timer_reliable_states 0xffffffff

Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#16 lore raw
Quoted 60 lines · Dario Faggioli
Quoted 3 lines · Roger Pau Monné

Thanks for looking into this, seems like a specially tricky issue to tackle!

It was tricky indeed! :-)

Quoted 17 lines · Dario Faggioli

[...]

Quoted 6 lines

For example, I have a trace showing that csched2_schedule() is invoked at
t=57970746155ns. At t=57970747658ns (+1503ns) the s_timer is set to fire at t=57979485083ns, i.e., 8738928ns in future. That's because credit
of snext is exactly that 8738928ns. Then, what I see is that the

Quoted 8 lines

next
call to burn_credits(), coming from csched2_schedule() for the same vCPU
happens at t=60083283617ns. That is *a lot* (2103798534ns) later than
when we expected and asked. Of course, that also means that delta is
2112537462ns, and therefore credits will sink to -2103798534!

Quoted 3 lines

Which timer does this hardware use? DYK if there's some relation between the timer hardware used and the issue?

Timers came to mind but I haven't checked yet.

FWIW, one thing I saw is that, without patches, my machine times out around...

[    2.364819] NET: Registered protocol family 16
[    2.368018] xen:grant_table: Grant tables using version 1 layout
[    2.372033] Grant table initialized
[    2.377115] ACPI: bus type PCI registered
[    2.380011] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    2.384660] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
[    2.388033] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
[    2.499080] PCI: Using configuration type 1 for base access
[    2.516768] ACPI: Added _OSI(Module Device)
[    2.524006] ACPI: Added _OSI(Processor Device)
[    2.536004] ACPI: Added _OSI(3.0 _SCP Extensions)
[    2.544003] ACPI: Added _OSI(Processor Aggregator Device)
[    2.816022] ACPI: 4 ACPI AML tables successfully acquired and loaded
[    2.852011] xen: registering gsi 9 triggering 0 polarity 0
[    2.856021] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored

... here, during dom0 boot.

[    2.871615] ACPI: Dynamic OEM Table Load:
[    2.941945] ACPI: Interpreter enabled
[    2.952021] ACPI: (supports S0 S3 S4 S5)
[    2.960004] ACPI: Using IOAPIC for interrupt routing
[    2.972031] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    2.993032] ACPI: Enabled 6 GPEs in block 00 to 3F
[    3.042478] ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff])
[    3.056010] acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    3.079707] acpi PNP0A03:02: _OSC: platform does not support [SHPCHotplug LTR]
[    3.098999] acpi PNP0A03:02: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
Quoted 1 line

What do you mean with "Which timer does this hardware use" ?

Xen uses a hardware timer (HPET, PMTIMER or PIT IIRC) in order to get interrupts at specified times, on my box I see for example:

(XEN) Platform timer is 23.999MHz HPET

You should also see something along those lines. I was wondering if there was some relation between the timer in use and the delay in timer interrupts that you are seeing.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#17 lore raw
Quoted 10 lines · Roger Pau Monné
Quoted 1 line · Dario Faggioli

What do you mean with "Which timer does this hardware use" ?

Xen uses a hardware timer (HPET, PMTIMER or PIT IIRC) in order to get interrupts at specified times, on my box I see for example:

(XEN) Platform timer is 23.999MHz HPET

You should also see something along those lines.

Right. If booted Xen, I see this, which looks fine to me (this is currently Xen 4.12.2):

# xl dmesg |grep time
(XEN) [00018d6c323bf350] Platform timer is 14.318MHz HPET  <===========
(XEN) [   22.097647] TSC deadline timer enabled
(XEN) [   23.068865] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [   27.316536] mcheck_poll: Machine check polling timer started.

Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#18 lore raw
In reply to Dario Faggioli
Quoted 43 lines · Dario Faggioli

Hello everyone,

There have been reports of a Credit2 issue due to which vCPUs where being starved, to the point that guest kernel would complain or even crash.

See the following xen-users and xen-devel threads:
https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00018.html
https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00015.html
https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg01158.html

I did some investigations, and figured out that the vCPUs in question are not scheduled for long time intervals because they somehow manage to be given an amount of credits which is less than the credit the idle vCPU has.

An example of this situation is shown here. In fact, we can see d0v1 sitting in the runqueue while all the CPUs are idle, as it has -1254238270 credits, which is smaller than -2^30 = −1073741824:

     (XEN) Runqueue 0:
     (XEN)   ncpus              = 28
     (XEN)   cpus               = 0-27
     (XEN)   max_weight         = 256
     (XEN)   pick_bias          = 22
     (XEN)   instload           = 1
     (XEN)   aveload            = 293391 (~111%)
     (XEN)   idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
     (XEN)   tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
     (XEN)   fully idle cores: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
     [...]
     (XEN) Runqueue 0:
     (XEN) CPU[00] runq=0, sibling=00,..., core=00,...
     (XEN) CPU[01] runq=0, sibling=00,..., core=00,...
     [...]
     (XEN) CPU[26] runq=0, sibling=00,..., core=00,...
     (XEN) CPU[27] runq=0, sibling=00,..., core=00,...
     (XEN) RUNQ:
     (XEN)     0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 (~100%)

This happens bacause --although very rarely-- vCPUs are allowed to execute for much more than the scheduler would want them to.

Quoted 8 lines

For example, I have a trace showing that csched2_schedule() is invoked at t=57970746155ns. At t=57970747658ns (+1503ns) the s_timer is set to fire at t=57979485083ns, i.e., 8738928ns in future. That's because credit of snext is exactly that 8738928ns. Then, what I see is that the next call to burn_credits(), coming from csched2_schedule() for the same vCPU happens at t=60083283617ns. That is *a lot* (2103798534ns) later than when we expected and asked. Of course, that also means that delta is 2112537462ns, and therefore credits will sink to -2103798534!

Current ideas are:

- Could it be the vcpu is busy for very long time in the hypervisor? So either fighting with another vcpu for a lock, doing a long running hypercall, ...

- The timer used is not reliable.

- The time base is not reliable (tsc or whatever is used for getting the time has jumped 2 seconds into the future).

- System management mode has kicked in.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#19 lore raw
Quoted 62 lines · Jürgen Groß
Quoted 50 lines · Dario Faggioli

Hello everyone,

There have been reports of a Credit2 issue due to which vCPUs where being starved, to the point that guest kernel would complain or even crash.

See the following xen-users and xen-devel threads:
https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00018.html

https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00015.html

https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg01158.html

I did some investigations, and figured out that the vCPUs in question are not scheduled for long time intervals because they somehow manage to be given an amount of credits which is less than the credit the idle vCPU has.

An example of this situation is shown here. In fact, we can see d0v1 sitting in the runqueue while all the CPUs are idle, as it has -1254238270 credits, which is smaller than -2^30 = −1073741824:

     (XEN) Runqueue 0:
     (XEN)   ncpus              = 28
     (XEN)   cpus               = 0-27
     (XEN)   max_weight         = 256
     (XEN)   pick_bias          = 22
     (XEN)   instload           = 1
     (XEN)   aveload            = 293391 (~111%)
     (XEN)   idlers:
00,00000000,00000000,00000000,00000000,00000000,0fffffff (XEN)   tickled:
00,00000000,00000000,00000000,00000000,00000000,00000000 (XEN)   fully idle cores:
00,00000000,00000000,00000000,00000000,00000000,0fffffff [...]
     (XEN) Runqueue 0:
     (XEN) CPU[00] runq=0, sibling=00,..., core=00,... (XEN) CPU[01] runq=0, sibling=00,..., core=00,... [...]
     (XEN) CPU[26] runq=0, sibling=00,..., core=00,... (XEN) CPU[27] runq=0, sibling=00,..., core=00,... (XEN) RUNQ:
     (XEN)     0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 (~100%)

This happens bacause --although very rarely-- vCPUs are allowed to execute for much more than the scheduler would want them to.

Quoted 10 lines

For example, I have a trace showing that csched2_schedule() is invoked at
t=57970746155ns. At t=57970747658ns (+1503ns) the s_timer is set to fire at t=57979485083ns, i.e., 8738928ns in future. That's because credit
of snext is exactly that 8738928ns. Then, what I see is that the next call to burn_credits(), coming from csched2_schedule() for the same vCPU happens at t=60083283617ns. That is *a lot* (2103798534ns) later than when we expected and asked. Of course, that also means that delta is 2112537462ns, and therefore credits will sink to -2103798534!

Quoted 5 lines

Current ideas are:

- Could it be the vcpu is busy for very long time in the hypervisor? So either fighting with another vcpu for a lock, doing a long running hypercall, ...

Using watchdog=2 might catch that.  (There is a counting issue which I've not had time to fix yet, which makes the watchdog more fragile with a smaller timeout, but 2 should be ok.)

Quoted 5 lines

- The timer used is not reliable.

- The time base is not reliable (tsc or whatever is used for getting the time has jumped 2 seconds into the future).

Worth instrumenting the TSC rendezvous for unexpectedly large jumps?

Quoted 2 lines

- System management mode has kicked in.

SMM handlers need to rendezvous to keep their secrets secret these days, but I suppose this is always a possibility.

There are non-architectural SMI_COUNT MSRs (0x34 on Intel, can't remember AMD off the top of my head) which can be used to see if any have occurred, and this has proved useful in the past for debugging.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#20 lore raw
Quoted 11 lines · Andrew Cooper
Quoted 4 lines · Jürgen Groß

- Could it be the vcpu is busy for very long time in the hypervisor?
So either fighting with another vcpu for a lock, doing a long running hypercall, ...

Using watchdog=2 might catch that. (There is a counting issue which I've not had time to fix yet, which makes the watchdog more fragile with
a smaller timeout, but 2 should be ok.)

Done (actually, I used "watchdog=1 watchdog_timeout=2"), both with and without these patches applies.

I don't see any watchdog firing.

While I was there, I tried 1 too. Nothing either.

Quoted 8 lines
Quoted 5 lines

- The timer used is not reliable.

- The time base is not reliable (tsc or whatever is used for getting
the time has jumped 2 seconds into the future).

Worth instrumenting the TSC rendezvous for unexpectedly large jumps?

Yes, we should look into that.

Quoted 6 lines
Quoted 1 line

- System management mode has kicked in.

There are non-architectural SMI_COUNT MSRs (0x34 on Intel, can't remember AMD off the top of my head) which can be used to see if any have occurred, and this has proved useful in the past for debugging.

I'll have to look up at how to check and dump these from where I need. Let's see if I can do that... hints, code pointer or anything else you may have handy would be much appreciated. :-)

Thanks and Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel