WARNING: Adjusting tsc more then 11%
John,
We've had a number of reports the last few weeks triggering this warning that you added back in October.
https://bugzilla.redhat.com/show_bug.cgi?id=798600
https://bugzilla.redhat.com/show_bug.cgi?id=799215
https://bugzilla.redhat.com/show_bug.cgi?id=799745
https://bugzilla.redhat.com/show_bug.cgi?id=799872
any idea what could have changed to start tripping that up ?
The reports seem to have started around 3.3-rc4.
Dave
Quoted 4 lines · Dave Jones
John,
We've had a number of reports the last few weeks triggering this warning that you added back in October.
Quoted 8 lines
https://bugzilla.redhat.com/show_bug.cgi?id=798600
https://bugzilla.redhat.com/show_bug.cgi?id=799215
https://bugzilla.redhat.com/show_bug.cgi?id=799745
https://bugzilla.redhat.com/show_bug.cgi?id=799872any idea what could have changed to start tripping that up ?
The reports seem to have started around 3.3-rc4.
Huh. No I don't know what would have started causing such a warning. I had expected that there would be some edge hardware that might trip that warning, but I'd expect the noise to start there w/ 3.2 after it was introduced. There's only been spelling & comment changes to the timekeeping core in the 3.3-rc series.
Looking at the logs, I'm curious if maybe some of the nohz changes have tweaked us here. If we're hitting longer idle times, it might be possible the ntp error could accumulate to be larger, and then the kernel's frequency adjustments might end up trying to adjust for more then 11%.
Its also interesting its only popping up on hardware using the TSC. acpi_pm can only do short idle times, so it wouldn't show up here, but hpet could do reasonably long idles, so I'd think we'd also see some similar hpet warnings.
Do you know if this is an occasional thing on any of the affected hardware, or if it happens after every reboot?
Are any of the reported boxes systems you have access to in order to reproduce?
thanks
-john
Quoted 11 lines · John Stultz
Quoted 3 lines · Dave Jones
any idea what could have changed to start tripping that up ?
The reports seem to have started around 3.3-rc4.
Huh. No I don't know what would have started causing such a warning. I had expected that there would be some edge hardware that might trip that warning, but I'd expect the noise to start there w/ 3.2 after it was introduced. There's only been spelling & comment changes to the timekeeping core in the 3.3-rc series.
thinking about this some more, while the reports starts around rc4, this may have been caused by something prior to that, as anyone moving from Fedora 16 or earlier to F17 alpha would have jumped quite a kernel version or two.
Quoted 2 lines
Do you know if this is an occasional thing on any of the affected hardware, or if it happens after every reboot?
Out of all the people running the Fedora 17 alpha, this has only shown up those four times, so it does seem to be a rare thing.
I suspect we'll get more instances of it as more people start testing.
Three of the reporters noted that it happened on boot.
Quoted 2 lines
Are any of the reported boxes systems you have access to in order to reproduce?
unfortunately not.
Dave
Quoted 13 lines · Dave Jones
Quoted 11 lines · John Stultz
Quoted 3 lines · Dave Jones
any idea what could have changed to start tripping that up ?
The reports seem to have started around 3.3-rc4.
Huh. No I don't know what would have started causing such a warning. I had expected that there would be some edge hardware that might trip that warning, but I'd expect the noise to start there w/ 3.2 after it was introduced. There's only been spelling & comment changes to the timekeeping core in the 3.3-rc series.
Quoted 3 lines
thinking about this some more, while the reports starts around rc4, this may have been caused by something prior to that, as anyone moving from Fedora 16 or earlier to F17 alpha would have jumped quite a kernel version or two.
Was F16 3.1 based? The warning was added in 3.2, so if you skipped it, it may not be new behavior then.
Quoted 7 lines
Quoted 2 lines
Do you know if this is an occasional thing on any of the affected hardware, or if it happens after every reboot?
Out of all the people running the Fedora 17 alpha, this has only shown up those four times, so it does seem to be a rare thing.
I suspect we'll get more instances of it as more people start testing.
Quoted 6 lines
Three of the reporters noted that it happened on boot.
Quoted 2 lines
Are any of the reported boxes systems you have access to in order to reproduce?
unfortunately not.
Ok. Well, just to level set: the warning is informative, and points to unexpected, but not necessarily unsafe behavior.
In fact, the risk (where mult is adjusted to be large enough to cause an overflow) we're warning about have been present 2.6.36 or even possibly before. The change in 3.2 which added the warning also added a more conservative mult calculation, so we're less likely to get overflow prone large mult values.
So it would be great to get further feedback from folks who are seeing this warning, so we can really hammer this out, but I don't want the warning spooking anyone into thinking things are terribly broken.
thanks
-john
Quoted 35 lines · John Stultz
Quoted 13 lines · Dave Jones
Quoted 11 lines · John Stultz
Quoted 3 lines · Dave Jones
any idea what could have changed to start tripping that up ?
The reports seem to have started around 3.3-rc4.
Huh. No I don't know what would have started causing such a warning. I had expected that there would be some edge hardware that might trip that warning, but I'd expect the noise to start there w/ 3.2 after it was introduced. There's only been spelling & comment changes to the timekeeping core in the 3.3-rc series.
Quoted 3 lines
thinking about this some more, while the reports starts around rc4, this may have been caused by something prior to that, as anyone moving from Fedora 16 or earlier to F17 alpha would have jumped quite a kernel version or two.
Was F16 3.1 based? The warning was added in 3.2, so if you skipped it, it may not be new behavior then.
Quoted 7 lines
Quoted 2 lines
Do you know if this is an occasional thing on any of the affected hardware, or if it happens after every reboot?
Out of all the people running the Fedora 17 alpha, this has only shown up those four times, so it does seem to be a rare thing.
I suspect we'll get more instances of it as more people start testing.Quoted 6 lines
Three of the reporters noted that it happened on boot.
Quoted 2 lines
Are any of the reported boxes systems you have access to in order to reproduce?
unfortunately not.
Quoted 8 lines
Ok. Well, just to level set: the warning is informative, and points to unexpected, but not necessarily unsafe behavior.
In fact, the risk (where mult is adjusted to be large enough to cause an overflow) we're warning about have been present 2.6.36 or even possibly before. The change in 3.2 which added the warning also added a more conservative mult calculation, so we're less likely to get overflow prone large mult values.
Is there a reason you decided to use a WARN_ONCE, which dumps a full stack trace, instead of just printk(KERN_ERR ?
Quoted 3 lines
So it would be great to get further feedback from folks who are seeing this warning, so we can really hammer this out, but I don't want the warning spooking anyone into thinking things are terribly broken.
Right... people see backtraces and start thinking "my kernel is broken."
I'm certainly not meaning to pick on you for this. Lately it seems all the rage to throw WARN_ONs for all kinds of error paths and leave the user to figure out how screwed they are.
josh
Quoted 7 lines · John Stultz
Quoted 4 lines
Quoted 1 line
Quoted 1 line
The reports seem to have started around 3.3-rc4.
thinking about this some more, while the reports starts around rc4, this may have been caused by something prior to that, as anyone moving from Fedora 16 or earlier to F17 alpha would have jumped quite a kernel version or two.
Was F16 3.1 based? The warning was added in 3.2, so if you skipped it, it may not be new behavior then.
initially 3.1 based. We recently pushed out a rebase to 3.2. This bug hasn't shown up there (yet?)
Quoted 9 lines
Ok. Well, just to level set: the warning is informative, and points to unexpected, but not necessarily unsafe behavior.
In fact, the risk (where mult is adjusted to be large enough to cause an overflow) we're warning about have been present 2.6.36 or even possibly before. The change in 3.2 which added the warning also added a more conservative mult calculation, so we're less likely to get overflow prone large mult values.
Quoted 3 lines
So it would be great to get further feedback from folks who are seeing this warning, so we can really hammer this out, but I don't want the warning spooking anyone into thinking things are terribly broken.
One of the reports was from someone using vbox. I'm now wondering if the other users are using some other flavour of virt. I'll ask.
Dave
Quoted 26 lines · Dave Jones
On Mon, Mar 05, 2012 at 11:50:10AM -0800, John Stultz wrote:
> > > > The reports seem to have started around 3.3-rc4.
> > thinking about this some more, while the reports starts around rc4, this > > may have been caused by something prior to that, as anyone moving from > > Fedora 16 or earlier to F17 alpha would have jumped quite a kernel version or two. >
> Was F16 3.1 based? The warning was added in 3.2, so if you skipped it, > it may not be new behavior then.initially 3.1 based. We recently pushed out a rebase to 3.2. This bug hasn't shown up there (yet?)
> Ok. Well, just to level set: the warning is informative, and points to > unexpected, but not necessarily unsafe behavior.
>
> In fact, the risk (where mult is adjusted to be large enough to cause an > overflow) we're warning about have been present 2.6.36 or even possibly > before. The change in 3.2 which added the warning also added a more > conservative mult calculation, so we're less likely to get overflow > prone large mult values.
>
> So it would be great to get further feedback from folks who are seeing > this warning, so we can really hammer this out, but I don't want the > warning spooking anyone into thinking things are terribly broken.
Quoted 2 lines
One of the reports was from someone using vbox. I'm now wondering if the other users are using some other flavour of virt. I'll ask.
I've reported this issue with regards to KVM before
(http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/85632) but according to this thread it's not really KVM specific.
It's easily reproducible using Dave's trinity running inside a KVM guest.
Quoted 39 lines · Josh Boyer
Quoted 38 lines · John Stultz
Quoted 13 lines · Dave Jones
Quoted 11 lines · John Stultz
Quoted 3 lines · Dave Jones
any idea what could have changed to start tripping that up ?
The reports seem to have started around 3.3-rc4.
Huh. No I don't know what would have started causing such a warning. I had expected that there would be some edge hardware that might trip that warning, but I'd expect the noise to start there w/ 3.2 after it was introduced. There's only been spelling & comment changes to the timekeeping core in the 3.3-rc series.
Quoted 3 lines
thinking about this some more, while the reports starts around rc4, this may have been caused by something prior to that, as anyone moving from Fedora 16 or earlier to F17 alpha would have jumped quite a kernel version or two.
Was F16 3.1 based? The warning was added in 3.2, so if you skipped it, it may not be new behavior then.
Quoted 7 lines
Quoted 2 lines
Do you know if this is an occasional thing on any of the affected hardware, or if it happens after every reboot?
Out of all the people running the Fedora 17 alpha, this has only shown up those four times, so it does seem to be a rare thing.
I suspect we'll get more instances of it as more people start testing.Quoted 6 lines
Three of the reporters noted that it happened on boot.
Quoted 2 lines
Are any of the reported boxes systems you have access to in order to reproduce?
unfortunately not.
Ok. Well, just to level set: the warning is informative, and points to unexpected, but not necessarily unsafe behavior.
Quoted 8 lines
Quoted 5 lines
In fact, the risk (where mult is adjusted to be large enough to cause an overflow) we're warning about have been present 2.6.36 or even possibly before. The change in 3.2 which added the warning also added a more conservative mult calculation, so we're less likely to get overflow prone large mult values.
Is there a reason you decided to use a
WARN_ONCE, which dumps a full stack trace, instead of just printk(KERN_ERR ?
Well, the WARN_ONCE behavior is really nice, since just a printk would end up possibly filling the logs, since you might get one every tick.
Quoted 9 lines
Quoted 3 lines
So it would be great to get further feedback from folks who are seeing this warning, so we can really hammer this out, but I don't want the warning spooking anyone into thinking things are terribly broken.
Right... people see backtraces and start thinking "my kernel is broken."
I'm certainly not meaning to pick on you for this. Lately it seems all the rage to throw
WARN_ONsfor all kinds of error paths and leave the user to figure out how screwed they are.
Its a trade-off, since we really do want to know if our code has been pushed outside of its expected boundaries (either by unexpected hadware behavior or by expectations being raised, like long nohz idle times), so we have to get folks attention somewhat. The type of error reporting Dave's managed to collect here is really great.
But at the same time, I agree there has been a few cases where the code is limited more narrowly then the reality of existing hardware, and we end up with a constant stream of error messages that get waved off as broken hardware.
There we need to either fix the code or drop the warnings, but I think it gets hard when we really want to know about "unexpected behavior, except on some wide swath of hardware that always acts poorly", where conditionalizing the warnings isn't easy.
thanks
-john
Quoted 7 lines · Sasha Levin
Quoted 6 lines · Dave Jones
Quoted 3 lines · John Stultz
So it would be great to get further feedback from folks who are seeing this warning, so we can really hammer this out, but I don't want the warning spooking anyone into thinking things are terribly broken.
Quoted 8 lines
Quoted 2 lines
One of the reports was from someone using vbox. I'm now wondering if the other users are using some other flavour of virt. I'll ask.
I've reported this issue with regards to KVM before
(http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/85632) but according to this thread it's not really KVM specific.It's easily reproducible using Dave's trinity running inside a KVM guest.
Great! Do you have a link to trinity? (Google just gives me a video of its use and links to various churches)
thanks
-john
Quoted 9 lines · John Stultz
Quoted 9 lines
Quoted 8 lines
Ok. Well, just to level set: the warning is informative, and points to unexpected, but not necessarily unsafe behavior.
In fact, the risk (where mult is adjusted to be large enough to cause an overflow) we're warning about have been present 2.6.36 or even possibly before. The change in 3.2 which added the warning also added a more conservative mult calculation, so we're less likely to get overflow prone large mult values.
Quoted 5 lines
Quoted 2 lines
Is there a reason you decided to use a
WARN_ONCE, which dumps a full stack trace, instead of just printk(KERN_ERR ?Well, the
WARN_ONCEbehavior is really nice, since just a printk would end up possibly filling the logs, since you might get one every tick.
We have printk_once too.
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
So it would be great to get further feedback from folks who are seeing this warning, so we can really hammer this out, but I don't want the warning spooking anyone into thinking things are terribly broken.
Right... people see backtraces and start thinking "my kernel is broken."
I'm certainly not meaning to pick on you for this. Lately it seems all the rage to throw
WARN_ONsfor all kinds of error paths and leave the user to figure out how screwed they are.
Quoted 5 lines
Its a trade-off, since we really do want to know if our code has been pushed outside of its expected boundaries (either by unexpected hadware behavior or by expectations being raised, like long nohz idle times), so we have to get folks attention somewhat. The type of error reporting Dave's managed to collect here is really great.
It is, yes. Do you know, aside from distro kernel maintainers, how many reports have you gotten from actual users directly?
Quoted 9 lines
But at the same time, I agree there has been a few cases where the code is limited more narrowly then the reality of existing hardware, and we end up with a constant stream of error messages that get waved off as broken hardware.
There we need to either fix the code or drop the warnings, but I think it gets hard when we really want to know about "unexpected behavior, except on some wide swath of hardware that always acts poorly", where conditionalizing the warnings isn't easy.
Oh my. Quirks in the timekeeping code would just give me nightmares ;).
josh
Quoted 11 lines · John Stultz
Quoted 10 lines · Sasha Levin
Quoted 8 lines · Dave Jones
On Mon, Mar 05, 2012 at 11:50:10AM -0800, John Stultz wrote:
> So it would be great to get further feedback from folks who are seeing > this warning, so we can really hammer this out, but I don't want the > warning spooking anyone into thinking things are terribly broken.
One of the reports was from someone using vbox. I'm now wondering if the other users are using some other flavour of virt. I'll ask.
Quoted 8 lines
Quoted 5 lines
I've reported this issue with regards to KVM before
(http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/85632) but according to this thread it's not really KVM specific.It's easily reproducible using Dave's trinity running inside a KVM guest.
Great! Do you have a link to trinity? (Google just gives me a video of its use and links to various churches)
Sure, the git tree is here: codemonkey.org.uk/projects/trinity/trinity.git
Basically just start 2*vcpu instances of trinity inside the guest and wait a bit.
Quoted 10 lines · Josh Boyer
Quoted 9 lines · John Stultz
Quoted 9 lines
Quoted 8 lines
Ok. Well, just to level set: the warning is informative, and points to unexpected, but not necessarily unsafe behavior.
In fact, the risk (where mult is adjusted to be large enough to cause an overflow) we're warning about have been present 2.6.36 or even possibly before. The change in 3.2 which added the warning also added a more conservative mult calculation, so we're less likely to get overflow prone large mult values.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Is there a reason you decided to use a
WARN_ONCE, which dumps a full stack trace, instead of just printk(KERN_ERR ?Well, the
WARN_ONCEbehavior is really nice, since just a printk would end up possibly filling the logs, since you might get one every tick.We have
printk_oncetoo.
Good point. I didn't look into that. The backtrace isn't very useful, so I'll see about changing it in the future.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
So it would be great to get further feedback from folks who are seeing this warning, so we can really hammer this out, but I don't want the warning spooking anyone into thinking things are terribly broken.
Right... people see backtraces and start thinking "my kernel is broken."
I'm certainly not meaning to pick on you for this. Lately it seems all the rage to throw
WARN_ONsfor all kinds of error paths and leave the user to figure out how screwed they are.
Quoted 8 lines
Quoted 5 lines
Its a trade-off, since we really do want to know if our code has been pushed outside of its expected boundaries (either by unexpected hadware behavior or by expectations being raised, like long nohz idle times), so we have to get folks attention somewhat. The type of error reporting Dave's managed to collect here is really great.
It is, yes. Do you know, aside from distro kernel maintainers, how many reports have you gotten from actual users directly?
Zero so far. Dave's are the first that I've been made aware of.
Quoted 11 lines
Quoted 9 lines
But at the same time, I agree there has been a few cases where the code is limited more narrowly then the reality of existing hardware, and we end up with a constant stream of error messages that get waved off as broken hardware.
There we need to either fix the code or drop the warnings, but I think it gets hard when we really want to know about "unexpected behavior, except on some wide swath of hardware that always acts poorly", where conditionalizing the warnings isn't easy.
Oh my. Quirks in the timekeeping code would just give me nightmares ;).
:)
-john
Quoted 18 lines · Sasha Levin
Quoted 17 lines · John Stultz
Quoted 15 lines · Sasha Levin
Quoted 8 lines · Dave Jones
Quoted 3 lines · John Stultz
So it would be great to get further feedback from folks who are seeing this warning, so we can really hammer this out, but I don't want the warning spooking anyone into thinking things are terribly broken.
One of the reports was from someone using vbox. I'm now wondering if the other users are using some other flavour of virt. I'll ask.
I've reported this issue with regards to KVM before
(http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/85632) but according to this thread it's not really KVM specific.It's easily reproducible using Dave's trinity running inside a KVM guest.
Quoted 7 lines
Quoted 2 lines
Great! Do you have a link to trinity? (Google just gives me a video of its use and links to various churches)
Sure, the git tree is here: codemonkey.org.uk/projects/trinity/trinity.git
Basically just start 2*vcpu instances of trinity inside the guest and wait a bit.
So I left trinity running in a 2vcpu kvm guest for ~5 hours and haven't seen anything yet. Could you send me a .config for the guest kernel and maybe your kvm command line?
Also, it shouldn't really matter, but are the guests 32bit or 64bit?
thanks
-john
Quoted 6 lines · Dave Jones
Quoted 2 lines
Do you know if this is an occasional thing on any of the affected hardware, or if it happens after every reboot?
Out of all the people running the Fedora 17 alpha, this has only shown up those four times, so it does seem to be a rare thing.
I suspect we'll get more instances of it as more people start testing.
As suspected, the reports are starting to come in more regularly.
See the duplicates marked at https://bugzilla.redhat.com/show_bug.cgi?id=798600
Dave
Quoted 27 lines · John Stultz
Quoted 18 lines · Sasha Levin
Quoted 17 lines · John Stultz
Quoted 15 lines · Sasha Levin
Quoted 8 lines · Dave Jones
On Mon, Mar 05, 2012 at 11:50:10AM -0800, John Stultz wrote:
> So it would be great to get further feedback from folks who are seeing > this warning, so we can really hammer this out, but I don't want the > warning spooking anyone into thinking things are terribly broken.
One of the reports was from someone using vbox. I'm now wondering if the other users are using some other flavour of virt. I'll ask.
I've reported this issue with regards to KVM before
(http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/85632) but according to this thread it's not really KVM specific.It's easily reproducible using Dave's trinity running inside a KVM guest.
Quoted 7 lines
Quoted 2 lines
Great! Do you have a link to trinity? (Google just gives me a video of its use and links to various churches)
Sure, the git tree is here: codemonkey.org.uk/projects/trinity/trinity.git
Basically just start 2*vcpu instances of trinity inside the guest and wait a bit.
Quoted 10 lines
So I left trinity running in a 2vcpu kvm guest for ~5 hours and haven't seen anything yet. Could you send me a .config for the guest kernel and maybe your kvm command line?
Also, it shouldn't really matter, but are the guests 32bit or 64bit?
thanks
-john
Sorry, I missed this mail.
You should start several trinity instances, probably 4-5 on a 2vcpu guest.
I've attached my .config.
(This issue still happens quite often with linux-next)
Quoted 30 lines · Sasha Levin
Quoted 29 lines · John Stultz
Quoted 15 lines · Sasha Levin
Quoted 14 lines · John Stultz
Quoted 13 lines · Sasha Levin
Quoted 8 lines · Dave Jones
Quoted 3 lines · John Stultz
So it would be great to get further feedback from folks who are seeing this warning, so we can really hammer this out, but I don't want the warning spooking anyone into thinking things are terribly broken.
One of the reports was from someone using vbox. I'm now wondering if the other users are using some other flavour of virt. I'll ask.
I've reported this issue with regards to KVM before
(http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/85632) but according to this thread it's not really KVM specific.Quoted 7 lines
Quoted 3 lines
Quoted 1 line
It's easily reproducible using Dave's trinity running inside a KVM guest.
Great! Do you have a link to trinity? (Google just gives me a video of its use and links to various churches)
Sure, the git tree is here: codemonkey.org.uk/projects/trinity/trinity.git
Basically just start 2*vcpu instances of trinity inside the guest and wait a bit.
So I left trinity running in a 2vcpu kvm guest for ~5 hours and haven't seen anything yet. Could you send me a .config for the guest kernel and maybe your kvm command line?
Also, it shouldn't really matter, but are the guests 32bit or 64bit?
Quoted 8 lines
Quoted 4 lines
thanks
-johnSorry, I missed this mail.
You should start several trinity instances, probably 4-5 on a 2vcpu guest.
Ah. Sorry I misread the 2*vcpu as "on a 2 vcpu instance". Thanks for the clarification. Re-running tests now.
thanks
-john