Time goes backwards in dom0 in xen-unstable
I'm seeing some "Time went backwards" errors reported in dom0 in near-tip (c/s 19515) xen-unstable build. It's rare and random and not reproducible, but here's the report that just showed up. There was no load running at the time.
I can move to 3.4-rc1 if that would be helpful but I don't remember seeing any time-related changes recently.
This was on my dual-core test machine which reports lots of power management info during Xen boot.
Dan
Timer ISR/0: Time went backwards: delta=-14971734 delta_cpu=25028266 shadow=3655 618577922 off=580668482 processed=3656214217729 cpu_processed=3656174217729
0: 3656174217729
1: 3656214217729
Timer ISR/0: Time went backwards: delta=-14971447 delta_cpu=25028553 shadow=3655 618577922 off=590668621 processed=3656224217729 cpu_processed=3656184217729
0: 3656184217729
1: 3656224217729
Timer ISR/0: Time went backwards: delta=-14970691 delta_cpu=25029309 shadow=3655 618577922 off=620669383 processed=3656254217729 cpu_processed=3656214217729
0: 3656214217729
1: 3656254217729
Timer ISR/0: Time went backwards: delta=-14970623 delta_cpu=25029377 shadow=3655 618577922 off=630669373 processed=3656264217729 cpu_processed=3656224217729
0: 3656224217729
1: 3656254217729
Timer ISR/0: Time went backwards: delta=-14969392 delta_cpu=25030608 shadow=3655 618577922 off=680670676 processed=3656314217729 cpu_processed=3656274217729
0: 3656274217729
1: 3656304217729
Timer ISR/0: Time went backwards: delta=-14967619 delta_cpu=25032381 shadow=3655 618577922 off=740672580 processed=3656374217729 cpu_processed=3656334217729
0: 3656334217729
1: 3656374217729
Timer ISR/0: Time went backwards: delta=-14967315 delta_cpu=25032685 shadow=3655 618577922 off=750672771 processed=3656384217729 cpu_processed=3656344217729
0: 3656344217729
1: 3656374217729
Quoted 9 lines
From: Dan Magenheimer
Sent: 2009年4月15日 3:37I'm seeing some "Time went backwards" errors reported in dom0 in near-tip (c/s 19515) xen-unstable build. It's rare and random and not reproducible, but here's the report that just showed up. There was no load running at the time.
Quoted 5 lines
I can move to 3.4-rc1 if that would be helpful but I don't remember seeing any time-related changes recently.
This was on my dual-core test machine which reports lots of power management info during Xen boot.
could you add "hpetbroadcast" in grub.conf? This is possibly caused by PIT broadcast again as in your Mcycles thread.
http://markmail.org/thread/uzqcmjuu6uc5xxqd is one write-up about those side-effects when cpuidle gets enabled by default in Xen 3.4. Preferred way to do broadcast is always to use HPET, however by far one missing feature makes PIT as default broadcast on most platforms. That missing feature is to emulate RTC interrupt with IRQ8 replaced by HPET if using legacy replacement mode. We're adding that emulation now and hopefully get it ready in several days and in final Xen 3.4 release. At that time, PIT will be only turned to on some old platform (then no deep C-state support) or weird BIOS not exporting HPET (then simply disable cpuidle).
Thanks
Kevin
Quoted 46 lines
Dan
Timer ISR/0: Time went backwards: delta=-14971734 delta_cpu=25028266 shadow=3655
618577922 off=580668482 processed=3656214217729 cpu_processed=3656174217729
0: 3656174217729
1: 3656214217729
Timer ISR/0: Time went backwards: delta=-14971447 delta_cpu=25028553 shadow=3655
618577922 off=590668621 processed=3656224217729 cpu_processed=3656184217729
0: 3656184217729
1: 3656224217729
Timer ISR/0: Time went backwards: delta=-14970691 delta_cpu=25029309 shadow=3655
618577922 off=620669383 processed=3656254217729 cpu_processed=3656214217729
0: 3656214217729
1: 3656254217729
Timer ISR/0: Time went backwards: delta=-14970623 delta_cpu=25029377 shadow=3655
618577922 off=630669373 processed=3656264217729 cpu_processed=3656224217729
0: 3656224217729
1: 3656254217729
Timer ISR/0: Time went backwards: delta=-14969392 delta_cpu=25030608 shadow=3655
618577922 off=680670676 processed=3656314217729 cpu_processed=3656274217729
0: 3656274217729
1: 3656304217729
Timer ISR/0: Time went backwards: delta=-14967619 delta_cpu=25032381 shadow=3655
618577922 off=740672580 processed=3656374217729 cpu_processed=3656334217729
0: 3656334217729
1: 3656374217729
Timer ISR/0: Time went backwards: delta=-14967315 delta_cpu=25032685 shadow=3655
618577922 off=750672771 processed=3656384217729 cpu_processed=3656344217729
0: 3656344217729
1: 3656374217729
Quoted 5 lines
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 3 lines · Kevin
At that time, PIT will be only turned to
on some old platform (then no deep C-state support) or weird BIOS not exporting HPET (then simply disable cpuidle).
Changeset 19545 now disables cpuidle by default if HPET broadcast is unavailable. Obviously the PIT alternative is not up to being enabled by default.
-- Keir
Quoted 5 lines
From: Keir Fraser [mailto:[email protected]]
Sent: 2009年4月15日 15:41On 15/04/2009 00:31, "Tian, Kevin" <[email protected]> wrote:
Quoted 9 lines
Quoted 3 lines
At that time, PIT will be only turned to
on some old platform (then no deep C-state support) or weird BIOS not exporting HPET (then simply disable cpuidle).Changeset 19545 now disables cpuidle by default if HPET broadcast is unavailable. Obviously the PIT alternative is not up to being enabled by
default.
Thanks, that's a wise decision.
Kevin
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 5 lines · Kevin
That missing feature
is to emulate RTC interrupt with IRQ8 replaced by HPET if using legacy replacement mode. We're adding that emulation now and hopefully get it ready in several days and in final Xen 3.4 release.
It's a bit late for 3.4 really. We'll be past -rc2 by the time you get this patch out so it's missed the boat.
-- Keir
Quoted 11 lines
From: Keir Fraser [mailto:[email protected]]
Sent: 2009年4月15日 15:44Quoted 5 lines · Kevin
That missing feature
is to emulate RTC interrupt with IRQ8 replaced by HPET if using legacy replacement mode. We're adding that emulation now and hopefully get it ready in several days and in final Xen 3.4 release.
Quoted 4 lines
It's a bit late for 3.4 really. We'll be past -rc2 by the time you get this
patch out so it's missed the boat.
We revised our plan here, after realizing not-negligible effort to enable RTC emulation. Instead we want to turn to an alternative that hpet broadcast is always enabled once available (current situation is to always give up if no MSI delivery support), and then disable it on the fly (disable HPET interrupt and reduce max_cstate to a level not requiring broadcast) once detecting user trying to enable UIE/PIE/AIE on RTC. That way is far simpler and makes cpuidle really available on most platforms with HPET (true for most with VT support), in the meantime not breaking occasional usage on RTC interrupt.
Will you accept such patch which handles mostly about policy? :-) We'll try to send it out today.
Thanks,
Kevin
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 11 lines · Kevin
We revised our plan here, after realizing not-negligible effort to enable RTC emulation. Instead we want to turn to an alternative that hpet broadcast is always enabled once available (current situation is to always give up if no MSI delivery support), and then disable it on the fly (disable HPET interrupt and reduce
max_cstateto a level not requiring broadcast) once detecting user trying to enable UIE/PIE/AIE on RTC. That way is far simpler and makes cpuidle really available on most platforms with HPET (true for most with VT support), in the meantime not breaking occasional usage on RTC interrupt.
Quoted 2 lines
Will you accept such patch which handles mostly about policy? :-) We'll try to send it out today.
It's got a better chance than RTC emulation. I'll certainly consider it.
-- Keir
PATCH 1/1 cpuidle: Enable hpet broadcast by default (RE: Time goes backwards in dom0 in xen-unstable)
cpuidle: Enable hpet broadcast by default
And stop legacy hpet broadcast and limit max C-state to shallower state if RTC interrupts are enabled.
Quoted 18 lines · Keir Fraser
Quoted 11 lines · Kevin
We revised our plan here, after realizing not-negligible effort to enable RTC emulation. Instead we want to turn to an alternative that hpet broadcast is always enabled once available (current situation is to always give up if no MSI delivery support), and then disable it on the fly (disable HPET interrupt and reduce
max_cstateto a level not requiring broadcast) once detecting user trying to enable UIE/PIE/AIE on RTC. That way is far simpler and makes cpuidle really available on most platforms with HPET (true for most with VT support), in the meantime not breaking occasional usage on RTC interrupt.Quoted 2 lines
Will you accept such patch which handles mostly about policy? :-) We'll try to send it out today.
It's got a better chance than RTC emulation. I'll certainly consider it.
Quoted 8 lines
-- Keir
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
cpuidle: Enable hpet broadcast by default
And stop legacy hpet broadcast and limit max C-state to shallower state if RTC interrupts are enabled.
xen/arch/x86/acpi/cpu_idle.c +17 -2
@@ -221,8 +221,8 @@ static void acpi_processor_idle(void)return ; } - next_state = power ? cpuidle_current_governor->select (power) :-1 ; -if ( next_state >0 ) +if ( max_cstate >0 && power && + (next_state = cpuidle_current_governor->select (power)) >0 ) { cx = &power->states[next_state]; if ( power->flags.bm_check && acpi_idle_bm_check()
@@ -853,3 +853,18 @@ int pmstat_reset_cx_stat(uint32_t cpuid)return 0 ; } +void cpuidle_disable_deep_cstate (void ) +{ +if ( max_cstate >1 ) + { +if ( local_apic_timer_c2_ok ) + max_cstate =2 ; +else + max_cstate =1 ; + } + +mb (); + +hpet_disable_legacy_broadcast (); +} +
xen/arch/x86/hpet.c +67 -4
109 lines not on this page · raw message
xen/arch/x86/traps.c +6 -0
@@ -1551,6 +1551,8 @@ static uint32_t guest_io_read(return data; } +extern void (*pv_rtc_handler)(unsigned int port,uint8_t value); +static void guest_io_write (unsigned int port,unsigned int bytes,uint32_t data,struct vcpu *v,struct cpu_user_regs *regs)
@@ -1565,6 +1567,8 @@ static void guest_io_write(outb ((uint8_t )data, port);if ( pv_post_outb_hook )pv_post_outb_hook (port, (uint8_t )data); +if ( ((port ==0x71 ) || (port ==0x70 )) && pv_rtc_handler ) +pv_rtc_handler (port, (uint8_t )data);break ;case 2 :outw ((uint16_t )data, port);
@@ -1936,6 +1940,8 @@ static int emulate_privileged_op(structio_emul (regs);if ( (op_bytes ==1 ) && pv_post_outb_hook )pv_post_outb_hook (port, regs->eax); +if ( ((port ==0x71 ) || (port ==0x70 )) && pv_rtc_handler ) +pv_rtc_handler (port, regs->eax); }else {
xen/include/asm-x86/hpet.h +1 -0
@@ -78,5 +78,6 @@ void hpet_broadcast_enter(void);void hpet_broadcast_enter (void );void hpet_broadcast_exit (void );int hpet_broadcast_is_available (void ); +void hpet_disable_legacy_broadcast (void );#endif /* __X86_HPET_H__ */
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
PATCH 1/1 cpuidle: Enable hpet broadcast by default (RE: Time goes backwards in dom0 in xen-unstable)
Quoted 4 lines
Quoted 1 line
Quoted 1 line
"Wei, Gang" <[email protected]> 15.04.09 16:09 >>>
cpuidle: Enable hpet broadcast by default
And stop legacy hpet broadcast and limit max C-state to shallower state if RTC interrupts are enabled.
Shouldn't the pv_rtc_handler() hook be called *before* the output actually happens?
Jan
RE: [PATCH] cpuidle: Enable hpet broadcast by default (RE: Time goes backwards in dom0 in xen-unstable)
Quoted 8 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line
"Wei, Gang" <[email protected]> 15.04.09 16:09 >>>
cpuidle: Enable hpet broadcast by default
And stop legacy hpet broadcast and limit max C-state to shallower state if RTC interrupts are enabled.
Shouldn't the
pv_rtc_handler()hook be called *before* the output actually happens?
Your are right, that would be safer. As this patch has already been checked in, could you send out the fix? It is your good point. :)
Jimmy
Quoted 11 lines · Dan Magenheimer
I'm seeing some "Time went backwards" errors reported in dom0 in near-tip (c/s 19515) xen-unstable build. It's rare and random and not reproducible, but here's the report that just showed up. There was no load running at the time.
I can move to 3.4-rc1 if that would be helpful but I don't remember seeing any time-related changes recently.
This was on my dual-core test machine which reports lots of power management info during Xen boot.
If you specify 'cpuidle=off' as a Xen boot parameter, does that make the timer_interrupt scalability problem go away, and this time backwards problem? I was going to enable by default in 3.4 but could go the other way.
-- Keir
I can confirm that cpuidle=off makes the timer_interrupt scaleability problem go away.
It also appears that the max cycles for the MSI interrupts becomes reasonably small again. Was this expected?
I'll leave it running for awhile but may not be able to confirm the "Time went backwards" error goes away as it seemed to appear only after a random long period of time.
(BTW, Kevin, hpetbroadcast did not make the problem go away.)
Dan
Quoted 37 lines
-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Wednesday, April 15, 2009 1:31 AM
To: Dan Magenheimer; Xen-Devel (E-mail)
Subject: Re: [Xen-devel] Time goes backwards in dom0 in xen-unstableQuoted 11 lines · Dan Magenheimer
I'm seeing some "Time went backwards" errors reported in dom0 in near-tip (c/s 19515) xen-unstable build. It's rare and random and not reproducible, but here's the report that just showed up. There was no load running at the time.
I can move to 3.4-rc1 if that would be helpful but I don't remember seeing any time-related changes recently.
This was on my dual-core test machine which reports lots of power management info during Xen boot.
If you specify 'cpuidle=off' as a Xen boot parameter, does that make the
timer_interruptscalability problem go away, and this time backwards problem? I was going to enable by default in 3.4 but could go the other way.-- Keir
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Hmmm... after only a few minutes with cpuidle=off, my test domPV froze up after printing a number of call traces starting with:
INFO: task xxx:nnn blocked for more than 480 seconds.
At the top of all of the traces is either
getnstimeofday+51 or io_schedule+44.
(Note that this PV domain is a 2.6.29 kernel... don't know if the messages are the same on an older kernel.)
Quoted 61 lines
-----Original Message-----
From: Dan Magenheimer
Sent: Wednesday, April 15, 2009 8:22 AM
To: Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstableI can confirm that cpuidle=off makes the
timer_interruptscaleability problem go away.It also appears that the max cycles for the MSI interrupts becomes reasonably small again. Was this expected?
I'll leave it running for awhile but may not be able to confirm the "Time went backwards" error goes away as it seemed to appear only after a random long period of time.
(BTW, Kevin, hpetbroadcast did not make the problem go away.)
Dan
Quoted 37 lines
-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Wednesday, April 15, 2009 1:31 AM
To: Dan Magenheimer; Xen-Devel (E-mail)
Subject: Re: [Xen-devel] Time goes backwards in dom0 in xen-unstableQuoted 11 lines · Dan Magenheimer
I'm seeing some "Time went backwards" errors reported in dom0 in near-tip (c/s 19515) xen-unstable build. It's rare and random and not reproducible, but here's the report that just showed up. There was no load running at the time.
I can move to 3.4-rc1 if that would be helpful but I don't remember seeing any time-related changes recently.
This was on my dual-core test machine which reports lots of power management info during Xen boot.
If you specify 'cpuidle=off' as a Xen boot parameter, does that make the
timer_interruptscalability problem go away, and this time backwards problem? I was going to enable by default in 3.4 but could go the other way.-- Keir
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Double hmmm... after destroying the domain and restarting it has been running fine for over two hours. So apparently it was a coincidence.
Quoted 82 lines
-----Original Message-----
From: Dan Magenheimer
Sent: Wednesday, April 15, 2009 8:59 AM
To: Dan Magenheimer; Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstableHmmm... after only a few minutes with cpuidle=off, my test domPV froze up after printing a number of call traces starting with:
INFO: task xxx:nnn blocked for more than 480 seconds.At the top of all of the traces is either
getnstimeofday+51 or io_schedule+44.(Note that this PV domain is a 2.6.29 kernel... don't know if the messages are the same on an older kernel.)
Quoted 19 lines
-----Original Message-----
From: Dan Magenheimer
Sent: Wednesday, April 15, 2009 8:22 AM
To: Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstableI can confirm that cpuidle=off makes the
timer_interruptscaleability problem go away.It also appears that the max cycles for the MSI interrupts becomes reasonably small again. Was this expected?
I'll leave it running for awhile but may not be able to confirm the "Time went backwards" error goes away as it seemed to appear only after a random long period of time.
Quoted 10 lines
(BTW, Kevin, hpetbroadcast did not make the problem go away.)
Dan
Quoted 5 lines
-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Wednesday, April 15, 2009 1:31 AM
To: Dan Magenheimer; Xen-Devel (E-mail)
Subject: Re: [Xen-devel] Time goes backwards in dom0 inxen-unstable
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines · Dan Magenheimer
I'm seeing some "Time went backwards" errors reported in dom0 in near-tip (c/s 19515) xen-unstable build. It's rare and random and not reproducible, but here's the report that just showed up. There was no load running at the time.
I can move to 3.4-rc1 if that would be helpful but I don't remember seeing any time-related changes recently.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
This was on my dual-core test machine which reports lots of power management info during Xen boot.
If you specify 'cpuidle=off' as a Xen boot parameter, does that make the
timer_interruptscalability problem go away, and thistime backwards
Quoted 12 lines
Quoted 12 lines
problem? I was going to enable by default in 3.4 but could go the other way.
-- Keir
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Back to the first hmmm... The same problem arose after about three hours. Exactly the same symptoms. Note that my test VM has 4 vcpus but is otherwise unremarkable.
I'll try again with tip (post-rc2) tomorrow.
Quoted 100 lines
-----Original Message-----
From: Dan Magenheimer
Sent: Wednesday, April 15, 2009 11:12 AM
To: Dan Magenheimer; Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstableDouble hmmm... after destroying the domain and restarting it has been running fine for over two hours. So apparently it was a coincidence.
Quoted 11 lines
-----Original Message-----
From: Dan Magenheimer
Sent: Wednesday, April 15, 2009 8:59 AM
To: Dan Magenheimer; Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstableHmmm... after only a few minutes with cpuidle=off, my test domPV froze up after printing a number of call traces starting with:
Quoted 13 lines
INFO: task xxx:nnn blocked for more than 480 seconds.At the top of all of the traces is either
getnstimeofday+51 or io_schedule+44.(Note that this PV domain is a 2.6.29 kernel... don't know if the messages are the same on an older kernel.)
Quoted 5 lines
-----Original Message-----
From: Dan Magenheimer
Sent: Wednesday, April 15, 2009 8:22 AM
To: Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
Subject: RE: [Xen-devel] Time goes backwards in dom0 inxen-unstable
Quoted 39 lines
Quoted 14 lines
I can confirm that cpuidle=off makes the
timer_interruptscaleability problem go away.It also appears that the max cycles for the MSI interrupts becomes reasonably small again. Was this expected?
I'll leave it running for awhile but may not be able to confirm the "Time went backwards" error goes away as it seemed to appear only after a random long period of time.
Quoted 10 lines
(BTW, Kevin, hpetbroadcast did not make the problem go away.)
Dan
Quoted 5 lines
-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Wednesday, April 15, 2009 1:31 AM
To: Dan Magenheimer; Xen-Devel (E-mail)
Subject: Re: [Xen-devel] Time goes backwards in dom0 inxen-unstable
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines · Dan Magenheimer
I'm seeing some "Time went backwards" errors reported in dom0 in near-tip (c/s 19515) xen-unstable build. It's rare and random and not reproducible, but here's the report that just showed up. There was no load running at the time.
I can move to 3.4-rc1 if that would be helpful but I don't remember seeing any time-related changes recently.
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
This was on my dual-core test machine which reports lots of power management info during Xen boot.
If you specify 'cpuidle=off' as a Xen boot parameter, does that make the
timer_interruptscalability problem go away, and thistime backwards
Quoted 12 lines
Quoted 12 lines
problem? I was going to enable by default in 3.4 but could go the other way.
-- Keir
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel