remove PV/x86 warnings (time went backwards)

RFC PATCH In review Atsushi SAKAI 4 participants 7 messages Dec 11, 2008

lore

Share & tools

lore

Atsushi SAKAI

author

2008-12-11 08:02 UTC #0 lore raw

RFC PATCH 1/1 remove PV/x86 warnings (time went backwards)

Hi,

This patch intends to remove warnings for Time went backwards (negative delta) like follows.
========from Xen BZ 195
Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 shadow=706605145 0: 71640046095
========

Currently (static u64) [email protected]
uses for calculating jiffies and wall_clock.
and the processed_system_time saves maximum vcpu time value of per_cpu(processed_system_time).

At [email protected],
they calculate "delta" compared to vcpu time(shadow). sometimes the value goes to negative.
This is because vcpu time is varies.(scheduler etc.)

This patch just removes the warnings of PV/x86,
(Other way is loosen "delta" checking.(current checking is < -10msec)
Or vcpu synchronization is needed.)

This problem occurs on current linux-2.6.18-xen tree only. Not occured on pv_ops kernel.

Of course, delta_cpu comparison keeps
since it is just compares per_cpu variable only.

==for paravirt_ops ==
For paravirt_ops, time.c is completely changed.
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=arch/x86/xen/time.c;h=c9f7cda48ed78ecbe1b421c540bbe2b777681400;hb=HEAD

See also
2006/07(which uses delta)
http://lists.xensource.com/archives/html/xen-devel/2006-07/msg00567.html
2007/03(which does not uses delta)
http://lists.xensource.com/archives/html/xen-devel/2007-03/msg00554.html

== Related Bugzilla ==
This patch related to following bugzilla.

XenBugzilla:
Many Timer ISR: Time went backwards logs
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=195

RedHat Bugzilla:
Many "Timer ISR: Time went backwards" after booting xen kernel
https://bugzilla.redhat.com/show_bug.cgi?id=222973

Signed-off-by Atsushi SAKAI <[email protected]>

Thanks
Atsushi SAKAI

arch/i386/kernel/time-xen.c +1 -2

diff -r 8e3025f94790 arch/i386/kernel/time-xen.c
@@ -674,8 +674,7 @@ irqreturn_t timer_interrupt(int irq, voi
 		} while (sched_time != runstate->state_entry_time);
 	} while (!time_values_up_to_date(cpu));
 
-	if ((unlikely(delta < -(s64)permitted_clock_jitter) ||
-	     unlikely(delta_cpu < -(s64)permitted_clock_jitter))
+	if (unlikely(delta_cpu < -(s64)permitted_clock_jitter)
 	    && printk_ratelimit()) {
 		printk("Timer ISR/%u: Time went backwards: "
 		       "delta=%lld delta_cpu=%lld shadow=%lld "

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Tian, Kevin

2008-12-11 08:17 UTC #1 lore raw

"Time went backward" warnings are indicator that something goes wrong. It's not a solution to remove the warning while leaving actual reason unsolved.

Which release are you trying? Does it happen on up-to-date xen unstable?

Thanks,
Kevin

Quoted 65 lines

From: Atsushi SAKAI
Sent: Thursday, December 11, 2008 4:03 PM
Hi,
This patch intends to remove warnings for Time went backwards (negative delta) like follows.
========from Xen BZ 195
Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 shadow=706605145 0: 71640046095
========
Currently (static u64) [email protected] uses for calculating jiffies and wall_clock.
and the processed_system_time saves maximum vcpu time value of per_cpu(processed_system_time).
At [email protected],
they calculate "delta" compared to vcpu time(shadow). sometimes the value goes to negative.
This is because vcpu time is varies.(scheduler etc.)
This patch just removes the warnings of PV/x86,
(Other way is loosen "delta" checking.(current checking is < -10msec) Or vcpu synchronization is needed.)
This problem occurs on current linux-2.6.18-xen tree only. Not occured on pv_ops kernel.
Of course, delta_cpu comparison keeps
since it is just compares per_cpu variable only.
==for paravirt_ops ==
For paravirt_ops, time.c is completely changed.
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob
;f=arch/x86/xen/time.c;h=c9f7cda48ed78ecbe1b421c540bbe2b7776814 00;hb=HEAD
See also
2006/07(which uses delta)
http://lists.xensource.com/archives/html/xen-devel/2006-07/msg0
0567.html
2007/03(which does not uses delta)
http://lists.xensource.com/archives/html/xen-devel/2007-03/msg0
0554.html
== Related Bugzilla ==
This patch related to following bugzilla.
XenBugzilla:
Many Timer ISR: Time went backwards logs
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=195
RedHat Bugzilla:
Many "Timer ISR: Time went backwards" after booting xen kernel
https://bugzilla.redhat.com/show_bug.cgi?id=222973
Signed-off-by Atsushi SAKAI <[email protected]>
Thanks
Atsushi SAKAI

Keir Fraser

2008-12-11 09:22 UTC #2 lore raw

In reply to Atsushi SAKAI

Quoted 7 lines · Atsushi SAKAI

This patch intends to remove warnings for Time went backwards (negative delta) like follows.
========from Xen BZ 195
Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 shadow=706605145 0: 71640046095
========

That might be considered skirting around the problem. Are users still seeing this with xen 3.3 and xen unstable?

-- Keir

Jan Beulich

2008-12-11 10:31 UTC #3 lore raw

Re: [PATCH][RFC] remove PV/x86 warnings (time wentbackwards)

Quoted 11 lines

Quoted 1 line
Quoted 1 line
Keir Fraser <[email protected]> 11.12.08 10:22 >>>
Quoted 7 lines · Atsushi SAKAI
This patch intends to remove warnings for Time went backwards (negative delta) like follows.
========from Xen BZ 195
Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 shadow=706605145 0: 71640046095
========

Quoted 2 lines

That might be considered skirting around the problem. Are users still seeing this with xen 3.3 and xen unstable?

Unfortunately yes, but rather seldom these days. We just got a fresh report about a larger IBM system suffering from it - while the initial report they sent had hundreds of milliseconds in them, the most up-to-date one we have shows that it's just a few milliseconds above the -10ms boundary. So far I haven't been able to figure what causes this.

Jan

Atsushi SAKAI

author

2008-12-11 11:00 UTC #4 lore raw

Re: [PATCH][RFC] remove PV/x86 warnings (timewentbackwards)

Hi, Keir, Kevin, and Jan

Thank you for various suggestions.

One more questions on this.
I think possible solution is
Sync vcpu0 per_cpu(processed_system_time)
to static u64 processed_system_time on PV kernel.

It must be solved on hypervisor layer in this case?

Thanks
Atsushi SAKAI

Quoted 27 lines · Jan Beulich

Quoted 11 lines
Quoted 1 line
Quoted 1 line
Keir Fraser <[email protected]> 11.12.08 10:22 >>>
Quoted 7 lines · Atsushi SAKAI
This patch intends to remove warnings for Time went backwards (negative delta) like follows.
========from Xen BZ 195
Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 shadow=706605145 0: 71640046095
========
Quoted 2 lines
That might be considered skirting around the problem. Are users still seeing this with xen 3.3 and xen unstable?
Unfortunately yes, but rather seldom these days. We just got a fresh report about a larger IBM system suffering from it - while the initial report they sent had hundreds of milliseconds in them, the most up-to-date one we have shows that it's just a few milliseconds above the -10ms boundary. So far I haven't been able to figure what causes this.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Keir Fraser

2008-12-11 11:03 UTC #5 lore raw

Re: [PATCH][RFC] remove PV/x86 warnings (time wentbackwards)

In reply to Jan Beulich

Quoted 8 lines · Jan Beulich

Quoted 2 lines
That might be considered skirting around the problem. Are users still seeing this with xen 3.3 and xen unstable?
Unfortunately yes, but rather seldom these days. We just got a fresh report about a larger IBM system suffering from it - while the initial report they sent had hundreds of milliseconds in them, the most up-to-date one we have shows that it's just a few milliseconds above the -10ms boundary. So far I haven't been able to figure what causes this.

Perhaps we should expand the default acceptable jitter a bit? Still, I'd hoped that 10ms would be plenty!

-- Keir

Jan Beulich

2008-12-11 11:12 UTC #6 lore raw

Re: [PATCH][RFC] remove PV/x86 warnings (timewentbackwards)

Quoted 6 lines

Quoted 1 line
Quoted 1 line
Keir Fraser <[email protected]> 11.12.08 12:03 >>>
Quoted 3 lines · Jan Beulich
Quoted 2 lines
That might be considered skirting around the problem. Are users still seeing this with xen 3.3 and xen unstable?

Quoted 8 lines

Quoted 5 lines
Unfortunately yes, but rather seldom these days. We just got a fresh report about a larger IBM system suffering from it - while the initial report they sent had hundreds of milliseconds in them, the most up-to-date one we have shows that it's just a few milliseconds above the -10ms boundary. So far I haven't been able to figure what causes this.
Perhaps we should expand the default acceptable jitter a bit? Still, I'd hoped that 10ms would be plenty!

So would I (and really I find 10ms pretty wide already), which is why I never proposed an adjustment here, despite getting reports about these messages popping up every once in a while. And then again, it's configurable on the command line and via sysctl, so if someone really just wants the messages go away, he can already do that.

Jan