Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

The Xen hypervisor uses the TSC as a time source. 1st and 2nd generation AMD
Operton processors have TSCs that change with frequency under power management.
The Xen hypervisor currently cannot cope with the way the TSCs change and the time will regularly go backwards on multicore systems.

This patch disables PowerNow! on SMP/CMP 1st and 2nd generation

processors and their Athlon 64, Turion, and Sempron equivalents. 3rd generation
Opteron, Athlon 64, and Phenom parts have pstate invariant TSCs and do not
suffer from the time regression.

There is an optional override by passing

on
the kernel command line, for users who are willing to trade potential time
regression or drift for improved power usage.

Signed-off-by Mark Langsdorf <[email protected]>

diff -r 47916bd012c6 arch/i386/kernel/cpu/cpufreq/powernow-k8.c
--- a/arch/i386/kernel/cpu/cpufreq/powernow-k8.c Fri Jan 18
13:37:53 2008 +0000
+++ b/arch/i386/kernel/cpu/cpufreq/powernow-k8.c Tue Jan 22
11:14:47 2008 -0600

arch/i386/kernel/cpu/cpufreq/powernow-k8.c +8 -0

@@ -60,6 +60,7 @@ static cpumask_t cpu_core_map[1];
 static cpumask_t cpu_core_map[1];
 #endif
 
+static int xen_tsc_override;
 /* Return a frequency in MHz, given an input fid */
 static u32 find_freq_from_fid(u32 fid)
 {

@@ -1317,6 +1318,10 @@ static int __cpuinit powernowk8_init(voi
 			supported_cpus++;
 	}
 
+	if (!xen_tsc_override &&
+		disable_if_tsc_problems(cpu_family, supported_cpus))
+		return -ENODEV;
+
 	if (supported_cpus == num_online_cpus()) {
 		powernow_k8_cpu_preinit_acpi();
 		printk(KERN_INFO PFX "Found %d %s "

@@ -1340,5 +1345,8 @@ MODULE_DESCRIPTION("AMD Athlon 64 and Op
 MODULE_DESCRIPTION("AMD Athlon 64 and Opteron processor frequency
 driver.");
 MODULE_LICENSE("GPL");
 
+module_param(xen_tsc_override,  int, 0);
+MODULE_PARM_DESC(xen_tsc_override, "Allow Xen PowerNow! even if the
 TSCs are variable.  May cause time to go backwards.");
+

late_initcall(powernowk8_init); module_exit(powernowk8_exit);
diff -r 47916bd012c6 arch/i386/kernel/cpu/cpufreq/powernow-k8.h
--- a/arch/i386/kernel/cpu/cpufreq/powernow-k8.h Fri Jan 18
13:37:53 2008 +0000
+++ b/arch/i386/kernel/cpu/cpufreq/powernow-k8.h Tue Jan 22
11:14:47 2008 -0600

arch/i386/kernel/cpu/cpufreq/powernow-k8.h +3 -0

@@ -221,3 +221,21 @@ static inline void define_siblings(int c
 	cpu_set(0, cpu_sharedcore_mask[0]);
 }
 #endif
+
+#ifdef CONFIG_XEN
+static inline int disable_if_tsc_problems(unsigned int cpu_family,

unsigned int supported_cpus)

(quoted diff) +2 -0

+{
+	/* Disable support for multi-core RevF on Xen due to tsc issues

*/

arch/i386/kernel/cpu/cpufreq/powernow-k8.h +13 -0

+	if ((cpu_family == CPU_OPTERON) &&
+	    (supported_cpus > 1))
+		return -ENODEV;
+	else
+		return 0;
+}
+#else
+static inline int disable_if_tsc_problems()
+{
+	return 0;
+}
+#endif
+

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Quoted 6 lines

The Xen hypervisor uses the TSC as a time source. 1st and 2nd generation AMD
Operton processors have TSCs that change with frequency under power management.
The Xen hypervisor currently cannot cope with the way the TSCs change and the time will regularly go backwards on multicore systems.

Surely it should be possible to inform Xen of frequency changes (other that T-states due to thermal throttling) and hence allow Xen to ensure monotonicity? Xen itself does not require that the CPUs run with the same rate TSC. It does need to know what frequency they're running at, and when they change.

From:Keir Fraser
Sent: 2008年1月23日 5:52

Quoted 5 lines · Ian Pratt

Quoted 5 lines

The Xen hypervisor uses the TSC as a time source. 1st and 2nd generation AMD
Operton processors have TSCs that change with frequency under power management.
The Xen hypervisor currently cannot cope with the way the

TSCs change

Quoted 3 lines

Quoted 1 line

and the time will regularly go backwards on multicore systems.

Surely it should be possible to inform Xen of frequency

changes (other

Quoted 1 line

that T-states due to thermal throttling) and hence allow Xen

to ensure

Quoted 2 lines

monotonicity? Xen itself does not require that the CPUs run with the same rate TSC. It does need to know what frequency they're

running at,

Quoted 1 line

and when they change.

Yes, and that is already done, hooked off the cpu notifier callback chain.
Nobbling the C- and P-state logic shouldn't be necessary.

-- Keir

Quoted 1 line

Surely it should be possible to inform Xen of frequency

changes (other

Quoted 1 line

that T-states due to thermal throttling) and hence allow

Xen to ensure

Quoted 2 lines

monotonicity? Xen itself does not require that the CPUs run with the same rate TSC. It does need to know what frequency they're

running at,

Quoted 1 line

and when they change.

Yes, and that is already done, hooked off the cpu notifier callback chain.
Nobbling the C- and P-state logic shouldn't be necessary.

Quoted 3 lines

Yes, and that is already done, hooked off the cpu notifier callback chain.
Nobbling the C- and P-state logic shouldn't be necessary.

And yet, people are still reporting "Time went backwards" messages upwards of 30-40 times an hour on RevF Opterons when PowerNow! is enabled on Xen.

I spent a month trying to debug this and I made no significant progress. If someone who understands Xen timekeeping wants to help me dive into this, I'll be glad to take another stab at it. As it is, even with informing Xen of frequency changes, Xen does not ensure monotinicity.

Quoted 8 lines

Quoted 3 lines

Yes, and that is already done, hooked off the cpu notifier callback chain.
Nobbling the C- and P-state logic shouldn't be necessary.

And yet, people are still reporting "Time went backwards" messages upwards of 30-40 times an hour on RevF Opterons when PowerNow! is enabled on Xen.

Quoted 5 lines

I spent a month trying to debug this and I made no significant progress. If someone who understands Xen timekeeping wants to help me dive into this, I'll be glad to take another stab at it. As it is, even with informing Xen of frequency changes, Xen does not ensure monotinicity.

Seeing them at that frequency (less than one a minute) is not too serious.
Perhaps we should get rid of the message, or reduce its priority.

Quoted 3 lines

Seeing them at that frequency (less than one a minute) is not too serious.
Perhaps we should get rid of the message, or reduce its priority.

Red Hat engineering is requesting the disable because they've gotten STREAM results with the test finishing before it started.

For myself, I don't mind the frequency of the messages. In the real world, it's serious, especially on financial servers.

Quoted 5 lines

I spent a month trying to debug this and I made no significant progress. If someone who understands Xen timekeeping wants to help me dive into this, I'll be glad to take another stab at it. As it is, even with informing Xen of frequency changes, Xen does not ensure monotinicity.

Seeing them at that frequency (less than one a minute) is not too serious. Perhaps we should get rid of the message, or reduce its priority.

Quoted 2 lines

Seeing them at that frequency (less than one a minute) is not too serious. Perhaps we should get rid of the message, or reduce its priority.

They are most definitely serious --- gettimeofday() going backwards breaks Unix semantics. It breaks applications. It _will_ corrupt application data --- things like database rollback logs that rely on timestamp monotonicity to order things correctly will break.

Quoted 7 lines · Stephen C. Tweedie

Quoted 2 lines

Seeing them at that frequency (less than one a minute) is not too serious. Perhaps we should get rid of the message, or reduce its priority.

They are most definitely serious --- gettimeofday() going backwards breaks Unix semantics. It breaks applications. It _will_ corrupt application data --- things like database rollback logs that rely on timestamp monotonicity to order things correctly will break.

I thought Linux now enforced a monotonic clock the same way we were forced to - that is, globally synchronise and make sure we return last value + 1, even when the Xen clock goes backwards across CPUs, which it does regularly, powernow or not?

Quoted 2 lines

Seeing them at that frequency (less than one a minute) is not too serious. Perhaps we should get rid of the message, or reduce its priority.

They are most definitely serious --- gettimeofday() going backwards breaks Unix semantics. It breaks applications. It _will_ corrupt application data --- things like database rollback logs that rely on timestamp monotonicity to order things correctly will break.

I think this is something that Xen upstream ought to be pretty concerned about. If we can't fix it properly, then the workaround of disabling powernow seems to be a must-have.

Quoted 3 lines

Yes, and that is already done, hooked off the cpu notifier callback chain.
Nobbling the C- and P-state logic shouldn't be necessary.

And yet, people are still reporting "Time went backwards" messages upwards of 30-40 times an hour on RevF Opterons when PowerNow! is enabled on Xen.

I spent a month trying to debug this and I made no significant progress. If someone who understands Xen timekeeping wants to help me dive into this, I'll be glad to take another stab at it. As it is, even with informing Xen of frequency changes, Xen does not ensure monotinicity.

Quoted 2 lines

Quoted 2 lines

Yes, and that is already done, hooked off the cpu notifier callback chain. Nobbling the C- and P-state logic

Quoted 10 lines

And yet, people are still reporting "Time went backwards" messages upwards of 30-40 times an hour on RevF Opterons when PowerNow! is enabled on Xen.

I spent a month trying to debug this and I made no significant progress. If someone who understands Xen timekeeping wants to help me dive into this, I'll be glad to take another stab at it. As it is, even with informing Xen of frequency changes, Xen does not ensure monotinicity.

I do think we need to understand what's going on here.

Are the 'time went backwards' messages correlated with power management operations?

How far is time stepping backwards?
ns, us, or ms?

Jan 19 00:11:09 server kernel: printk: 14 messages suppressed.
Jan 19 00:11:09 server kernel: Timer ISR/0: Time went backwards:

delta=-123862858 delta_cpu=-123862858 shadow= 2993571302800 off=705014593 processed=2994400167600

Jan 19 00:11:09 server kernel:  0: 2994400167600
Jan 19 00:11:09 server kernel:  1: 2994213786880
Jan 19 00:11:09 server kernel: Timer ISR/1: Time went backwards:

delta=-50043804 delta_cpu=136336916 shadow=29 93571302800 off=778821372 processed=2994400167600

Jan 19 00:11:09 server kernel:  0: 2994400167600
Jan 19 00:11:09 server kernel:  1: 2994213786880
Jan 19 00:11:09 server kernel: Timer ISR/1: Time went backwards:

delta=-70033718 delta_cpu=66347002 shadow=299 3571302800 off=838831281 processed=2994480167600

Jan 19 00:11:09 server kernel:  0: 2994480167600
Jan 19 00:11:09 server kernel:  1: 2994343786880

This happens also if I switch manually with cpufreq-set -f 2.60GHz of cpufrequtils.

Are you sure the systems aren't overheating and performing emergency T-state thermal throttling?

Yes, I can regularly generate messages by switching the governor from powersave (minimum frequency) to performance (maximum frequency).

Quoted 2 lines

How far is time stepping backwards?
ns, us, or ms?

Rolf's message is typical:

Quoted 4 lines

Jan 19 00:11:09 server kernel: printk: 14 messages suppressed.
Jan 19 00:11:09 server kernel: Timer ISR/0: Time went backwards:

delta=-123862858 delta_cpu=-123862858 shadow= 2993571302800 off=705014593 processed=2994400167600

cpu_processed=2994400167600

It seems to happen most reliably when there's a large jump in frequency (2.6 GHz to 800 MHz or vice versa) but I've seen it happen for smaller switches. Usually there's 1 large backward movement, and then 2-4 smaller ones. It doesn't always happen, especially if there are multiple shifts in a short period (ie, constant changes on the ondemand governor) and it seems to happen more consistently when increasing frequency.

Quoted 7 lines

It seems to happen most reliably when there's a large jump in frequency (2.6 GHz to 800 MHz or vice versa) but I've seen it happen for smaller switches. Usually there's 1 large backward movement, and then 2-4 smaller ones. It doesn't always happen, especially if there are multiple shifts in a short period (ie, constant changes on the ondemand governor) and it seems to happen more consistently when increasing frequency.

Increasing the frequency requires (1) instructing the regulator to increasing the voltage, (2) waiting for it to settle, (3) increasing the multipler, (4) wait for the PLL to settle,
(5) unpause the CPU.

Which of these steps are under software control on PowerNow?

Since we probably don't know how long we had to wait between when the multipler was increased and when the CPU was un-paused, it will be necessary to recalibrate the CPU to the global time source (PIT/HPET).

Increasing the frequency requires (1) instructing the regulator to increasing the voltage, (2) waiting for it to settle, (3) increasing the multipler, (4) wait for the PLL to settle, (5) unpause the CPU.

Which of these steps are under software control on PowerNow?

Since we probably don't know how long we had to wait between when the multipler was increased and when the CPU was un-paused, it will be necessary to recalibrate the CPU to the global time source (PIT/HPET).

Quoted 3 lines · Ian Pratt

Since we probably don't know how long we had to wait between when the multipler was increased and when the CPU was un-paused, it will be necessary to recalibrate the CPU to the global time source (PIT/HPET).

Yes, actually that is a good point. Possibly we would get better results by setting t->stime_local_stamp = t->stime_master_stamp = read_platform_stime() in xen/arch/x86/time.c:cpu_frequency_change(). Currently we set the local stamp to get_s_time(), but get_s_time() may not be trustworthy.

I tried this suggestion and also implemented the trace function in cpu_frequency_change() and local_time_calibration() you suggested in a previous message.

It made no discernable difference in my tests. I did high to low and low to high changes roughly every five minutes, and always got an excessive difference inside of Xen and usually got error messages in Linux.

Quoted 9 lines · Mark Langsdorf

I tried this suggestion and also implemented the trace function in cpu_frequency_change() and local_time_calibration() you suggested in a previous message.

It made no discernable difference in my tests. I did high to low and low to high changes roughly every five minutes, and always got an excessive difference inside of Xen and usually got error messages in Linux.

It's a bit confusing as to what is going on since there are 4 CPUs in the system. Are arbitrary or all CPUs changing frequency, or just one of them? Is the single-step PSTATE_CTRL MSR being used, or the multi-step FID_VID method where voltage adjustments are under software control? What is the system's CPU topology, and do any of the cores have their frequency controls tied together?

It'd be useful to have each tracing line stamped with the CPU that it comes from. As it is the tracing looks very suspect because the systimes on adjacent lines are much less than one second, and calibration is supposed to happen no more than once per second.

-- Keir

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Are we in fact talking about the new Barcelona chips? If so, their frequency settings are independent per core. Unless I have misunderstood the question.

Shingoshi

Keir Fraser wrote:

	On 25/1/08 22:58, "Mark Langsdorf" <[email protected]>

<mailto:[email protected]> wrote:

I tried this suggestion and also implemented the trace
function in
cpu_frequency_change() and local_time_calibration() you
suggested in
a previous message.

It made no discernable difference in my tests. I did

to low
and low to high changes roughly every five minutes, and
always got
an excessive difference inside of Xen and usually got

messages
in Linux.

It's a bit confusing as to what is going on since there are 4
CPUs in the
system. Are arbitrary or all CPUs changing frequency, or just

of them?
Is the single-step PSTATE_CTRL MSR being used, or the multi-step
FID_VID
method where voltage adjustments are under software control?

is the
system's CPU topology, and do any of the cores have their
frequency controls
tied together?

It'd be useful to have each tracing line stamped with the CPU
that it comes
from. As it is the tracing looks very suspect because the
systimes on
adjacent lines are much less than one second, and calibration is
supposed to

	happen no more than once per second.

	 -- Keir



	_______________________________________________
	Xen-devel mailing list
	[email protected]
	http://lists.xensource.com/xen-devel

Quoted 4 lines

It made no discernable difference in my tests. I did high to low and low to high changes roughly every five minutes, and always got an excessive difference inside of Xen and usually got error messages in Linux.

It's a bit confusing as to what is going on since there are 4 CPUs in the system.

Are arbitrary or all CPUs changing frequency, or just one of them?

Is the single-step PSTATE_CTRL MSR being used, or the multi-step FID_VID method where voltage adjustments are under software control?

It'd be useful to have each tracing line stamped with the CPU that it comes from.

It'd be useful to have each tracing line stamped with the CPU that it comes from. As it is the tracing looks very suspect because the systimes on adjacent lines are much less than one second, and calibration is supposed to happen no more than once per second.

Quoted 5 lines

It'd be useful to have each tracing line stamped with the CPU that it comes from. As it is the tracing looks very suspect because the systimes on adjacent lines are much less than one second, and calibration is supposed to happen no more than once per second.

Keir -

There seems to be a bug in continue_hypercall_on_cpu(). I'm getting vcpu_set_affinity() failures with error code -22. Any idea what that is?

vcpu_set_affinity() needs to not check dom0_vcpus_pin when called from continue_hypercall_on_cpu(). I will cook up a patch...

Please test with latest xen-unstable. As of changeset 16897 I have fixed the vcpu affinity change, so that should actually work,

Quoted 3 lines

Please test with latest xen-unstable. As of changeset 16897 I have fixed the vcpu affinity change, so that should actually work,

It doesn't, for some reason. It seems to lock up on the

v->arch.schedule_tail = continue_hypercall_on_cpu_helper

for no reason that I can tell. If I comment out that line, the system boots fine, but if that is in Xen, the machine softlocks shortly after starting cpufreq. Unfortunately, cpufreq starts well before any console, so I can't get a lot of information out the system.

I'll continue working on this, but any advance would be appreciated.

-Mark Langsdorf
Operating System Research Center
AMD

Quoted 3 lines

Please test with latest xen-unstable. As of changeset 16897 I have fixed the vcpu affinity change, so that should actually work,

It doesn't, for some reason. It seems to lock up on the

v->arch.schedule_tail = continue_hypercall_on_cpu_helper

for no reason that I can tell. If I comment out that line, the system boots fine, but if that is in Xen, the machine softlocks shortly after starting cpufreq. Unfortunately, cpufreq starts well before any console, so I can't get a lot of information out the system.

I'll continue working on this, but any advance would be appreciated.

As promised I've instrumented and kicked that code path, and I execute it with no problems, with the VCPU concerned being correctly migrated before and after the continue_hypercall_on_cpu() is executed. I've tried it with 'dom0_vcpus_pin' and also I've tried making fairly extreme calls to cpu_frequency_change() inside the hypercall-continuation 'helper' function. None of this causes my dom0 boot to hang. So it looks like something a bit subtle must be going wrong on your test system.

Quoted 7 lines · Keir Fraser

As promised I've instrumented and kicked that code path, and I execute it with no problems, with the VCPU concerned being correctly migrated before and after the continue_hypercall_on_cpu() is executed. I've tried it with 'dom0_vcpus_pin' and also I've tried making fairly extreme calls to cpu_frequency_change() inside the hypercall-continuation 'helper' function. None of this causes my dom0 boot to hang. So it looks like something a bit subtle must be going wrong on your test system.

Actually I've now eyeballed the problem. We access a stack variable after the stack frame is gone away. Probably we therefore end up setting a CPU's frequency to zero, which is not good at all. I'll work up a fix.

Quoted 5 lines

Actually I've now eyeballed the problem. We access a stack variable after the stack frame is gone away. Probably we therefore end up setting a CPU's frequency to zero, which is not good at all. I'll work up a fix.

Changeset 16927 should fix the problem for you.

Quoted 4 lines

Quoted 3 lines

Actually I've now eyeballed the problem. We access a stack variable after the stack frame is gone away. Probably we therefore end up setting a CPU's frequency to zero, which
is not good at all. I'll work up a fix.

Changeset 16927 should fix the problem for you.

16927 seems to have resolved the problem. I'll be running a stress test in the background all day, but the prelimary results show that:

a) the machine isn't locking up on boot any more
b) pstate transitions are possible without having the time go backwards on a regular basis

Ian was right about the design of Xen's time-keeping.

-Mark Langsdorf
Operating System Research Center
AMD

PS - Ralk, could you please test this on your machine?

Changeset 16927 should fix the problem for you.

Quoted 1 line

Changeset 16927 should fix the problem for you.

I'm getting about 5 "Time went backwards" messages a day right now, which is a drastic improvement but still more than I'd like. In three instances, the shadow time value does not increase.

The weird thing is the cpu time values are always of the form:
0: xxxxxx7444952
1: yyyyyy7447746
2: yyyyyy7447746
3: yyyyyy7447746

the y values are usually but not always the same as x or slightly higher. When they are different, they differ by 1000000, 2000000, or 3000000.

I suspect it is related to the fact that platform calibration is only performed if smp_processor_id() == 0, but I can't figure out why this is occurring.

-Mark Langsdorf
Operating System Research Center
AMD

What are the deltas when time goes backwards? Bear in mind we already default to being silent for backwards deltas up to 10ms.

The issue here could well be that if you take a timer interrupt towards the
end of a frequency change but before Xen has been told that the frequency
has changed, the time values that Xen and dom0 calculate will be off by
potentially quite a bit. It's a small window of opportunity, but clearly you
hit it a few times each day. I'm not sure how easy it is to close off that
window cleanly with the MSR twiddling being controlled from dom0...

Would turning off interrupts solve it? It might be a bit brute force, but there's certainly a prechange notifier event and a postchange notifier event to use to turn them on and off.

Alternately, the prechange event could sleep until woken by the timer interrupt, which would increase the chance that all frequency changes take place at the start of an interrupt.

I tried this suggestion and also implemented the trace function in cpu_frequency_change() and local_time_calibration() you suggested in a previous message.

It made no discernable difference in my tests. I did high to low and low to high changes roughly every five minutes, and always got an excessive difference inside of Xen and usually got error messages in Linux.

Changed code follows and the xm dmesg log is attached. diff -r 62fc84adc8ed xen/arch/x86/time.c

xen/arch/x86/time.c +2 -0

@@ -724,6 +724,20 @@ void update_domain_wallclock_time(struct
     spin_unlock(&wc_lock);
 }
 
+void checktime(char *buf);
+void checktime(char *buf)

xen/arch/x86/time.c +2 -0

+%lu, systime %lu\n", buf, diff, p_tmp, s_tmp); }
+
 int cpu_frequency_change(u64 freq)
 {
     struct cpu_time *t = &this_cpu(cpu_time); @@ -732,11 +746,14 @@

int cpu_frequency_change(u64 freq)

     local_irq_disable();
     rdtscll(curr_tsc);
     t->local_tsc_stamp = curr_tsc;

xen/arch/x86/time.c +5 -1

-    t->stime_local_stamp = get_s_time();
+//    t->stime_local_stamp = get_s_time();
     t->stime_master_stamp = read_platform_stime();
+    t->stime_local_stamp = t->stime_master_stamp;
     set_time_scale(&t->tsc_scale, freq);
+checktime("frequency_change");
     local_irq_enable();
 
+printk(KERN_ERR "freq change at %lu\n", t->stime_master_stamp);
     /* A full epoch should pass before we check for deviation. */
     set_timer(&t->calibration_timer, NOW() + EPOCH);
     if ( smp_processor_id() == 0 )

@@ -808,6 +825,7 @@ static void local_time_calibration(void
     rdtscll(curr_tsc);
     curr_local_stime  = get_s_time();
     curr_master_stime = read_platform_stime();
+checktime("calibration");
     local_irq_enable();
 
 #if 0

Quoted 4 lines

I tried this suggestion and also implemented the trace function in cpu_frequency_change() and local_time_calibration() you suggested in a previous message.

Please provide the output of the tracing.

Presumably the time-goes-backwards messages only come out when stepping frequency down?

Presumably it is the CPU that is down-stepping that thinks time has gone backwards?

The messages comes out shortly after the downstep?

Once the multiplier change has happened, you need to create a new time record for the CPU based on the global clock PIT/HPET (and update the frequency info in the record).

Implemented correctly, this really ought to work just fine.

Quoted 2 lines

Are the 'time went backwards' messages correlated with power management operations?

Yes, I can regularly generate messages by switching the governor from powersave (minimum frequency) to performance (maximum frequency).

Quoted 2 lines

How far is time stepping backwards?
ns, us, or ms?

Rolf's message is typical:

Quoted 4 lines

Jan 19 00:11:09 server kernel: printk: 14 messages suppressed.
Jan 19 00:11:09 server kernel: Timer ISR/0: Time went backwards:

delta=-123862858 delta_cpu=-123862858 shadow= 2993571302800 off=705014593 processed=2994400167600