dynticks: CONFIG_VIRT_CPU_ACCOUNTING + CONFIG_CONTEXT_TRACKING breaks accounting on core2 CPUs only
Greetings,
Turning on new NO_HZ feature on my Q6600 box in master, I see that tasks accrue zero utime/stime. However, the same exact kernel on E5620 box works fine, so it would appear there's a CPU dependency somewhere.
Is core2 expected to go dysfunctional with context tracking enabled? CONFIG_VIRT_CPU_ACCOUNTING alone works fine in 3.9-stable, turn on CONFIG_CONTEXT_TRACKING_FORCE, and CPU accounting stops working on core2 boxen only, same exact kernel continues to work just fine on E5620 (Westmere) box.
-Mike
marge:/usr/local/src/kernel/linux-3.9 # egrep 'NO_HR|CPU_ACCOUNTING|RCU| CONTEXT' .configCONFIG_VIRT_CPU_ACCOUNTING=y
# CONFIG_TICK_CPU_ACCOUNTING is not set CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
# RCU SubsystemCONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not setCONFIG_RCU_STALL_COMMON=yCONFIG_CONTEXT_TRACKING=y
# CONFIG_RCU_USER_QS is not set CONFIG_CONTEXT_TRACKING_FORCE=yCONFIG_RCU_FANOUT=64CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_RCU_FAST_NO_HZ is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_RCU_NOCB_CPU is not set CONFIG_HAVE_CONTEXT_TRACKING=y
# RCU Debugging
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set CONFIG_RCU_CPU_STALL_TIMEOUT=21
# CONFIG_RCU_CPU_STALL_INFO is not set
# CONFIG_RCU_TRACE is not setCONFIG_CONTEXT_SWITCH_TRACER=y
Quoted 5 lines · Mike Galbraith
Greetings,
Turning on new
NO_HZfeature on my Q6600 box in master, I see that tasks accrue zero utime/stime. However, the same exact kernel on E5620 box works fine, so it would appear there's a CPU dependency somewhere.
Ah indeed, I just managed to reproduce the same issue.
Quoted 6 lines
Is core2 expected to go dysfunctional with context tracking enabled?
CONFIG_VIRT_CPU_ACCOUNTINGalone works fine in 3.9-stable, turn onCONFIG_CONTEXT_TRACKING_FORCE, and CPU accounting stops working on core2 boxen only, same exact kernel continues to work just fine on E5620 (Westmere) box.
There was no known issue with core2. The box where I'm seeing the it is a Phenom quad core that had NR_CPUS=2. May be the issue is more likely to happen with this low number. I don't know.
I'm investigating further.
Thanks.
Quoted 16 lines · Frederic Weisbecker
Quoted 5 lines · Mike Galbraith
Greetings,
Turning on new
NO_HZfeature on my Q6600 box in master, I see that tasks accrue zero utime/stime. However, the same exact kernel on E5620 box works fine, so it would appear there's a CPU dependency somewhere.Ah indeed, I just managed to reproduce the same issue.
Quoted 6 lines
Is core2 expected to go dysfunctional with context tracking enabled?
CONFIG_VIRT_CPU_ACCOUNTINGalone works fine in 3.9-stable, turn onCONFIG_CONTEXT_TRACKING_FORCE, and CPU accounting stops working on core2 boxen only, same exact kernel continues to work just fine on E5620 (Westmere) box.
Quoted 5 lines
There was no known issue with core2. The box where I'm seeing the it is a Phenom quad core that had NR_CPUS=2. May be the issue is more likely to happen with this low number. I don't know.
I'm investigating further.
Me too.
bash-6023 [001] d... 290.494214: vtime_delta: clock: 289702961236 vtime_snap: 290493017701
Always. Not good.
I see..
current->vtime_snap = sched_clock();
and..
clock = local_clock();
Things that make ya go hmm. The below "fixes" it (not).
kernel/sched/cputime.c +5 -2
@@ -634,14 +634,17 @@ void thread_group_cputime_adjusted(struct task_struct *p, cputime_t *ut, cputime#endif /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN -static unsigned long long vtime_delta (struct task_struct *tsk) +static noinline unsigned long long vtime_delta (struct task_struct *tsk) {unsigned long long clock; - clock =local_clock (); +// clock = local_clock(); + clock =sched_clock (); +trace_printk ("clock: %Lu vtime_snap: %Lu\n" , clock, tsk->vtime_snap);if (clock < tsk->vtime_snap)return 0 ; +trace_printk ("clock: %Lu vtime_snap: %Lu returns :%Lu\n" , clock, tsk->vtime_snap, clock - tsk->vtime_snap);return clock - tsk->vtime_snap; }
Quoted 16 lines · Frederic Weisbecker
Quoted 5 lines · Mike Galbraith
Greetings,
Turning on new
NO_HZfeature on my Q6600 box in master, I see that tasks accrue zero utime/stime. However, the same exact kernel on E5620 box works fine, so it would appear there's a CPU dependency somewhere.Ah indeed, I just managed to reproduce the same issue.
Quoted 6 lines
Is core2 expected to go dysfunctional with context tracking enabled?
CONFIG_VIRT_CPU_ACCOUNTINGalone works fine in 3.9-stable, turn onCONFIG_CONTEXT_TRACKING_FORCE, and CPU accounting stops working on core2 boxen only, same exact kernel continues to work just fine on E5620 (Westmere) box.
Quoted 5 lines
There was no known issue with core2. The box where I'm seeing the it is a Phenom quad core that had NR_CPUS=2. May be the issue is more likely to happen with this low number. I don't know.
I'm investigating further.
So with CONFIG_HAVE_UNSTABLE_SCHED_CLOCK, you can't mix sched_clock() (pure tsc) with local_clock()/sched_clock_cpu(cpu). The former is always quite a bit ahead of the later, so mixing clocks is a nogo on crusty old (but beloved) core2 box.
-Mike
Quoted 21 lines · Mike Galbraith
Quoted 20 lines · Frederic Weisbecker
Quoted 5 lines · Mike Galbraith
Greetings,
Turning on new
NO_HZfeature on my Q6600 box in master, I see that tasks accrue zero utime/stime. However, the same exact kernel on E5620 box works fine, so it would appear there's a CPU dependency somewhere.Ah indeed, I just managed to reproduce the same issue.
Quoted 6 lines
Is core2 expected to go dysfunctional with context tracking enabled?
CONFIG_VIRT_CPU_ACCOUNTINGalone works fine in 3.9-stable, turn onCONFIG_CONTEXT_TRACKING_FORCE, and CPU accounting stops working on core2 boxen only, same exact kernel continues to work just fine on E5620 (Westmere) box.There was no known issue with core2. The box where I'm seeing the it is a Phenom quad core that had NR_CPUS=2. May be the issue is more likely to happen with this low number. I don't know.
Quoted 6 lines
Quoted 1 line
I'm investigating further.
So with
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK, you can't mixsched_clock()(pure tsc) with local_clock()/sched_clock_cpu(cpu). The former is always quite a bit ahead of the later, so mixing clocks is a nogo on crusty old (but beloved) core2 box.
Right I have the same issue. So let's use local_clock() everywhere here, it takes care of unstable tsc.
Does the following fix the issue for you?
kernel/sched/cputime.c +2 -2
@@ -747,7 +748,7 @@ void arch_vtime_task_switch(struct task_struct *prev)write_seqlock (¤t->vtime_seqlock); current->vtime_snap_whence =VTIME_SYS ; - current->vtime_snap =sched_clock (); + current->vtime_snap =local_clock ();write_sequnlock (¤t->vtime_seqlock); }
@@ -757,7 +758,7 @@ void vtime_init_idle(struct task_struct *t)write_seqlock_irqsave (&t->vtime_seqlock, flags); t->vtime_snap_whence =VTIME_SYS ; - t->vtime_snap =sched_clock (); + t->vtime_snap =local_clock ();write_sequnlock_irqrestore (&t->vtime_seqlock, flags); }
Quoted 29 lines · Frederic Weisbecker
Quoted 21 lines · Mike Galbraith
Quoted 20 lines · Frederic Weisbecker
Quoted 5 lines · Mike Galbraith
Greetings,
Turning on new
NO_HZfeature on my Q6600 box in master, I see that tasks accrue zero utime/stime. However, the same exact kernel on E5620 box works fine, so it would appear there's a CPU dependency somewhere.Ah indeed, I just managed to reproduce the same issue.
Quoted 6 lines
Is core2 expected to go dysfunctional with context tracking enabled?
CONFIG_VIRT_CPU_ACCOUNTINGalone works fine in 3.9-stable, turn onCONFIG_CONTEXT_TRACKING_FORCE, and CPU accounting stops working on core2 boxen only, same exact kernel continues to work just fine on E5620 (Westmere) box.There was no known issue with core2. The box where I'm seeing the it is a Phenom quad core that had NR_CPUS=2. May be the issue is more likely to happen with this low number. I don't know.
Quoted 6 lines
Quoted 1 line
I'm investigating further.
So with
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK, you can't mixsched_clock()(pure tsc) with local_clock()/sched_clock_cpu(cpu). The former is always quite a bit ahead of the later, so mixing clocks is a nogo on crusty old (but beloved) core2 box.
Quoted 4 lines
Right I have the same issue. So let's use
local_clock()everywhere here, it takes care of unstable tsc.Does the following fix the issue for you?
Yeah, both can use sched_clock_cpu() instead though.
Quoted 22 lines
kernel/sched/cputime.c +2
-2@@ -747,7 +748,7 @@ void arch_vtime_task_switch(struct task_struct *prev)write_seqlock (¤t->vtime_seqlock); current->vtime_snap_whence =VTIME_SYS ; - current->vtime_snap =sched_clock (); + current->vtime_snap =local_clock ();write_sequnlock (¤t->vtime_seqlock); }@@ -757,7 +758,7 @@ void vtime_init_idle(struct task_struct *t)write_seqlock_irqsave (&t->vtime_seqlock, flags); t->vtime_snap_whence =VTIME_SYS ; - t->vtime_snap =sched_clock (); + t->vtime_snap =local_clock ();write_sequnlock_irqrestore (&t->vtime_seqlock, flags); }
Quoted 30 lines · Mike Galbraith
Quoted 29 lines · Frederic Weisbecker
Quoted 27 lines · Mike Galbraith
Quoted 16 lines · Frederic Weisbecker
Quoted 5 lines · Mike Galbraith
Greetings,
Turning on new
NO_HZfeature on my Q6600 box in master, I see that tasks accrue zero utime/stime. However, the same exact kernel on E5620 box works fine, so it would appear there's a CPU dependency somewhere.Ah indeed, I just managed to reproduce the same issue.
Quoted 6 lines
Is core2 expected to go dysfunctional with context tracking enabled?
CONFIG_VIRT_CPU_ACCOUNTINGalone works fine in 3.9-stable, turn onCONFIG_CONTEXT_TRACKING_FORCE, and CPU accounting stops working on core2 boxen only, same exact kernel continues to work just fine on E5620 (Westmere) box.Quoted 5 lines
There was no known issue with core2. The box where I'm seeing the it is a Phenom quad core that had NR_CPUS=2. May be the issue is more likely to happen with this low number. I don't know.
I'm investigating further.
So with
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK, you can't mixsched_clock()(pure tsc) with local_clock()/sched_clock_cpu(cpu). The former is always quite a bit ahead of the later, so mixing clocks is a nogo on crusty old (but beloved) core2 box.
Quoted 6 lines
Quoted 4 lines
Right I have the same issue. So let's use
local_clock()everywhere here, it takes care of unstable tsc.Does the following fix the issue for you?
Yeah, both can use
sched_clock_cpu()instead though.
Right, given that irqs are already disabled. I'm preparing the patch.
Thanks!