Skip to contentSkip to conversation

lots of brief rcu stalls.

DISCUSSION Dave Jones 4 participants 11 messages Dec 4, 2013 – Dec 5, 2013 RCU

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

Paul,
I'm seeing this happening more and more lately...

[  771.786462] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  771.786552] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786574] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786595] 	(detected by 0, t=6502 jiffies, g=20611, c=20610, q=0)
[  771.786620] INFO: Stall ended before state dump start
[  966.724546] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  966.724854] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.724931] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.725005] 	(detected by 0, t=26007 jiffies, g=20611, c=20610, q=0)
[  966.725093] INFO: Stall ended before state dump start
[ 1161.661459] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1161.661763] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661840] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661915] 	(detected by 0, t=45512 jiffies, g=20611, c=20610, q=0)
[ 1161.662001] INFO: Stall ended before state dump start
[ 1356.598205] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1356.598513] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598590] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598664] 	(detected by 0, t=65017 jiffies, g=20611, c=20610, q=0)
[ 1356.598751] INFO: Stall ended before state dump start
[ 1551.536099] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1551.536408] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536485] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536559] 	(detected by 0, t=84522 jiffies, g=20611, c=20610, q=0)
[ 1551.536645] INFO: Stall ended before state dump start

While it's apparently a non-problem, it's pretty noisy. Any ideas?

Dave

#1 lore raw
Quoted 29 lines · Dave Jones

Paul,
I'm seeing this happening more and more lately...

[  771.786462] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  771.786552] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786574] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786595] 	(detected by 0, t=6502 jiffies, g=20611, c=20610, q=0)
[  771.786620] INFO: Stall ended before state dump start
[  966.724546] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  966.724854] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.724931] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.725005] 	(detected by 0, t=26007 jiffies, g=20611, c=20610, q=0)
[  966.725093] INFO: Stall ended before state dump start
[ 1161.661459] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1161.661763] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661840] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661915] 	(detected by 0, t=45512 jiffies, g=20611, c=20610, q=0)
[ 1161.662001] INFO: Stall ended before state dump start
[ 1356.598205] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1356.598513] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598590] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598664] 	(detected by 0, t=65017 jiffies, g=20611, c=20610, q=0)
[ 1356.598751] INFO: Stall ended before state dump start
[ 1551.536099] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1551.536408] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536485] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536559] 	(detected by 0, t=84522 jiffies, g=20611, c=20610, q=0)
[ 1551.536645] INFO: Stall ended before state dump start
Quoted 2 lines

While it's apparently a non-problem, it's pretty noisy. Any ideas?

Does the following help?

Thanx, Paul

------------------------------------------------------------------------

rcu: Kick CPU halfway to RCU CPU stall warning

When an RCU CPU stall warning occurs, the CPU invokes resched_cpu() on itself. This can help move the grace period forward in some situations, but it would be even better to do this -before- the RCU CPU stall warning. This commit therefore causes resched_cpu() to be called every five jiffies once the system is halfway to an RCU CPU stall warning.

kernel/rcu/tree.c +25 -1
index dd081987a8ec..5243ebea0fc1 100644
@@ -755,6 +755,12 @@ static int dyntick_save_progress_counter(struct rcu_data *rdp,
 }
 
 /*
+ * This function really isn't for public consumption, but RCU is special in
+ * that context switches can allow the state machine to make progress.
+ */
+extern void resched_cpu(int cpu);
+
+/*
  * Return true if the specified CPU has passed through a quiescent
  * state by virtue of being in or having passed through an dynticks
  * idle state since the last call to dyntick_save_progress_counter()
@@ -812,16 +818,34 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
 	 */
 	rcu_kick_nohz_cpu(rdp->cpu);
 
+	/*
+	 * Alternatively, the CPU might be running in the kernel
+	 * for an extended period of time without a quiescent state.
+	 * Attempt to force the CPU through the scheduler to gain the
+	 * needed quiescent state, but only if the grace period has gone
+	 * on for an uncommonly long time.  If there are many stuck CPUs,
+	 * we will beat on the first one until it gets unstuck, then move
+	 * to the next.  Only do this for the primary flavor of RCU.
+	 */
+	if (rdp->rsp == rcu_state &&
+	    ULONG_CMP_GE(ACCESS_ONCE(jiffies), rdp->rsp->jiffies_resched)) {
+		rdp->rsp->jiffies_resched += 5;
+		resched_cpu(rdp->cpu);
+	}
+
 	return 0;
 }
 
 static void record_gp_stall_check_time(struct rcu_state *rsp)
 {
 	unsigned long j = ACCESS_ONCE(jiffies);
+	unsigned long j1;
 
 	rsp->gp_start = j;
 	smp_wmb(); /* Record start time before stall time. */
-	rsp->jiffies_stall = j + rcu_jiffies_till_stall_check();
+	j1 = rcu_jiffies_till_stall_check();
+	rsp->jiffies_stall = j + j1;
+	rsp->jiffies_resched = j + j1 / 2;
 }
 
 /*
kernel/rcu/tree.h +2 -0
index 52be957c9fe2..8e34d8674a4e 100644
@@ -453,6 +453,8 @@ struct rcu_state {
 						/*  but in jiffies. */
 	unsigned long jiffies_stall;		/* Time at which to check */
 						/*  for CPU stalls. */
+	unsigned long jiffies_resched;		/* Time at which to resched */
+						/*  a reluctant CPU. */
 	unsigned long gp_max;			/* Maximum GP duration in */
 						/*  jiffies. */
 	const char *name;			/* Name of structure. */
author
#2 lore raw
Quoted 30 lines · Paul E. McKenney
Quoted 29 lines · Dave Jones

Paul,
I'm seeing this happening more and more lately...

[  771.786462] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  771.786552] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786574] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786595] 	(detected by 0, t=6502 jiffies, g=20611, c=20610, q=0)
[  771.786620] INFO: Stall ended before state dump start
[  966.724546] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  966.724854] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.724931] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.725005] 	(detected by 0, t=26007 jiffies, g=20611, c=20610, q=0)
[  966.725093] INFO: Stall ended before state dump start
[ 1161.661459] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1161.661763] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661840] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661915] 	(detected by 0, t=45512 jiffies, g=20611, c=20610, q=0)
[ 1161.662001] INFO: Stall ended before state dump start
[ 1356.598205] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1356.598513] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598590] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598664] 	(detected by 0, t=65017 jiffies, g=20611, c=20610, q=0)
[ 1356.598751] INFO: Stall ended before state dump start
[ 1551.536099] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1551.536408] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536485] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536559] 	(detected by 0, t=84522 jiffies, g=20611, c=20610, q=0)
[ 1551.536645] INFO: Stall ended before state dump start
Quoted 4 lines
Quoted 2 lines

While it's apparently a non-problem, it's pretty noisy. Any ideas?

Does the following help?

Seems so so far!

Dave

#3 lore raw
Quoted 31 lines · Dave Jones
Quoted 30 lines · Paul E. McKenney
Quoted 29 lines · Dave Jones

Paul,
I'm seeing this happening more and more lately...

[  771.786462] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  771.786552] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786574] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786595] 	(detected by 0, t=6502 jiffies, g=20611, c=20610, q=0)
[  771.786620] INFO: Stall ended before state dump start
[  966.724546] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  966.724854] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.724931] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.725005] 	(detected by 0, t=26007 jiffies, g=20611, c=20610, q=0)
[  966.725093] INFO: Stall ended before state dump start
[ 1161.661459] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1161.661763] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661840] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661915] 	(detected by 0, t=45512 jiffies, g=20611, c=20610, q=0)
[ 1161.662001] INFO: Stall ended before state dump start
[ 1356.598205] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1356.598513] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598590] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598664] 	(detected by 0, t=65017 jiffies, g=20611, c=20610, q=0)
[ 1356.598751] INFO: Stall ended before state dump start
[ 1551.536099] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1551.536408] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536485] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536559] 	(detected by 0, t=84522 jiffies, g=20611, c=20610, q=0)
[ 1551.536645] INFO: Stall ended before state dump start
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

While it's apparently a non-problem, it's pretty noisy. Any ideas?

Does the following help?

Seems so so far!

Keeping fingers firmly crossed...

Thanx, Paul

#4 lore raw
In reply to Dave Jones
Quoted 31 lines · Dave Jones
Quoted 30 lines · Paul E. McKenney
Quoted 29 lines · Dave Jones

Paul,
I'm seeing this happening more and more lately...

[  771.786462] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  771.786552] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786574] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  771.786595] 	(detected by 0, t=6502 jiffies, g=20611, c=20610, q=0)
[  771.786620] INFO: Stall ended before state dump start
[  966.724546] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  966.724854] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.724931] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[  966.725005] 	(detected by 0, t=26007 jiffies, g=20611, c=20610, q=0)
[  966.725093] INFO: Stall ended before state dump start
[ 1161.661459] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1161.661763] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661840] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1161.661915] 	(detected by 0, t=45512 jiffies, g=20611, c=20610, q=0)
[ 1161.662001] INFO: Stall ended before state dump start
[ 1356.598205] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1356.598513] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598590] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1356.598664] 	(detected by 0, t=65017 jiffies, g=20611, c=20610, q=0)
[ 1356.598751] INFO: Stall ended before state dump start
[ 1551.536099] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1551.536408] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536485] 	Tasks blocked on level-0 rcu_node (CPUs 0-3):
[ 1551.536559] 	(detected by 0, t=84522 jiffies, g=20611, c=20610, q=0)
[ 1551.536645] INFO: Stall ended before state dump start
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

While it's apparently a non-problem, it's pretty noisy. Any ideas?

Does the following help?

Seems so so far!

Nice!

But what workload are you running that causes this? The only way I can think of to make this happen requires that you have at least one CPU spending more than 21 seconds running continuously in kernel mode. Is that what you are doing, or have you come up with some other creative way for this to happen? In the latter case, I would really like to know about it in case it exposes yet another flaw in my RCU design. ;-)

Thanx, Paul

author
#5 lore raw
Quoted 6 lines · Paul E. McKenney

But what workload are you running that causes this? The only way I can think of to make this happen requires that you have at least one CPU spending more than 21 seconds running continuously in kernel mode. Is that what you are doing, or have you come up with some other creative way for this to happen? In the latter case, I would really like to know about it in case it exposes yet another flaw in my RCU design. ;-)

just the regular trinity runs, so it could be anything.
(Running a few instances of it, with a couple dozen child processes each to drive up the loadavg)

if only I had a trace or something that I could start to guess at what was happening :)

Dave

#6 lore raw
Quoted 11 lines · Dave Jones
Quoted 6 lines · Paul E. McKenney

But what workload are you running that causes this? The only way I can think of to make this happen requires that you have at least one CPU spending more than 21 seconds running continuously in kernel mode. Is that what you are doing, or have you come up with some other creative way for this to happen? In the latter case, I would really like to know about it in case it exposes yet another flaw in my RCU design. ;-)

just the regular trinity runs, so it could be anything.
(Running a few instances of it, with a couple dozen child processes each to drive up the loadavg)

Well, trinity has caused kernel-mode execution periods in excess of 21 seconds in the past, so I will assume that this is the cause until further notice.

Quoted 1 line

if only I had a trace or something that I could start to guess at what was happening :)

Maybe creating such a trace would expose additional bugs. ;-)

Thanx, Paul

#7 lore raw
In reply to Paul E. McKenney

kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp, · line 837

Quoted 8 lines · Paul E. McKenney
kernel/rcu/tree.c +6 -0
+	if (rdp->rsp == rcu_state &&
+	    ULONG_CMP_GE(ACCESS_ONCE(jiffies), rdp->rsp->jiffies_resched)) {
+		rdp->rsp->jiffies_resched += 5;
+		resched_cpu(rdp->cpu);
+	}
+
 	return 0;
 }

jiffies should not need ACCESS_ONCE(), right ?

It is one of the few variables marked with volatile keyword.

#8 lore raw
Quoted 2 lines · Eric Dumazet

kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp, · line 831

Quoted 1 line · Paul E. McKenney
kernel/rcu/tree.c +1 -0
+	    ULONG_CMP_GE(ACCESS_ONCE(jiffies), rdp->rsp->jiffies_resched)) {

perhaps time_before_eq

Quoted 3 lines

jiffies should not need ACCESS_ONCE(), right ?

It is one of the few variables marked with volatile keyword.

It does seem redundant

$ git grep -n "ACCESS_ONCE(jiffies)"
kernel/rcu/torture.c:1354:      jiffies_snap = ACCESS_ONCE(jiffies);
kernel/rcu/torture.c:1363:              jiffies_snap = ACCESS_ONCE(jiffies);
kernel/rcu/tree.c:820:  unsigned long j = ACCESS_ONCE(jiffies);
kernel/rcu/tree.c:975:  j = ACCESS_ONCE(jiffies);
kernel/sched/core.c:2324:       unsigned long next, now = ACCESS_ONCE(jiffies);
kernel/sched/proc.c:536:        unsigned long curr_jiffies = ACCESS_ONCE(jiffies);
kernel/sched/proc.c:558:        unsigned long curr_jiffies = ACCESS_ONCE(jiffies);
#9 lore raw
Quoted 10 lines · Joe Perches
Quoted 2 lines · Eric Dumazet

kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp, · line 831

Quoted 1 line · Paul E. McKenney
kernel/rcu/tree.c +1 -0
+	    ULONG_CMP_GE(ACCESS_ONCE(jiffies), rdp->rsp->jiffies_resched)) {

perhaps time_before_eq

Quoted 3 lines

jiffies should not need ACCESS_ONCE(), right ?

It is one of the few variables marked with volatile keyword.

Quoted 7 lines

It does seem redundant

$ git grep -n "ACCESS_ONCE(jiffies)"
kernel/rcu/torture.c:1354:      jiffies_snap = ACCESS_ONCE(jiffies);
kernel/rcu/torture.c:1363:              jiffies_snap = ACCESS_ONCE(jiffies);
kernel/rcu/tree.c:820:  unsigned long j = ACCESS_ONCE(jiffies);
kernel/rcu/tree.c:975:  j = ACCESS_ONCE(jiffies);

I took care of the first four, plus one more that I have in my local tree.

Thanx, Paul

Quoted 5 lines
kernel/sched/core.c:2324:       unsigned long next, now = ACCESS_ONCE(jiffies);
kernel/sched/proc.c:536:        unsigned long curr_jiffies = ACCESS_ONCE(jiffies);
kernel/sched/proc.c:558:        unsigned long curr_jiffies = ACCESS_ONCE(jiffies);
#10 lore raw
In reply to Eric Dumazet
Quoted 12 lines · Eric Dumazet

kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp, · line 837

Quoted 8 lines · Paul E. McKenney
kernel/rcu/tree.c +6 -0
+	if (rdp->rsp == rcu_state &&
+	    ULONG_CMP_GE(ACCESS_ONCE(jiffies), rdp->rsp->jiffies_resched)) {
+		rdp->rsp->jiffies_resched += 5;
+		resched_cpu(rdp->cpu);
+	}
+
 	return 0;
 }

jiffies should not need ACCESS_ONCE(), right ?

Quoted 1 line

It is one of the few variables marked with volatile keyword.

Good point! I have queued a patch fixing this and four other occurrences with your Submitted-by.

Thanx, Paul