Skip to contentSkip to conversation

PROBLEM: CONFIG_NO_HZ could cause software timeouts

DISCUSSION Norbert van Bolhuis 4 participants 7 messages Sep 3, 2009 – Sep 6, 2009

lore
Share & tools
lore
#0 lore raw

The problem occurs when e.g. drivers use time_after(jiffes, timeout).

CONFIG_NO_HZ could make jiffies advance by more than 1.
This is done by: tick_nohz_update_jiffies->tick_do_update_jiffies64->do_timer

If drivers use a timeout value of jiffies+1, "time_after(jiffies, timeout)" will be true after 1 interrupt (given that it advances jiffies by at least 2).

This is exactly what happens in cfi_cmdset_0002.c:do_write_buffer for our case (Powerpc MPC8313, linux-2.6.28, CONFIG_HZ=250, CONFIG_NO_HZ=y).

do_write_buffer does the following:
unsigned long uWriteTimeout = ( HZ / 1000 ) + 1;
...
timeo = jiffies + uWriteTimeout;
...
for (;;) {
...
if (time_after(jiffies, timeo) && !chip_ready(map, adr))
break;
if (chip_ready(map, adr)) {
xip_enable(map, chip, adr);
goto op_done;
}
UDELAY(map, chip, adr, 1);
}
/* software timeout */
ret = -EIO;
opdone:
...

I've seen a few software timeouts after the for-loop
looped only 13 times (= 13 us delay, i.s.o. the expected 1 ms). Typically our NOR flash (S29GL01GP) may need upto ~ 200 us to be ready.

disabling CONFIG_NO_HZ fixes the problem.
replacing time_after by a for-loop counter to loop max 1000 times also fixes the problem.

the latest kernel seems to have the same problem.

do I miss something here or is this a known problem of CONFIG_NO_HZ ?

#1 lore raw

Is XIP enabled? It seems that the xip_udelay is not very accurate, and may be the cause of what you are seeing.

Quoted 51 lines · Norbert van Bolhuis

The problem occurs when e.g. drivers use time_after(jiffes, timeout).

CONFIG_NO_HZ could make jiffies advance by more than 1. This is done by:
tick_nohz_update_jiffies->tick_do_update_jiffies64->do_timer

If drivers use a timeout value of jiffies+1, "time_after(jiffies, timeout)" will be true after 1 interrupt (given that it advances jiffies by at least 2).

This is exactly what happens in cfi_cmdset_0002.c:do_write_buffer for our case (Powerpc MPC8313, linux-2.6.28, CONFIG_HZ=250, CONFIG_NO_HZ=y).

do_write_buffer does the following:
 unsigned long uWriteTimeout = ( HZ / 1000 ) + 1;
 ...
 timeo = jiffies + uWriteTimeout;
 ...
 for (;;) {
 ...
 if (time_after(jiffies, timeo) && !chip_ready(map, adr)) break;
 if (chip_ready(map, adr)) {
  xip_enable(map, chip, adr);
  goto op_done;
 }
 UDELAY(map, chip, adr, 1);
 }
 /* software timeout */
 ret = -EIO;
opdone:
 ...

I've seen a few software timeouts after the for-loop
looped only 13 times (= 13 us delay, i.s.o. the expected 1 ms). Typically our NOR flash (S29GL01GP) may need upto ~ 200 us to be ready.

disabling CONFIG_NO_HZ fixes the problem.
replacing time_after by a for-loop counter to loop max 1000 times also fixes the problem.

the latest kernel seems to have the same problem.

do I miss something here or is this a known problem of CONFIG_NO_HZ ? --
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Signature · 10 lines
__________________________________________________________________________

dott. Corrado Zoccolo                          mailto:[email protected]
PhD - Department of Computer Science - University of Pisa, Italy
--------------------------------------------------------------------------
The self-confidence of a warrior is not the self-confidence of the average
man. The average man seeks certainty in the eyes of the onlooker and calls
that self-confidence. The warrior seeks impeccability in his own eyes and
calls that humbleness.
                               Tales of Power - C. Castaneda
#2 lore raw
Quoted 3 lines · Corrado Zoccolo

Is XIP enabled? It seems that the xip_udelay is not very accurate, and may be the cause of what you are seeing.

no it isn't. CONFIG_MTD_XIP has been disabled.
nor did someone supsended the write (chip->state remains FL_WRITING).

#3 lore raw
In reply to Norbert van Bolhuis
Quoted 35 lines · Norbert van Bolhuis

The problem occurs when e.g. drivers use time_after(jiffes, timeout).

CONFIG_NO_HZ could make jiffies advance by more than 1. This is done by:
tick_nohz_update_jiffies->tick_do_update_jiffies64->do_timer

If drivers use a timeout value of jiffies+1, "time_after(jiffies, timeout)" will be true after 1 interrupt (given that it advances jiffies by at least 2).

This is exactly what happens in cfi_cmdset_0002.c:do_write_buffer for our case (Powerpc MPC8313, linux-2.6.28, CONFIG_HZ=250, CONFIG_NO_HZ=y).

do_write_buffer does the following:
unsigned long uWriteTimeout = ( HZ / 1000 ) + 1;
...
timeo = jiffies + uWriteTimeout;
...
for (;;) {
...
if (time_after(jiffies, timeo) && !chip_ready(map, adr))
break;

  if (chip_ready(map, adr)) {
   xip_enable(map, chip, adr);
   goto op_done;

}
UDELAY(map, chip, adr, 1);
}
/* software timeout */
ret = -EIO;
opdone:
...

Quoted 2 lines

I've seen a few software timeouts after the for-loop
looped only 13 times (= 13 us delay, i.s.o. the expected 1 ms). Typically

Are you sure? UDELAY may call schedule(), which can return to this thread after much longer time than 13us...

Marcin

#4 lore raw
Quoted 36 lines · Marcin Slusarz
Quoted 35 lines · Norbert van Bolhuis

The problem occurs when e.g. drivers use time_after(jiffes, timeout).

CONFIG_NO_HZ could make jiffies advance by more than 1. This is done by:
tick_nohz_update_jiffies->tick_do_update_jiffies64->do_timer

If drivers use a timeout value of jiffies+1, "time_after(jiffies, timeout)" will be true after 1 interrupt (given that it advances jiffies by at least 2).

This is exactly what happens in cfi_cmdset_0002.c:do_write_buffer for our case (Powerpc MPC8313, linux-2.6.28, CONFIG_HZ=250, CONFIG_NO_HZ=y).

do_write_buffer does the following:
unsigned long uWriteTimeout = ( HZ / 1000 ) + 1;
...
timeo = jiffies + uWriteTimeout;
...
for (;;) {
...
if (time_after(jiffies, timeo) && !chip_ready(map, adr))
break;

  if (chip_ready(map, adr)) {
   xip_enable(map, chip, adr);
   goto op_done;

}
UDELAY(map, chip, adr, 1);
}
/* software timeout */
ret = -EIO;
opdone:
...

Quoted 5 lines
Quoted 2 lines

I've seen a few software timeouts after the for-loop
looped only 13 times (= 13 us delay, i.s.o. the expected 1 ms). Typically

Are you sure? UDELAY may call schedule(), which can return to this thread after much longer time than 13us...

Too long wait is expected, but AFAICS he's complaining about too short delay and that's a hard bug.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#5 lore raw
Quoted 36 lines · Pavel Machek
Quoted 35 lines · Marcin Slusarz
Quoted 34 lines · Norbert van Bolhuis

The problem occurs when e.g. drivers use time_after(jiffes, timeout).

CONFIG_NO_HZ could make jiffies advance by more than 1. This is done by:
tick_nohz_update_jiffies->tick_do_update_jiffies64->do_timer

If drivers use a timeout value of jiffies+1, "time_after(jiffies, timeout)" will be true after 1 interrupt (given that it advances jiffies by at least 2).

This is exactly what happens in cfi_cmdset_0002.c:do_write_buffer for our case (Powerpc MPC8313, linux-2.6.28, CONFIG_HZ=250, CONFIG_NO_HZ=y).

do_write_buffer does the following:
unsigned long uWriteTimeout = ( HZ / 1000 ) + 1;
...
timeo = jiffies + uWriteTimeout;
...
for (;;) {
...
if (time_after(jiffies, timeo) && !chip_ready(map, adr))
break;

  if (chip_ready(map, adr)) {
   xip_enable(map, chip, adr);
   goto op_done;

}
UDELAY(map, chip, adr, 1);
}
/* software timeout */
ret = -EIO;
opdone:
...

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I've seen a few software timeouts after the for-loop
looped only 13 times (= 13 us delay, i.s.o. the expected 1 ms). Typically

Are you sure? UDELAY may call schedule(), which can return to this thread after much longer time than 13us...

Too long wait is expected, but AFAICS he's complaining about too short delay and that's a hard bug.

Yeah, I know. But conclusion is a bit fishy - 13 iterations don't necessarily mean 13us. Bug might be elsewhere.

Marcin

#6 lore raw
In reply to Norbert van Bolhuis
Quoted 5 lines · Norbert van Bolhuis

The problem occurs when e.g. drivers use time_after(jiffes, timeout).

CONFIG_NO_HZ could make jiffies advance by more than 1.
This is done by: tick_nohz_update_jiffies->tick_do_update_jiffies64->do_timer

jiffies really should be up-to-date as long as you are running anything else than idle loop. If not, that's a bug ... and nasty one.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html