2.4.20: Proccess stuck in __lock_page ...

Hello !

I am running the 2.4.20 kernel on a system with 3.5 GB RAM and dual CPU. I am running bonnie accross four drives in parallel:

bonnie -s 1000 -d /<dir-name>

bdflush settings on this system:

[root@dyn-10-123-130-235 vm]# cat bdflush
2 50 32 100 50 300 1 0 0

All the bonnie process and any other process (like df, ps -ef etc.) are hung in __lock_page. Breaking into kdb, I observe the following for one such bonnie process:

schedule(..)
__lock_page(..)
lock_page(..)
do_generic_file_read(..)
generic_file_read(..)

After this, the processes never exit the hang. At times, a couple of bonnie processes complete but the hang still occurs with the remaining processes and with the other processes.

I tried out the 2.5.33 kernel (one of the 2.5 series) and observed that the hang does not occur. If I run, two bonnie processes, they never get stuck. Actually, if I run 4 parallel mke2fs, they too get stuck.

Any clues where this could be happening?

Quoted 30 lines · manish

Hello !

I am running the 2.4.20 kernel on a system with 3.5 GB RAM and dual CPU. I am running bonnie accross four drives in parallel:

bonnie -s 1000 -d /<dir-name>

bdflush settings on this system:

[root@dyn-10-123-130-235 vm]# cat bdflush
2 50 32 100 50 300 1 0 0

All the bonnie process and any other process (like df, ps -ef etc.) are hung in __lock_page. Breaking into kdb, I observe the following for one such bonnie process:

schedule(..)
__lock_page(..)
lock_page(..)
do_generic_file_read(..)
generic_file_read(..)

After this, the processes never exit the hang. At times, a couple of bonnie processes complete but the hang still occurs with the remaining processes and with the other processes.

I tried out the 2.5.33 kernel (one of the 2.5 series) and observed that the hang does not occur. If I run, two bonnie processes, they never get stuck. Actually, if I run 4 parallel mke2fs, they too get stuck.

Quoted 2 lines

Any clues where this could be happening?

Hi,

Are you sure there is no disk activity ?

Run vmstat and check that, please.

Quoted 33 lines · Marcelo Tosatti

Quoted 30 lines · manish

Hello !

I am running the 2.4.20 kernel on a system with 3.5 GB RAM and dual CPU. I am running bonnie accross four drives in parallel:

bonnie -s 1000 -d /<dir-name>

bdflush settings on this system:

[root@dyn-10-123-130-235 vm]# cat bdflush
2 50 32 100 50 300 1 0 0

All the bonnie process and any other process (like df, ps -ef etc.) are hung in __lock_page. Breaking into kdb, I observe the following for one such bonnie process:

schedule(..)
__lock_page(..)
lock_page(..)
do_generic_file_read(..)
generic_file_read(..)

After this, the processes never exit the hang. At times, a couple of bonnie processes complete but the hang still occurs with the remaining processes and with the other processes.

I tried out the 2.5.33 kernel (one of the 2.5 series) and observed that the hang does not occur. If I run, two bonnie processes, they never get stuck. Actually, if I run 4 parallel mke2fs, they too get stuck.

Quoted 9 lines

Quoted 2 lines

Any clues where this could be happening?

Hi,

Are you sure there is no disk activity ?

Run vmstat and check that, please.

Hello !

Thanks for the response.

The light on the controller does not blink at all. Intitially, it does
blink. However, after this hang, it does not at all.

vmstat after the hang

1 1 0 780 2056892 5784 1415324 0 0 0 4 102 7
49 1 50
1 1 0 780 2056892 5784 1415324 0 0 0 4 102 9
49 1 50

 1  1  0    780 2056892   5784 1415324   0   0     0     5  104    10
29  21  50
 0  1  0    780 2056708   5784 1415324   0   0     0     1  104    12
0  13  86
 1  1  0    780 2222904   5784 1249396   0   0     0   172  126    25
0   4  96
 0  1  0    780 3081052   5784 391324   0   0     0   403  161    43
0  12  88
   procs                      memory    swap          io
system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  1  0    780 3080952   5788 391408   0   0    29     9  120    72
0   0 100
 0  1  0    780 3080952   5788 391408   0   0     0     0  111    19
0   0 100
 0  1  0    780 3080952   5788 391408   0   0     0     1  103     9
0   0 100
 0  1  0    780 3080952   5788 391408   0   0     0     0  101     9
0   0 100
 0  1  0    780 3080952   5788 391408   0   0     0     0  101     7
0   0 100
 0  1  0    780 3080952   5788 391408   0   0     0     0  101     9
0   0 100
 0  1  0    780 3080952   5788 391408   0   0     0     0  102     9
0   0 100
 0  1  0    780 3080952   5788 391408   0   0     0     1  101     8
0   0 100
 0  1  0    780 3081308   5788 391420   0   0     0   231  150    92
3   0  97
 0  1  0    780 3081308   5788 391420   0   0     0     0  102     7
0   0 100
 0  1  0    780 3081308   5788 391420   0   0     0     0  102     7
0   0 100
 0  1  0    780 3081304   5788 391420   0   0     0     0  101     9

0   0 100
 0  1  0    780 3081304   5788 391420   0   0     0     0  102     8
0   0 100
 0  1  0    780 3081300   5788 391420   0   0     0     0  101     8
0   0 100
 0  1  0    780 3081300   5788 391420   0   0     0     0  101     9
0   0 100
 0  1  0    780 3081296   5788 391420   0   0     0     0  101     7

Quoted 90 lines · manish

Quoted 32 lines · Marcelo Tosatti

Quoted 30 lines · manish

Hello !

I am running the 2.4.20 kernel on a system with 3.5 GB RAM and dual CPU. I am running bonnie accross four drives in parallel:

bonnie -s 1000 -d /<dir-name>

bdflush settings on this system:

[root@dyn-10-123-130-235 vm]# cat bdflush
2 50 32 100 50 300 1 0 0

All the bonnie process and any other process (like df, ps -ef etc.) are hung in __lock_page. Breaking into kdb, I observe the following for one such bonnie process:

schedule(..)
__lock_page(..)
lock_page(..)
do_generic_file_read(..)
generic_file_read(..)

After this, the processes never exit the hang. At times, a couple of bonnie processes complete but the hang still occurs with the remaining processes and with the other processes.

I tried out the 2.5.33 kernel (one of the 2.5 series) and observed that the hang does not occur. If I run, two bonnie processes, they never get stuck. Actually, if I run 4 parallel mke2fs, they too get stuck.

Quoted 8 lines

Quoted 2 lines

Any clues where this could be happening?

Hi,

Are you sure there is no disk activity ?

Run vmstat and check that, please.

Hello !

Thanks for the response.

The light on the controller does not blink at all. Intitially, it does blink. However, after this hang, it does not at all.

vmstat after the hang

1 1 0 780 2056892 5784 1415324 0 0 0 4 102 7
49 1 50
1 1 0 780 2056892 5784 1415324 0 0 0 4 102 9
49 1 50
1 1 0 780 2056892 5784 1415324 0 0 0 5 104 10
29 21 50

0  1  0    780 2056708   5784 1415324   0   0     0     1  104    12
0  13  86
1  1  0    780 2222904   5784 1249396   0   0     0   172  126    25

0 4 96
0 1 0 780 3081052 5784 391324 0 0 0 403 161 43
0 12 88
procs memory swap io
system cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 1 0 780 3080952 5788 391408 0 0 29 9 120 72
0 0 100
0 1 0 780 3080952 5788 391408 0 0 0 0 111 19
0 0 100
0 1 0 780 3080952 5788 391408 0 0 0 1 103 9
0 0 100
0 1 0 780 3080952 5788 391408 0 0 0 0 101 9
0 0 100
0 1 0 780 3080952 5788 391408 0 0 0 0 101 7
0 0 100
0 1 0 780 3080952 5788 391408 0 0 0 0 101 9
0 0 100
0 1 0 780 3080952 5788 391408 0 0 0 0 102 9
0 0 100
0 1 0 780 3080952 5788 391408 0 0 0 1 101 8
0 0 100
0 1 0 780 3081308 5788 391420 0 0 0 231 150 92
3 0 97
0 1 0 780 3081308 5788 391420 0 0 0 0 102 7
0 0 100
0 1 0 780 3081308 5788 391420 0 0 0 0 102 7
0 0 100
0 1 0 780 3081304 5788 391420 0 0 0 0 101 9

Quoted 9 lines

0 0 100
0 1 0 780 3081304 5788 391420 0 0 0 0 102 8
0 0 100
0 1 0 780 3081300 5788 391420 0 0 0 0 101 8
0 0 100
0 1 0 780 3081300 5788 391420 0 0 0 0 101 9
0 0 100
0 1 0 780 3081296 5788 391420 0 0 0 0 101 7

Ok, and does it happen with the stock kernel?

Quoted 1 line

Ok, and does it happen with the stock kernel?

Yes, with the stock kernel too but after long hrs of runtime ..

Quoted 30 lines · manish

Hello !

I am running the 2.4.20 kernel on a system with 3.5 GB RAM and dual CPU. I am running bonnie accross four drives in parallel:

bonnie -s 1000 -d /<dir-name>

bdflush settings on this system:

[root@dyn-10-123-130-235 vm]# cat bdflush
2 50 32 100 50 300 1 0 0

All the bonnie process and any other process (like df, ps -ef etc.) are hung in __lock_page. Breaking into kdb, I observe the following for one such bonnie process:

schedule(..)
__lock_page(..)
lock_page(..)
do_generic_file_read(..)
generic_file_read(..)

After this, the processes never exit the hang. At times, a couple of bonnie processes complete but the hang still occurs with the remaining processes and with the other processes.

I tried out the 2.5.33 kernel (one of the 2.5 series) and observed that the hang does not occur. If I run, two bonnie processes, they never get stuck. Actually, if I run 4 parallel mke2fs, they too get stuck.

Quoted 2 lines

Any clues where this could be happening?

Hi,

Are you sure there is no disk activity ?

Run vmstat and check that, please.

Quoted 7 lines · manish

Hello !

I am running the 2.4.20 kernel on a system with 3.5 GB RAM and dual CPU. I am running bonnie accross four drives in parallel:

bonnie -s 1000 -d /<dir-name>

Quoted 7 lines

bdflush settings on this system:

[root@dyn-10-123-130-235 vm]# cat bdflush
2 50 32 100 50 300 1 0 0

All the bonnie process and any other process (like df, ps -ef etc.) are hung in __lock_page. Breaking into kdb, I observe the following for one

Christian,
this looks supiciously like the problem you are experiencing since 2.4.19-pre. Maybe we can fix this for good.

Christian,

this looks supiciously like the problem you are experiencing since 2.4.19-pre. Maybe we can fix this for good.

Quoted 16 lines · Marcelo Tosatti

Quoted 12 lines · manish

Hello !

I am running the 2.4.20 kernel on a system with 3.5 GB RAM and dual CPU. I am running bonnie accross four drives in parallel:

bonnie -s 1000 -d /<dir-name>

bdflush settings on this system:

[root@dyn-10-123-130-235 vm]# cat bdflush
2 50 32 100 50 300 1 0 0

Quoted 2 lines

Quoted 2 lines

All the bonnie process and any other process (like df, ps -ef etc.) are hung in __lock_page. Breaking into kdb, I observe the following for one

Following is SysRq-T output for stuck processes during such a pause from Christian Klose. Only processes in D state are listed for brevity. Especially the last two call traces are interesting.

Quoted 3 lines

Following is SysRq-T output for stuck processes during such a pause from Christian Klose. Only processes in D state are listed for brevity. Especially the last two call traces are interesting.

A "pause" is perfectly fine (to some extent, of course), now a hang is not. Is this backtrace from a hanged, unusable kernel or ?

Quoted 3 lines

Following is SysRq-T output for stuck processes during such a pause from Christian Klose. Only processes in D state are listed for brevity. Especially the last two call traces are interesting.

A "pause" is perfectly fine (to some extent, of course), now a hang is not. Is this backtrace from a hanged, unusable kernel or ?

On Tuesday 27 May 2003 19:27, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 5 lines

Quoted 3 lines

Following is SysRq-T output for stuck processes during such a pause from Christian Klose. Only processes in D state are listed for brevity. Especially the last two call traces are interesting.

A "pause" is perfectly fine (to some extent, of course), now a hang is not. Is this backtrace from a hanged, unusable kernel or ?

A pause is _not_ perfectly fine, even not to some extent. That pause we are discussing about is a pause of the _whole_ machine, not just disk i/o pauses. Mouse stops, keyboard stops, everything stops, who knows wtf.

That behaviour is absolutely bullshit for desktop users. For serverusage you may not notice it in this dimension (mostly no X so no mouse), but also for a server environment this may be very bad.

Quoted 3 lines

A pause is _not_ perfectly fine, even not to some extent. That pause we are discussing about is a pause of the _whole_ machine, not just disk i/o pauses. Mouse stops, keyboard stops, everything stops, who knows wtf.

Do you also notice them?

Quoted 3 lines

That behaviour is absolutely bullshit for desktop users. For serverusage you may not notice it in this dimension (mostly no X so no mouse), but also for a server environment this may be very bad.

Agreed.

On Tuesday 27 May 2003 19:47, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

Quoted 3 lines

A pause is _not_ perfectly fine, even not to some extent. That pause we are discussing about is a pause of the _whole_ machine, not just disk i/o pauses. Mouse stops, keyboard stops, everything stops, who knows wtf.

Do you also notice them?

I do, people I know do also, numbers of those people only _I_ know are about ~30. I've reported this problem over a year ago while 2.4.19-pre time.

Quoted 4 lines

Quoted 3 lines

That behaviour is absolutely bullshit for desktop users. For serverusage you may not notice it in this dimension (mostly no X so no mouse), but also for a server environment this may be very bad.

Agreed.

Quoted 8 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 19:47, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

Quoted 4 lines

A pause is _not_ perfectly fine, even not to some extent. That pause we are discussing about is a pause of the _whole_ machine, not just disk i/o pauses. Mouse stops, keyboard stops, everything stops, who knows wtf.

Quoted 5 lines

Quoted 2 lines

Do you also notice them?

I do, people I know do also, numbers of those people only _I_ know are about ~30. I've reported this problem over a year ago while 2.4.19-pre time.

Can you please try to reproduce it with -aa?

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

That behaviour is absolutely bullshit for desktop users. For serverusage you may not notice it in this dimension (mostly no X so no mouse), but also for a server environment this may be very bad.

Agreed.

Quoted 3 lines

I do, people I know do also, numbers of those people only _I_ know are about ~30. I've reported this problem over a year ago while 2.4.19-pre time.

Can you please try to reproduce it with -aa?

On Tuesday 27 May 2003 19:57, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

Quoted 3 lines

I do, people I know do also, numbers of those people only _I_ know are about ~30. I've reported this problem over a year ago while 2.4.19-pre time.

Can you please try to reproduce it with -aa?

not again ;)

I've tried almost all known kernel tree's around, every kernel has the same effect. I even tried SuSE and Redhat Kernels.

I've 'wasted' tons of time just find a solution for it.

Andrea introduced, to address _exact_ this problem (pauses, stops, mouse is dead etc.), his lowlatency elevator. Side effect: decreases i/o throughput,

and the "pauses/stops" are still there. Much less but not gone.

The _only_ workaround yet (known to the public) is to change nr_requests in drivers/block/ll_rw_blk.c from 128 to 4 which gives a performance hit of about 40% (not acceptable in any way).

.oO( I am quite sure I've mailed you all this stuff privately in response to
your private mail to me ;) )Oo.

ciao, Marc

Quoted 10 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 19:57, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

Quoted 3 lines

I do, people I know do also, numbers of those people only _I_ know are about ~30. I've reported this problem over a year ago while 2.4.19-pre time.

Can you please try to reproduce it with -aa?

not again ;)

Quoted 7 lines

I've tried almost all known kernel tree's around, every kernel has the same effect. I even tried SuSE and Redhat Kernels.

I've 'wasted' tons of time just find a solution for it.

Andrea introduced, to address _exact_ this problem (pauses, stops, mouse is dead etc.), his lowlatency elevator. Side effect: decreases i/o throughput,

not exactly decreases I/O throughput, the latest I/O benchmarks I seen from Randy (dbench/tiotest/bonnie/etc..) were still the fastest and it included the lowlatency elevator patch. So it may not help latency but it doesn't hurt in the numbers, at least not in the high end (that in theory is the one that needs the overkill length in the I/O queue most).

However it definitely helps latency for me and I had a number of positive reports.

Also make sure that you elvtune -r 0 -w 0 /dev/hda, also the journaling may affect the latency so you can try with plain ext2 to be sure it's not a fs issue.

the lowlatency elevator patch may not be perfect but it definitely seems to work better here. especially since there's no apparent throughput loss, it makes lots of sense to keep it applied, or it would waste lots of ram for apparently no gain.

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the problem its trying to fix and how?

On Tuesday 27 May 2003 20:33, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the problem its trying to fix and how?

Please have also a look here:

http://hypermail.idiosynkrasia.net/linux-kernel/archived/2002/week45/0305.html

ciao, Marc

Quoted 8 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 20:33, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the problem its trying to fix and how?

Quoted 6 lines

Please have also a look here:

http://hypermail.idiosynkrasia.net/linux-kernel/archived/2002/week45/0305.html

ciao, Marc

Hello !

I applied the fix-pausing-2 patch to the 2.4.20 kernel. This time on, the stack trace:

sys_write
generic_file_write
ext2_get_group_desc
bread
__wait_on_buffer
schedule

Quoted 21 lines · manish

Quoted 8 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 20:33, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the problem its trying to fix and how?

Quoted 6 lines

Please have also a look here:

http://hypermail.idiosynkrasia.net/linux-kernel/archived/2002/week45/0305.html

ciao, Marc

Hello !

I applied the fix-pausing-2 patch to the 2.4.20 kernel. This time on, the stack trace:

Quoted 7 lines

sys_write
generic_file_write
ext2_get_group_desc
bread
__wait_on_buffer
schedule

Huh? You mean bonnie still deadlocks or ?

Quoted 21 lines · manish

Quoted 8 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 20:33, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the problem its trying to fix and how?

Quoted 6 lines

Please have also a look here:

http://hypermail.idiosynkrasia.net/linux-kernel/archived/2002/week45/0305.html

ciao, Marc

Hello !

I applied the fix-pausing-2 patch to the 2.4.20 kernel. This time on, the stack trace:

Quoted 7 lines

sys_write
generic_file_write
ext2_get_group_desc
bread
__wait_on_buffer
schedule

Huh? You mean bonnie still deadlocks or ?

Quoted 32 lines · Marcelo Tosatti

Quoted 21 lines · manish

Quoted 8 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 20:33, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the problem its trying to fix and how?

Quoted 6 lines

Please have also a look here:

http://hypermail.idiosynkrasia.net/linux-kernel/archived/2002/week45/0305.html

ciao, Marc

Hello !

I applied the fix-pausing-2 patch to the 2.4.20 kernel. This time on, the stack trace:

Quoted 7 lines

sys_write
generic_file_write
ext2_get_group_desc
bread
__wait_on_buffer
schedule

Quoted 2 lines

Huh? You mean bonnie still deadlocks or ?

At the time the processes get stuck:

[root@dyn-10-123-130-235 vm]# more /proc/meminfo
total: used: free: shared: buffers: cached:

Mem:  3709870080 3699126272 10743808        0 18313216 3531255808
Swap: 1077501952        0 1077501952
MemTotal:      3622920 kB
MemFree:         10492 kB
MemShared:           0 kB
Buffers:         17884 kB
Cached:        3448492 kB
SwapCached:          0 kB
Active:          25252 kB
Inactive:      3445344 kB
HighTotal:     2752512 kB
HighFree:         2120 kB
LowTotal:       870408 kB
LowFree:          8372 kB
SwapTotal:     1052248 kB
SwapFree:      1052248 kB

Quoted 39 lines · manish

Quoted 23 lines · Marcelo Tosatti

Quoted 21 lines · manish

Quoted 8 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 20:33, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the problem its trying to fix and how?

Quoted 6 lines

Please have also a look here:

http://hypermail.idiosynkrasia.net/linux-kernel/archived/2002/week45/0305.html

ciao, Marc

Hello !

I applied the fix-pausing-2 patch to the 2.4.20 kernel. This time on, the stack trace:

Quoted 9 lines

Quoted 7 lines

sys_write
generic_file_write
ext2_get_group_desc
bread
__wait_on_buffer
schedule

Huh? You mean bonnie still deadlocks or ?

At the time the processes get stuck:

[root@dyn-10-123-130-235 vm]# more /proc/meminfo
total: used: free: shared: buffers: cached:

Quoted 17 lines

Mem:  3709870080 3699126272 10743808        0 18313216 3531255808
Swap: 1077501952        0 1077501952
MemTotal:      3622920 kB
MemFree:         10492 kB
MemShared:           0 kB
Buffers:         17884 kB
Cached:        3448492 kB
SwapCached:          0 kB
Active:          25252 kB
Inactive:      3445344 kB
HighTotal:     2752512 kB
HighFree:         2120 kB
LowTotal:       870408 kB
LowFree:          8372 kB
SwapTotal:     1052248 kB
SwapFree:      1052248 kB

Ok, so just to confirm: You're still getting pauses with Andrea's patches but no hangs anymore?

Correct?

Quoted 60 lines · Marcelo Tosatti

Quoted 39 lines · manish

Quoted 23 lines · Marcelo Tosatti

Quoted 21 lines · manish

Quoted 8 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 20:33, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the problem its trying to fix and how?

Quoted 6 lines

Please have also a look here:

http://hypermail.idiosynkrasia.net/linux-kernel/archived/2002/week45/0305.html

ciao, Marc

Hello !

I applied the fix-pausing-2 patch to the 2.4.20 kernel. This time on, the stack trace:

Quoted 9 lines

Quoted 7 lines

sys_write
generic_file_write
ext2_get_group_desc
bread
__wait_on_buffer
schedule

Huh? You mean bonnie still deadlocks or ?

At the time the processes get stuck:

[root@dyn-10-123-130-235 vm]# more /proc/meminfo
total: used: free: shared: buffers: cached:

Quoted 17 lines

Mem:  3709870080 3699126272 10743808        0 18313216 3531255808
Swap: 1077501952        0 1077501952
MemTotal:      3622920 kB
MemFree:         10492 kB
MemShared:           0 kB
Buffers:         17884 kB
Cached:        3448492 kB
SwapCached:          0 kB
Active:          25252 kB
Inactive:      3445344 kB
HighTotal:     2752512 kB
HighFree:         2120 kB
LowTotal:       870408 kB
LowFree:          8372 kB
SwapTotal:     1052248 kB
SwapFree:      1052248 kB

Quoted 5 lines

Ok, so just to confirm: You're still getting pauses with Andrea's patches but no hangs anymore?

Correct?

Hi Marcelo,

I have applied Andrea's patch to two kernels:

1. Stock 2.4.20
2. 2.4.20 with the io_request_lock removed.

The tests on the first one are still going. The tests on the second one showed processes getting stuck for long times (> 5 minutes) and not paused ...

Quoted 5 lines

1. Stock 2.4.20
2. 2.4.20 with the io_request_lock removed.
The tests on the first one are still going. The tests on the second one showed processes getting stuck for long times (> 5 minutes) and not paused ...

sorry if it's a dumb question but what is the "io_request_lock removed" thing? Hope you didn't delete any io_request_lock, if you did you can get worse things than crashes (i.e. mm/fs corruption). the pausing bug was a genuine race (quite innocent, if you could trigger a disk unplug you could recover from it)

Andrea

what is this io_request_lock patch you are talking about?

ciao, Marc

On Tuesday 27 May 2003 22:20, Andrea Arcangeli wrote:

Hi Andrea,

Quoted 6 lines

Quoted 6 lines

1. Stock 2.4.20
2. 2.4.20 with the io_request_lock removed.
The tests on the first one are still going. The tests on the second one showed processes getting stuck for long times (> 5 minutes) and not paused ...

Quoted 8 lines

sorry if it's a dumb question but what is the "io_request_lock removed" thing? Hope you didn't delete any io_request_lock, if you did you can get worse things than crashes (i.e. mm/fs corruption). the pausing bug was a genuine race (quite innocent, if you could trigger a disk unplug you could recover from it)

Andrea

funny. I asked him the same ;)

see his response:

-----------------------------------------------------------------------

Quoted 4 lines

what is this io_request_lock patch you are talking about?

ciao, Marc

We made some changes to the 2.4.20 kernel to remove the io_request_lock and replace with queue_lock and host_lock.
-----------------------------------------------------------------------

ciao, Marc

Quoted 33 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 22:20, Andrea Arcangeli wrote:

Hi Andrea,

Quoted 6 lines

Quoted 6 lines

1. Stock 2.4.20
2. 2.4.20 with the io_request_lock removed.
The tests on the first one are still going. The tests on the second one showed processes getting stuck for long times (> 5 minutes) and not paused ...

Quoted 8 lines

sorry if it's a dumb question but what is the "io_request_lock removed" thing? Hope you didn't delete any io_request_lock, if you did you can get worse things than crashes (i.e. mm/fs corruption). the pausing bug was a genuine race (quite innocent, if you could trigger a disk unplug you could recover from it)

Andrea

funny. I asked him the same ;)

see his response:

-----------------------------------------------------------------------

Quoted 4 lines

what is this io_request_lock patch you are talking about?

ciao, Marc

We made some changes to the 2.4.20 kernel to remove the io_request_lock and replace with queue_lock and host_lock.
-----------------------------------------------------------------------

Quoted 2 lines

ciao, Marc

We made a change in the 2.4.20 kernel to remove the io_request_lock and replace with the host_lock and the queue_lock. Probably, not a right thing to do

Quoted 36 lines · manish

Quoted 34 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 22:20, Andrea Arcangeli wrote:

Hi Andrea,

Quoted 6 lines

Quoted 6 lines

1. Stock 2.4.20
2. 2.4.20 with the io_request_lock removed.
The tests on the first one are still going. The tests on the second one showed processes getting stuck for long times (> 5 minutes) and not paused ...

Quoted 8 lines

sorry if it's a dumb question but what is the "io_request_lock removed" thing? Hope you didn't delete any io_request_lock, if you did you can get worse things than crashes (i.e. mm/fs corruption). the pausing bug was a genuine race (quite innocent, if you could trigger a disk unplug you could recover from it)

Andrea

funny. I asked him the same ;)

see his response:

-----------------------------------------------------------------------

Quoted 4 lines

what is this io_request_lock patch you are talking about?

ciao, Marc

We made some changes to the 2.4.20 kernel to remove the io_request_lock and replace with queue_lock and host_lock.
-----------------------------------------------------------------------

ciao, Marc

Quoted 2 lines

We made a change in the 2.4.20 kernel to remove the io_request_lock and replace with the host_lock and the queue_lock. Probably, not a right thing to do

right you are, but never mind, only remeber e2fsck the fs before booting the box so you don't risk fs corruption later with the solid kernels.

Andrea

Quoted 44 lines · Andrea Arcangeli

Quoted 41 lines · manish

Quoted 27 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 22:20, Andrea Arcangeli wrote:

Hi Andrea,

Quoted 6 lines

Quoted 6 lines

1. Stock 2.4.20
2. 2.4.20 with the io_request_lock removed.
The tests on the first one are still going. The tests on the second one showed processes getting stuck for long times (> 5 minutes) and not paused ...

Quoted 8 lines

sorry if it's a dumb question but what is the "io_request_lock removed" thing? Hope you didn't delete any io_request_lock, if you did you can get worse things than crashes (i.e. mm/fs corruption). the pausing bug was a genuine race (quite innocent, if you could trigger a disk unplug you could recover from it)

Andrea

funny. I asked him the same ;)

see his response:

-----------------------------------------------------------------------

Quoted 2 lines

what is this io_request_lock patch you are talking about?

Quoted 8 lines

Quoted 2 lines

ciao, Marc

We made some changes to the 2.4.20 kernel to remove the io_request_lock and replace with queue_lock and host_lock.
-----------------------------------------------------------------------

ciao, Marc

We made a change in the 2.4.20 kernel to remove the io_request_lock and replace with the host_lock and the queue_lock. Probably, not a right thing to do

Quoted 6 lines

right you are, but never mind, only remeber e2fsck the fs before booting the box so you don't risk fs corruption later with the solid kernels.

Andrea

So, does it imply that we cannot remove the io_request_lock in 2.4 at all?

u

Quoted 28 lines · Andrea Arcangeli

Quoted 10 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 19:57, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

Quoted 3 lines

I do, people I know do also, numbers of those people only _I_ know are about ~30. I've reported this problem over a year ago while 2.4.19-pre time.

Can you please try to reproduce it with -aa?

not again ;)

Quoted 7 lines

I've tried almost all known kernel tree's around, every kernel has the same effect. I even tried SuSE and Redhat Kernels.

I've 'wasted' tons of time just find a solution for it.

Andrea introduced, to address _exact_ this problem (pauses, stops, mouse is dead etc.), his lowlatency elevator. Side effect: decreases i/o throughput,

not exactly decreases I/O throughput, the latest I/O benchmarks I seen from Randy (dbench/tiotest/bonnie/etc..) were still the fastest and it included the lowlatency elevator patch. So it may not help latency but it doesn't hurt in the numbers, at least not in the high end (that in theory is the one that needs the overkill length in the I/O queue most).

However it definitely helps latency for me and I had a number of positive reports.

Quoted 8 lines

Also make sure that you elvtune -r 0 -w 0 /dev/hda, also the journaling may affect the latency so you can try with plain ext2 to be sure it's not a fs issue.

the lowlatency elevator patch may not be perfect but it definitely seems to work better here. especially since there's no apparent throughput loss, it makes lots of sense to keep it applied, or it would waste lots of ram for apparently no gain.

Andrea,

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the

problem its trying to fix and how?

Its too late to fix that in 2.4.21 (rc5 is going out in hours).

Quoted 2 lines

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the

yes, not just one but multiple of them, all similar. lots of boxes were hanging in a weird manner until I found and fixed this glitch.

Quoted 1 line

problem its trying to fix and how?

I'm attaching the old email, it should have all the explanataions.

but don't use that old patch (that was the first revision and it missed one last race in wait_for_request noticed by Chris or Andrew [or both?]), use this one instead (seems just the second revision, should be that one plus that last race fix):

	http://www.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.21rc2aa1/9980_fix-pausing-2

Quoted 15 lines · Andrea Arcangeli

Quoted 2 lines

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the

yes, not just one but multiple of them, all similar. lots of boxes were hanging in a weird manner until I found and fixed this glitch.

Quoted 1 line

problem its trying to fix and how?

I'm attaching the old email, it should have all the explanataions.

but don't use that old patch (that was the first revision and it missed one last race in wait_for_request noticed by Chris or Andrew [or both?]), use this one instead (seems just the second revision, should be that one plus that last race fix):

Quoted 1 line

	http://www.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.21rc2aa1/9980_fix-pausing-2

I wonder if the additional wakeups result in performance degradation (not that it matters much in case there is no other way to fix the problem).

But anyway I would like to have some numbers with/without the patch.

Do you have them ?

However the last numbers from Randy showed my tree going faster than 2.5 with bonnie and tiotest so I think we don't need to worry and I would probably not fix it in a different way in 2.4 even if it would mean a 1% degradation.

Quoted 5 lines · Andrea Arcangeli

However the last numbers from Randy showed my tree going faster than 2.5 with bonnie and tiotest so I think we don't need to worry and I would probably not fix it in a different way in 2.4 even if it would mean a 1% degradation.

That could be because -aa quadruples the size of the VM readahead window.

Changes such as that should be removed when assessing the performance impact of this particular patch.

Quoted 11 lines · Andrew Morton

Quoted 5 lines · Andrea Arcangeli

However the last numbers from Randy showed my tree going faster than 2.5 with bonnie and tiotest so I think we don't need to worry and I would probably not fix it in a different way in 2.4 even if it would mean a 1% degradation.

That could be because -aa quadruples the size of the VM readahead window.

Changes such as that should be removed when assessing the performance impact of this particular patch.

I understand that was a generic benchmark against 2.5, not meant to evaluate the effect of the fixed readahead (see the name of the patch "readahead-got-broken-somehwere"). I don't see any good reason why should Randy cripple down my tree before benchmarking against 2.5? if something it's ok to apply some of my patches to 2.5, that's great, the other way around not IMHO.

Quoted 14 lines · Andrea Arcangeli

Quoted 11 lines · Andrew Morton

Quoted 5 lines · Andrea Arcangeli

However the last numbers from Randy showed my tree going faster than 2.5 with bonnie and tiotest so I think we don't need to worry and I would probably not fix it in a different way in 2.4 even if it would mean a 1% degradation.

That could be because -aa quadruples the size of the VM readahead window.

Changes such as that should be removed when assessing the performance impact of this particular patch.

Quoted 7 lines

I understand that was a generic benchmark against 2.5, not meant to evaluate the effect of the fixed readahead (see the name of the patch "readahead-got-broken-somehwere"). I don't see any good reason why should Randy cripple down my tree before benchmarking against 2.5? if something it's ok to apply some of my patches to 2.5, that's great, the other way around not IMHO.

No.

What I am saying is that evaluation of the effect of an IO scheduler change cannot be performed when there is a 4:1 change in the readhead window present in the same tree.

ie: we cannot conclude anything about the effect of the IO scheduler change from Randy's numbers. Too many variables.

Andrea,

It seems your "fix-pausing" patch is fixing a potential wakeup miss, right? (I looked quickly throught it). Could you explain me the problem its trying to fix and how?

Its too late to fix that in 2.4.21 (rc5 is going out in hours).

not exactly decreases I/O throughput, the latest I/O benchmarks I seen

from Randy (dbench/tiotest/bonnie/etc..) were still the fastest and it included the lowlatency elevator patch. So it may not help latency but it doesn't hurt in the numbers, at least not in the high end (that in theory is the one that needs the overkill length in the I/O queue most).

However it definitely helps latency for me and I had a number of positive reports.

Also make sure that you elvtune -r 0 -w 0 /dev/hda, also the journaling

may affect the latency so you can try with plain ext2 to be sure it's not a fs issue.

the lowlatency elevator patch may not be perfect but it definitely seems to work better here. especially since there's no apparent throughput loss, it makes lots of sense to keep it applied, or it would waste lots of ram for apparently no gain.

On Tuesday 27 May 2003 20:25, Andrea Arcangeli wrote:

Hi Andrea,

Quoted 1 line

not exactly decreases I/O throughput, the latest I/O benchmarks I seen

it decreases performance. I've seen this, Con also saw this (well it's better than the 'nr_requests = 4' change ;) but mouse stops are still there.

Quoted 4 lines

from Randy (dbench/tiotest/bonnie/etc..) were still the fastest and it included the lowlatency elevator patch. So it may not help latency but it doesn't hurt in the numbers, at least not in the high end (that in theory is the one that needs the overkill length in the I/O queue most).

I agree with the last sentence, in theory, but practice showed something different (about 10% to 15% performance decrease)

But I am quite sure that this depends on your machine/hardware. Using IDE instead of SCSI for example.

Quoted 3 lines

However it definitely helps latency for me and I had a number of positive reports.

It helps but it's not as good as 2.4.18 stock.

Quoted 2 lines

Also make sure that you elvtune -r 0 -w 0 /dev/hda, also the journaling

I also tried that.

Quoted 2 lines

may affect the latency so you can try with plain ext2 to be sure it's not a fs issue.

Sure, I did this too. FS independent, where ReiserFS is still the best for this scenario with the most few pauses than any other FS (ext2, ext3, ...)

But for desktop usage: not acceptable! No way, No go!

Quoted 4 lines

the lowlatency elevator patch may not be perfect but it definitely seems to work better here. especially since there's no apparent throughput loss, it makes lots of sense to keep it applied, or it would waste lots of ram for apparently no gain.

hehe, well wasting RAM for no gain is my next part on my todo ;) (cache everything even if there is no RAM for example, well but this is not the point in this thread)

ciao, Marc

10/15 performance drop doesn't sound good, no matter what hardware ;).

However in contest I recall there was quite an improvement in latency at least (I mean, it had some positive effect too)

Getting the best throughput and latency at the same time is normally not possible, however evaluating if it's losing excessive throughput given a certain latency improvement is difficult.

I'll try to find what's the precise reason of the interactivity drop

with the 2.4.18->2.4.19 blkdev changes on Thu. I think I shortly looked into it once but there was no definitive answer, or anyways going back to the 2.4.18 code didn't appeal or make much sense.

However I suspect this responsiveness issue could be storage hardware dependent.

The sentence by Linus in the last few days while talking with Jens, about storage that reorders stuff and starve requests at the two ends of the platter was very scary, maybe you're really bitten by something like that. Linux does the right thing but your hardware keeps posting stuff under the os and mine doesn't.

I try to backport BIO and then AS for quite over 2 weeks now, but it seems, at least for me, that it's an impossible mission ;(

Quoted 1 line

I'll try to find what's the precise reason of the interactivity drop

cool. Thanks.

Quoted 3 lines

with the 2.4.18->2.4.19 blkdev changes on Thu. I think I shortly looked into it once but there was no definitive answer, or anyways going back to the 2.4.18 code didn't appeal or make much sense.

Yeah, that's not an option. The throughput has been increased in 2.4.19 compared to 2.4.18.

Quoted 3 lines

However I suspect this responsiveness issue could be storage hardware dependent.

Hmm, I am quite sure that it isn't. I have ton's of mostly totally different hardware in my company, also test machines for WOLK at freenet.de (the biggest I had was a QUAD Xeon 1GHz with 16GB memory and hardware RAID (Compaq ML570 to be exact (f*cking nice machine btw. ;) and I even hit it on that machine. Friends of mine having also different hardware then me, also hitting that bug. _If_ it's the case of storage hardware, then many storage hardware is affected ;)

Quoted 5 lines

The sentence by Linus in the last few days while talking with Jens, about storage that reorders stuff and starve requests at the two ends of the platter was very scary, maybe you're really bitten by something like that. Linux does the right thing but your hardware keeps posting stuff under the os and mine doesn't.

Oh, did I miss something at lkml or was it privately?

Quoted 2 lines

I try to backport BIO and then AS for quite over 2 weeks now, but it seems, at least for me, that it's an impossible mission ;(

bio breaks all drivers, not a good idea to backport ;)

note that the anticipatory scheduler generates very bad results with the winmark. it certainly has merits but it has large downsides too.

I would be also curious if you could compare anticipatory with CFQ. The CFQ was designed to provide the highest possible degree of fariness.

I read it on l-k yesterday a few days ago, search emails from Linus with Jens somewhere in CC and you should find it.

On Tuesday 27 May 2003 22:45, Andrea Arcangeli wrote:

Hi Andrea,

Quoted 3 lines

Quoted 2 lines

I try to backport BIO and then AS for quite over 2 weeks now, but it seems, at least for me, that it's an impossible mission ;(

bio breaks all drivers, not a good idea to backport ;)

HAHAHAH. Another wasted 2 weeks in my life ;-)

But why does it brake all drivers? Could you please elaborate a bit?

Are you serious? Please tell me you haven't spend two weeks on the project not realising this?

On Tuesday 27 May 2003 23:00, Jens Axboe wrote:

Hi Jens,

Quoted 2 lines

Are you serious? Please tell me you haven't spend two weeks on the project not realising this?

Well, 2 weeks means in hours not more than 5 or 6 just delayed over many days.

And it was further just to go deeper into the code, not a real attempt to backport it. NM.

I try to backport BIO and then AS for quite over 2 weeks now, but it seems, at least for me, that it's an impossible mission ;(

But still, why on earth waste your time with something like this now when we are so close to 2.6? 2.4 is a stable code base, it should stay that way. I'm really not interested in more esoteric 2.4 backports, the vendor kernels are bad enough as it is.

I, for one, got a good laugh out of it. =) Makes me wonder if the 2.4 distro backport trees' diffs are bigger than 2.4 itself yet.

They've backported everything else, so I guess it stood to reason it'd happen eventually.

Quoted 8 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 19:27, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 4 lines

Quoted 4 lines

Following is SysRq-T output for stuck processes during such a pause from Christian Klose. Only processes in D state are listed for brevity. Especially the last two call traces are interesting.

Quoted 7 lines

Quoted 3 lines

A "pause" is perfectly fine (to some extent, of course), now a hang is not. Is this backtrace from a hanged, unusable kernel or ?

A pause is _not_ perfectly fine, even not to some extent. That pause we are discussing about is a pause of the _whole_ machine, not just disk i/o pauses. Mouse stops, keyboard stops, everything stops, who knows wtf.

Do you also notice them?

Quoted 4 lines

That behaviour is absolutely bullshit for desktop users. For serverusage you may not notice it in this dimension (mostly no X so no mouse), but also for a server environment this may be very bad.

Agreed.

With respect to the hangs that you noticed, did the processes complete after a "pause" or did they stay hung (deadlocked)?

On Tuesday 27 May 2003 19:53, manish wrote:

Hi Manish,

Quoted 2 lines

With respect to the hangs that you noticed, did the processes complete after a "pause" or did they stay hung (deadlocked)?

yes, no processes get ever deadlocked nor anything else in this area. The whole system just does _nothing_ for an amount of time (1-15 seconds, depends). _Sometimes_ (not always) even a ping is stoped for the amount of time the machine does nothing but pausing.

Also not a hardware problem. I made this clear before reporting this bug. Tested tons of different hardware, different drivers for the network card etc.

I repeat this now for the $high_number'th time ;):
- 2.4.18 worked perfect
- 2.4.19-pre not

Quoted 3 lines

I repeat this now for the $high_number'th time ;):
- 2.4.18 worked perfect
- 2.4.19-pre not

Thats very useful information. Can you track down which -pre introduced the hangs?

Quoted 5 lines · Marc-Christian Petersen

Quoted 2 lines

A "pause" is perfectly fine (to some extent, of course), now a hang is not. Is this backtrace from a hanged, unusable kernel or ?

A pause is _not_ perfectly fine, even not to some extent. That pause we are discussing about is a pause of the _whole_ machine, not just disk i/o pauses. Mouse stops, keyboard stops, everything stops, who knows wtf.

Do you also notice them?

Quoted 6 lines · Carl-Daniel Hailfinger

Quoted 3 lines · Marcelo Tosatti

Quoted 2 lines · manish

All the bonnie process and any other process (like df, ps -ef etc.) are hung in __lock_page. Breaking into kdb, I observe the following for one

Quoted 3 lines

Following is SysRq-T output for stuck processes during such a pause from Christian Klose. Only processes in D state are listed for brevity. Especially the last two call traces are interesting.

A "pause" is perfectly fine (to some extent, of course), now a hang is not. Is this backtrace from a hanged, unusable kernel or ?

Quoted 9 lines · Marcelo Tosatti

Quoted 7 lines · Carl-Daniel Hailfinger

Quoted 5 lines · Marcelo Tosatti

Quoted 3 lines · manish

All the bonnie process and any other process (like df, ps -ef etc.) are hung in __lock_page. Breaking into kdb, I observe the following for one

Quoted 7 lines

Quoted 4 lines

Following is SysRq-T output for stuck processes during such a pause from Christian Klose. Only processes in D state are listed for brevity. Especially the last two call traces are interesting.

A "pause" is perfectly fine (to some extent, of course), now a hang is not. Is this backtrace from a hanged, unusable kernel or ?

AFAIK, the kernel is not unusable, but a 20 second pause with no disk access at all is not nice either.

Regards,
Carl-Daniel

It is not a system hang but the processes hang showing the same stack trace. This is certainly not a pause since the bonnie processes that were hung (or deadlocked) never completed after several hrs. The stack trace was the same.

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/:
speak _NOW_ please, doesn't matter who you are!

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/: speak _NOW_ please, doesn't matter who you are!

On Tuesday 27 May 2003 19:50, manish wrote:

Hi Manish,

Quoted 4 lines

It is not a system hang but the processes hang showing the same stack trace. This is certainly not a pause since the bonnie processes that were hung (or deadlocked) never completed after several hrs. The stack trace was the same.

then you are hitting a different bug or a bug related to the issues Christian Klose and me and $tons of others were complaining.

The bug you are hitting might be the problem with "process stuck in D state" Andrea Arcangeli fixed, let me guess, over half a year ago or so.

In case you have a good mind to try to address your issue, you might want to try out the patch you can find here:

http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.21rc2
aa1/9980_fix-pausing-2

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/: speak _NOW_ please, doesn't matter who you are!

Quoted 15 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 19:50, manish wrote:

Hi Manish,

Quoted 4 lines

It is not a system hang but the processes hang showing the same stack trace. This is certainly not a pause since the bonnie processes that were hung (or deadlocked) never completed after several hrs. The stack trace was the same.

then you are hitting a different bug or a bug related to the issues Christian Klose and me and $tons of others were complaining.

The bug you are hitting might be the problem with "process stuck in D state" Andrea Arcangeli fixed, let me guess, over half a year ago or so.

Quoted 8 lines

In case you have a good mind to try to address your issue, you might want to try out the patch you can find here:

http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.21rc2
aa1/9980_fix-pausing-2

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/: speak _NOW_ please, doesn't matter who you are!

Yo!

I'll throw my babushka into the ring too. I think it's obvious from MCP's comments that I've been involved in testing this problem. I've spent hours, possibly days trying to find a way to fix the pauses introduced since 2.4.19pre1. I agree with what MCP describes that the machine can come to a standstill under any sort of disk i/o and is unusable for a variable length of time. I've been playing with all sorts of numbers in my patchset to try and limit it with only mild success. The best results I've had without a major decrease in throughput was using akpm's read latency 2 patch but by significantly reducing the nr_requests. It was changing the number of requests that I discovered dropping them to 4 fixed the problem but destroyed write throughput. I was pleased to see AA give the problem recognition after my contest results on his kernel but disappointed that the problem only was reduced, not fixed.

Quoted 28 lines · Con Kolivas

Quoted 16 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 19:50, manish wrote:

Hi Manish,

Quoted 4 lines

It is not a system hang but the processes hang showing the same stack trace. This is certainly not a pause since the bonnie processes that were hung (or deadlocked) never completed after several hrs. The stack trace was the same.

then you are hitting a different bug or a bug related to the issues Christian Klose and me and $tons of others were complaining.

The bug you are hitting might be the problem with "process stuck in D state" Andrea Arcangeli fixed, let me guess, over half a year ago or so.

Quoted 8 lines

In case you have a good mind to try to address your issue, you might want to try out the patch you can find here:

http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.2
1rc2 aa1/9980_fix-pausing-2

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/: speak _NOW_ please, doesn't matter who you are!

Yo!

Quoted 13 lines

I'll throw my babushka into the ring too. I think it's obvious from MCP's comments that I've been involved in testing this problem. I've spent hours, possibly days trying to find a way to fix the pauses introduced since 2.4.19pre1. I agree with what MCP describes that the machine can come to a standstill under any sort of disk i/o and is unusable for a variable length of time. I've been playing with all sorts of numbers in my patchset to try and limit it with only mild success. The best results I've had without a major decrease in throughput was using akpm's read latency 2 patch but by significantly reducing the nr_requests. It was changing the number of requests that I discovered dropping them to 4 fixed the problem but destroyed write throughput. I was pleased to see AA give the problem recognition after my contest results on his kernel but disappointed that the problem only was reduced, not fixed.

Does the problem change at all if you force batch_requests to 0?

Quoted 43 lines · Jens Axboe

Quoted 41 lines · Con Kolivas

Quoted 16 lines · Marc-Christian Petersen

On Tuesday 27 May 2003 19:50, manish wrote:

Hi Manish,

Quoted 4 lines

It is not a system hang but the processes hang showing the same stack trace. This is certainly not a pause since the bonnie processes that were hung (or deadlocked) never completed after several hrs. The stack trace was the same.

then you are hitting a different bug or a bug related to the issues Christian Klose and me and $tons of others were complaining.

The bug you are hitting might be the problem with "process stuck in D state" Andrea Arcangeli fixed, let me guess, over half a year ago or so.

Quoted 8 lines

In case you have a good mind to try to address your issue, you might want to try out the patch you can find here:

http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.2
1rc2 aa1/9980_fix-pausing-2

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/: speak _NOW_ please, doesn't matter who you are!

Yo!

I'll throw my babushka into the ring too. I think it's obvious from MCP's comments that I've been involved in testing this problem. I've spent hours, possibly days trying to find a way to fix the pauses introduced since 2.4.19pre1. I agree with what MCP describes that the machine can come to a standstill under any sort of disk i/o and is unusable for a variable length of time. I've been playing with all sorts of numbers in my patchset to try and limit it with only mild success. The best results I've had without a major decrease in throughput was using akpm's read latency 2 patch but by significantly reducing the nr_requests. It was changing the number of requests that I discovered dropping them to 4 fixed the problem but destroyed write throughput. I was pleased to see AA give the problem recognition after my contest results on his kernel but disappointed that the problem only was reduced, not fixed.

Quoted 1 line

Does the problem change at all if you force batch_requests to 0?

I've tried batch_requests to 1 by itself (without changing the nr_request) and that didn't fix it, but recall dropping nr_requests to 2 (which would make batch requests==0) made the machine fail to boot so I haven't tried batch requests 0 by itself. Should it boot with it == 0?

If you leave nr_requests as it is, I don't see why it should not boot with batch_requests == 0.
I can't see in all of these mails whether backing out akpm's starvation patch makes the problem go away. Does it?

On Wednesday 28 May 2003 09:13, Jens Axboe wrote:

Hi Jens,

Quoted 4 lines

If you leave nr_requests as it is, I don't see why it should not boot with batch_requests == 0.
I can't see in all of these mails whether backing out akpm's starvation patch makes the problem go away. Does it?

If you mean

"http://linux.bkbits.net:8080/linux-2.4/diffs/drivers/block/[email protected]?nav=index.html|ChangeSet@-2y|[email protected]|hist/drivers/block/ll_rw_blk.c"

that one, the answer is YES.

Quoted 2 lines

that one, the answer is YES.

That's the one, yes. Andrew, looks like your patch brought out some really bad behaviour.

Quoted 5 lines · Jens Axboe

Quoted 2 lines

that one, the answer is YES.

That's the one, yes. Andrew, looks like your patch brought out some really bad behaviour.

Yes, but why?

It'd be interesting if any of these changes make a difference.

 drivers/block/ll_rw_blk.c |    7 
 fs/buffer.c               | 3030 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 3033 insertions(+), 4 deletions(-)

diff -puN drivers/block/ll_rw_blk.c~a drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~a +2 -2

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	generic_unplug_device(q);
-	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
+	add_wait_queue(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
+		generic_unplug_device(q);
 		if (q->rq[rw].count == 0)
 			schedule();
 		spin_lock_irq(&io_request_lock);

The unplug() move could be the key, in theory we could end up having to unplug the queue again.

Question to the ones seeing the stalls - does a sysrq-s make things go again?

Yes, but why?

It'd be interesting if any of these changes make a difference.

It'd be interesting if any of these changes make a difference.

 drivers/block/ll_rw_blk.c |    7 
 fs/buffer.c               | 3030 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 3033 insertions(+), 4 deletions(-)

diff -puN drivers/block/ll_rw_blk.c~a drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~a +2 -2

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	generic_unplug_device(q);
-	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
+	add_wait_queue(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
+		generic_unplug_device(q);
 		if (q->rq[rw].count == 0)
 			schedule();
 		spin_lock_irq(&io_request_lock);

Quoted 28 lines · Andrew Morton

It'd be interesting if any of these changes make a difference.

 drivers/block/ll_rw_blk.c |    7 
 fs/buffer.c               | 3030 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 3033 insertions(+), 4 deletions(-)

diff -puN drivers/block/ll_rw_blk.c~a drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~a +2 -3

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	generic_unplug_device(q);
-	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
+	add_wait_queue(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
+		generic_unplug_device(q);
 		if (q->rq[rw].count == 0)
 			schedule();
 		spin_lock_irq(&io_request_lock);

@@ -829,8 +829,7 @@ void blkdev_release_request(struct reque
 	 */
 	if (q) {
 		list_add(&req->queue, &q->rq[rw].free);
-		if (++q->rq[rw].count >= q->batch_requests &&

drivers/block/ll_rw_blk.c void blkdev_release_request(struct reque · line 835

Quoted 6 lines

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~a +1 -1

-				waitqueue_active(&q->wait_for_requests[rw]))
+		if (++q->rq[rw].count >= q->batch_requests)
 			wake_up(&q->wait_for_requests[rw]);
 	}
 }

Works fine on my notebook. Good throughput and no mouse hangs anymore.

Works fine on my notebook. Good throughput and no mouse hangs anymore.

Quoted 2 lines · Matthias Mueller

Works fine on my notebook. Good throughput and no mouse hangs anymore.

Interesting.

Could you please work out which change caused it? Go back to stock 2.4 and then apply this:

diff -puN drivers/block/ll_rw_blk.c~1 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +0 -0

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;

then this:

diff -puN drivers/block/ll_rw_blk.c~2 drivers/block/ll_rw_blk.c

Then this (totally unlikely, don't bother):

diff -puN drivers/block/ll_rw_blk.c~3 drivers/block/ll_rw_blk.c

Quoted 14 lines · Andrew Morton

Quoted 1 line · Matthias Mueller

Works fine on my notebook. Good throughput and no mouse hangs anymore.

Interesting.

Could you please work out which change caused it? Go back to stock 2.4 and then apply this:

diff -puN drivers/block/ll_rw_blk.c~1 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +0 -0

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;

drivers/block/ll_rw_blk.c static struct request *__get_request_wai · line 599

Quoted 10 lines

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +1 -1

 	DECLARE_WAITQUEUE(wait, current);
 
-	generic_unplug_device(q);
 	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
+		generic_unplug_device(q);
 		if (q->rq[rw].count == 0)
 			schedule();
 		spin_lock_irq(&io_request_lock);

I think it was already established that this wasn't the reason. Was my first suspect too, though...

Quoted 6 lines

then this:

diff -puN drivers/block/ll_rw_blk.c~2 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c static struct request *__get_request_wai · line 596

Quoted 8 lines

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~2 +1 -1

@@ -590,7 +590,7 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
+	add_wait_queue(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		generic_unplug_device(q);

Since we do a general wake_up(), only the order of wakeups matter here right (lifo vs fifo). Given that, the _exclusive() should be more fair possibly at the cost of a bit of throughput.

Quoted 2 lines

Then this (totally unlikely, don't bother):

drivers/block/ll_rw_blk.c void blkdev_release_request(struct reque · line 835

Quoted 13 lines

diff -puN drivers/block/ll_rw_blk.c~3 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~3 +1 -2

@@ -829,8 +829,7 @@ void blkdev_release_request(struct reque
 	 */
 	if (q) {
 		list_add(&req->queue, &q->rq[rw].free);
-		if (++q->rq[rw].count >= q->batch_requests &&
-				waitqueue_active(&q->wait_for_requests[rw]))
+		if (++q->rq[rw].count >= q->batch_requests)
 			wake_up(&q->wait_for_requests[rw]);
 	}
 }

Well it's the only one left :). But you are right, try one of them at the time, establishing the effect of each of them.

Quoted 50 lines · Jens Axboe

Quoted 14 lines · Andrew Morton

Quoted 1 line · Matthias Mueller

Works fine on my notebook. Good throughput and no mouse hangs anymore.

Interesting.

Could you please work out which change caused it? Go back to stock 2.4 and then apply this:

diff -puN drivers/block/ll_rw_blk.c~1 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +0 -0

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;

drivers/block/ll_rw_blk.c static struct request *__get_request_wai · line 599

Quoted 10 lines

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +1 -1

 	DECLARE_WAITQUEUE(wait, current);
 
-	generic_unplug_device(q);
 	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
+		generic_unplug_device(q);
 		if (q->rq[rw].count == 0)
 			schedule();
 		spin_lock_irq(&io_request_lock);

I think it was already established that this wasn't the reason. Was my first suspect too, though...

Quoted 6 lines

then this:

diff -puN drivers/block/ll_rw_blk.c~2 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c static struct request *__get_request_wai · line 596

Quoted 8 lines

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~2 +1 -1

@@ -590,7 +590,7 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
+	add_wait_queue(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		generic_unplug_device(q);

Since we do a general wake_up(), only the order of wakeups matter here right (lifo vs fifo). Given that, the _exclusive() should be more fair possibly at the cost of a bit of throughput.

Quoted 2 lines

Then this (totally unlikely, don't bother):

Quoted 16 lines

drivers/block/ll_rw_blk.c void blkdev_release_request(struct reque · line 835

Quoted 13 lines

diff -puN drivers/block/ll_rw_blk.c~3 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~3 +1 -2

@@ -829,8 +829,7 @@ void blkdev_release_request(struct reque
 	 */
 	if (q) {
 		list_add(&req->queue, &q->rq[rw].free);
-		if (++q->rq[rw].count >= q->batch_requests &&
-				waitqueue_active(&q->wait_for_requests[rw]))
+		if (++q->rq[rw].count >= q->batch_requests)
 			wake_up(&q->wait_for_requests[rw]);
 	}
 }

Well it's the only one left :). But you are right, try one of them at the time, establishing the effect of each of them.

THIS IS IT! The last one. No pauses writing a 2Gb file now unless I do a read midstream.

Quoted 3 lines

THIS IS IT! The last one. No pauses writing a 2Gb file now unless I do a read midstream.

Cool, especially since we can easily apply this to -rc5 without any worries. Marcelo, if you please...?

===== drivers/block/ll_rw_blk.c 1.44 vs edited =====
--- 1.44/drivers/block/ll_rw_blk.c Mon Apr 14 12:53:03 2003
+++ edited/drivers/block/ll_rw_blk.c Wed May 28 12:49:30 2003
@@ -829,8 +829,7 @@
*/
if (q) {
list_add(&req->queue, &q->rq[rw].free);
- if (++q->rq[rw].count >= q->batch_requests &&
- waitqueue_active(&q->wait_for_requests[rw]))
+ if (++q->rq[rw].count >= q->batch_requests)
wake_up(&q->wait_for_requests[rw]);
}
}

umm, I'd like confirmation of that.

The waitqueue_active() test is wrong because of a missing barrier, but only on SMP. And if it does make a mistake it will surely correct itself when the next request is put back. (That's why I left it there...) More testing, please.

Does the attached one make sense?

Quoted 1 line

Does the attached one make sense?

Nope.

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

On Wednesday 28 May 2003 13:27, Andrew Morton wrote:

Hi Akpm,

Quoted 2 lines

Quoted 1 line

Does the attached one make sense?

Nope.

nm.

Quoted 3 lines

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

Quoted 6 lines · Marc-Christian Petersen

Quoted 3 lines · Andrew Morton

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

Quoted 3 lines

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoted 12 lines · Jens Axboe

Quoted 9 lines · Marc-Christian Petersen

Quoted 3 lines · Andrew Morton

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

Quoted 2 lines

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoting Con Kolivas:

dd if=/dev/zero of=dump bs=4096 count=512000

Quoted 6 lines

Quoted 2 lines

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoting Con Kolivas:

dd if=/dev/zero of=dump bs=4096 count=512000

already tried that, no go. on ide/scsi? what filesystem? how much ram? anything else running? smp/up?

Quoted 14 lines · Carl-Daniel Hailfinger

Quoted 12 lines · Jens Axboe

Quoted 8 lines · Marc-Christian Petersen

Quoted 3 lines · Andrew Morton

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoted 3 lines

Quoting Con Kolivas:

dd if=/dev/zero of=dump bs=4096 count=512000

already tried that, no go. on ide/scsi? what filesystem? how much ram? anything else running? smp/up?

Quoted 19 lines · Jens Axboe

Quoted 17 lines · Carl-Daniel Hailfinger

Quoted 14 lines · Jens Axboe

Quoted 9 lines · Marc-Christian Petersen

Quoted 3 lines · Andrew Morton

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoted 6 lines

Quoted 3 lines

Quoting Con Kolivas:

dd if=/dev/zero of=dump bs=4096 count=512000

already tried that, no go. on ide/scsi? what filesystem? how much ram? anything else running? smp/up?

I'm using UP on IDE. I reproduce it easily on a P3 256Mb laptop with 5400rpm drive, and less easily but still occurs on a P4 2.53 512Mb pc with 2x7200rpm software raid 0 IDE drives. Even if the only thing you try to do is move the mouse, the mouse will freeze for up to 30secs. When you first start the write no disk activity happens for up to a few seconds, then it will start writing madly and the machine will come to a standstill for a variable length of time. Then it will come back to life for a few seconds only to die again for a few seconds and so on till the write is complete.

Still testing combinations to see which is the best, but 1+2 seems better than 3 alone as doing reads midstream in the write don't cause hangs. I haven't seen zombie processes ever.

Quoted 33 lines · Con Kolivas

Quoted 15 lines · Jens Axboe

Quoted 14 lines · Carl-Daniel Hailfinger

Quoted 12 lines · Jens Axboe

Quoted 8 lines · Marc-Christian Petersen

Quoted 3 lines · Andrew Morton

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoted 6 lines

Quoted 3 lines

Quoting Con Kolivas:

dd if=/dev/zero of=dump bs=4096 count=512000

already tried that, no go. on ide/scsi? what filesystem? how much ram? anything else running? smp/up?

I'm using UP on IDE. I reproduce it easily on a P3 256Mb laptop with 5400rpm drive, and less easily but still occurs on a P4 2.53 512Mb pc with 2x7200rpm software raid 0 IDE drives. Even if the only thing you try to do is move the mouse, the mouse will freeze for up to 30secs. When you first start the write no disk activity happens for up to a few seconds, then it will start writing madly and the machine will come to a standstill for a variable length of time. Then it will come back to life for a few seconds only to die again for a few seconds and so on till the write is complete.

Quoted 3 lines

Still testing combinations to see which is the best, but 1+2 seems better than 3 alone as doing reads midstream in the write don't cause hangs. I haven't seen zombie processes ever.

Just curious - which compiler did you use?

Quoted 1 line · Carl-Daniel Hailfinger

dd if=/dev/zero of=dump bs=4096 count=512000

already tried that, no go. on ide/scsi? what filesystem? how much ram? anything else running? smp/up?

Doesn't matter if IDE or SCSI, to be honest, SCSI with the old aic7xxx from vanilla 2.4.20 is even worse than IDE.

My box is up, had only my window manager with some open xterms running, nothing which should create any load.

Quoted 7 lines

Quoted 3 lines

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoting Con Kolivas:

dd if=/dev/zero of=dump bs=4096 count=512000

already tried that, no go. on ide/scsi? what filesystem? how much ram? anything else running? smp/up?

Quoted 6 lines · Jens Axboe

Quoted 6 lines

Quoted 3 lines

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoting Con Kolivas:

Quoted 4 lines

Quoted 1 line

dd if=/dev/zero of=dump bs=4096 count=512000

already tried that, no go. on ide/scsi? what filesystem? how much ram? anything else running? smp/up?

I think we've got a few different problems. On SMP boxes, you need to have the fix-pausing patch from andrea applied to catch all the corner cases.

On UP boxes it's possible the requests are starving in the drive, SCSI users should try with the max tags set down to something sensible, between 8 and 32.

IDE people can try lowering the max_kb_per_request paramater in /proc/ide/<drive>/settings, but this should only affect starvation with the writeback cache on.

I made a patch a while ago that timed how long people spent waiting in __get_request_wait, it might help us figure out where the starvation is really happening.

Quoted 8 lines

On UP boxes it's possible the requests are starving in the drive, SCSI users should try with the max tags set down to something sensible, between 8 and 32.

IDE people can try lowering the max_kb_per_request paramater in /proc/ide/<drive>/settings, but this should only affect starvation with the writeback cache on.

Quoted 3 lines

I made a patch a while ago that timed how long people spent waiting in __get_request_wait, it might help us figure out where the starvation is really happening.

But this seems totally unrelated to the reported problems, we are talking about complete stalls of the mouse. No amount of io starvation should provoke something like that.

Quoted 12 lines · Jens Axboe

Quoted 11 lines

On UP boxes it's possible the requests are starving in the drive, SCSI users should try with the max tags set down to something sensible, between 8 and 32.

IDE people can try lowering the max_kb_per_request paramater in /proc/ide/<drive>/settings, but this should only affect starvation with the writeback cache on.

I made a patch a while ago that timed how long people spent waiting in __get_request_wait, it might help us figure out where the starvation is really happening.

Quoted 3 lines

But this seems totally unrelated to the reported problems, we are talking about complete stalls of the mouse. No amount of io starvation should provoke something like that.

Well, if it wasn't io related starvation, andrew's batch requests patch wouldn't change things. I'm hoping the stats patch will get us some numbers to go along with the perceived stalls, almost done merging.

Correction then, it doesn't appear to be starvation in the usual sense. But you are right, pulling some stats out of the situation would be nice. I still can't reproduce here.

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoted 6 lines · Marc-Christian Petersen

Quoted 3 lines

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoted 6 lines

Quoted 3 lines

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

May I ask how you are reproducing the bad results? I'm trying in vain here...

I can reproduce across spindles with cvs import'ing a kernel tree, make sure you're running X11 and try and do things in it, e.g. scrolling windows, dragging etc.

Quoted 11 lines · Marc-Christian Petersen

On Wednesday 28 May 2003 13:27, Andrew Morton wrote:

Hi Akpm,

Quoted 2 lines

Quoted 1 line

Does the attached one make sense?

Nope.

nm.

Quoted 3 lines

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

Quoted 3 lines

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

May I ask how you are reproducing the bad results? I'm trying in vain here...

Quoted 19 lines · Jens Axboe

Quoted 17 lines · Marc-Christian Petersen

On Wednesday 28 May 2003 13:27, Andrew Morton wrote:

Hi Akpm,

Quoted 3 lines

Quoted 1 line

Does the attached one make sense?

Nope.

nm.

Quoted 3 lines

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

Quoted 2 lines

May I ask how you are reproducing the bad results? I'm trying in vain here...

It might be useful to check what video hardware and X servers people are using here. If the behavior is just mouse freezups, the "silken mouse" feature of XFree might have some effect, since it involves XFree binding a signal to mouse device events.

Quoted 20 lines · Elladan

Quoted 19 lines · Jens Axboe

Quoted 17 lines · Marc-Christian Petersen

On Wednesday 28 May 2003 13:27, Andrew Morton wrote:

Hi Akpm,

Quoted 3 lines

Quoted 1 line

Does the attached one make sense?

Nope.

nm.

Quoted 3 lines

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

Quoted 7 lines

Quoted 2 lines

May I ask how you are reproducing the bad results? I'm trying in vain here...

It might be useful to check what video hardware and X servers people are using here. If the behavior is just mouse freezups, the "silken mouse" feature of XFree might have some effect, since it involves XFree binding a signal to mouse device events.

Xfree 3.3.6, 4.2,4.3
Drivers nvidia, nv, sis, sisfb, vesa, vesafb

are the drivers on the machines where I've seen it happen so far - ie without discrimination.

Quoted 29 lines · Con Kolivas

Quoted 20 lines · Elladan

Quoted 19 lines · Jens Axboe

Quoted 17 lines · Marc-Christian Petersen

On Wednesday 28 May 2003 13:27, Andrew Morton wrote:

Hi Akpm,

Quoted 3 lines

Quoted 1 line

Does the attached one make sense?

Nope.

nm.

Quoted 3 lines

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

Quoted 7 lines

Quoted 2 lines

May I ask how you are reproducing the bad results? I'm trying in vain here...

It might be useful to check what video hardware and X servers people are using here. If the behavior is just mouse freezups, the "silken mouse" feature of XFree might have some effect, since it involves XFree binding a signal to mouse device events.

Quoted 5 lines

Xfree 3.3.6, 4.2,4.3
Drivers nvidia, nv, sis, sisfb, vesa, vesafb

are the drivers on the machines where I've seen it happen so far - ie without discrimination.

what about the window manager? do you use focus follow mouse? Just trying to find a pattern. For the record KDE 3.1 + focus follow mouse and X 4.3.0 here, I guess Jens uses the same software combination. the mouse for me is always perfectly fluid no matter how fast and how long I write, no matter if I don't touch the mouse for minutes, ALT+TAB as well. I definitely can't reproduce in any way the mouse stalls (I'm using cp /dev/zero . on a ext3 fs in ordered mode). hardware is 1G of ram smp IDE single spindle primary master matrox GS450. I almost couldn't notice the background write flood if I only would increase the xmms buffer (infact I thought it stopped writing for a dozen seconds out of space, and instead it was still writing). (kernel is 2.4.21rc4aa1 of course)

Quoted 3 lines

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

As I said, I will test it this evening. ATM I don't have time to recompile and reboot. This evening I will test extensively, even on SMP, SCSI, IDE and so on.

Quoted 1 line · Marc-Christian Petersen

Does the attached one make sense?

Nope.

Guys, you're the ones who can reproduce this. Please spend more time working out which chunk (or combination thereof) actually fixes the problem. If indeed any of them do.

I'm suspecting that Con's fingers slipped.

On Wednesday 28 May 2003 12:59, Andrew Morton wrote:

Hi Andrew,

Quoted 6 lines

umm, I'd like confirmation of that.

The waitqueue_active() test is wrong because of a missing barrier, but only on SMP. And if it does make a mistake it will surely correct itself when the next request is put back. (That's why I left it there...) More testing, please.

Does the attached one make sense?

ciao, Marc

Quoted 4 lines · Jens Axboe

Quoted 2 lines · Andrew Morton

Then this (totally unlikely, don't bother):

Quoted 17 lines

Quoted 5 lines

diff -puN drivers/block/ll_rw_blk.c~3 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c void blkdev_release_request(struct reque · line 835

Quoted 9 lines

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~3 +1 -2

@@ -829,8 +829,7 @@ void blkdev_release_request(struct reque
 	 */
 	if (q) {
 		list_add(&req->queue, &q->rq[rw].free);
-		if (++q->rq[rw].count >= q->batch_requests &&
-				waitqueue_active(&q->wait_for_requests[rw]))
+		if (++q->rq[rw].count >= q->batch_requests)
 			wake_up(&q->wait_for_requests[rw]);
 	}
 }

Well it's the only one left :). But you are right, try one of them at the time, establishing the effect of each of them.

THIS IS IT! The last one. No pauses writing a 2Gb file now unless I do a read midstream.

Quoted 1 line · Matthias Mueller

Works fine on my notebook. Good throughput and no mouse hangs anymore.

Interesting.

Could you please work out which change caused it? Go back to stock 2.4 and then apply this:

diff -puN drivers/block/ll_rw_blk.c~1 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +0 -0

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;

Quoted 7 lines

diff -puN drivers/block/ll_rw_blk.c~1 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +0 -0

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 

drivers/block/ll_rw_blk.c static struct request *__get_request_wai · line 599

Quoted 8 lines

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +1 -1

-	generic_unplug_device(q);
 	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
+		generic_unplug_device(q);
 		if (q->rq[rw].count == 0)
 			schedule();
 		spin_lock_irq(&io_request_lock);

It's not this because this is the layout in my -ck* and it still exhibits the pauses.

Could you please work out which change caused it? Go back to stock 2.4 and then apply this:

diff -puN drivers/block/ll_rw_blk.c~1 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +1 -1

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	generic_unplug_device(q);
 	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
+		generic_unplug_device(q);
 		if (q->rq[rw].count == 0)
 			schedule();
 		spin_lock_irq(&io_request_lock);

then this:

diff -puN drivers/block/ll_rw_blk.c~2 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~2 +1 -1

@@ -590,7 +590,7 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
+	add_wait_queue(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		generic_unplug_device(q);

Then this (totally unlikely, don't bother):

diff -puN drivers/block/ll_rw_blk.c~3 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~3 +1 -2

@@ -829,8 +829,7 @@ void blkdev_release_request(struct reque
 	 */
 	if (q) {
 		list_add(&req->queue, &q->rq[rw].free);
-		if (++q->rq[rw].count >= q->batch_requests &&
-				waitqueue_active(&q->wait_for_requests[rw]))
+		if (++q->rq[rw].count >= q->batch_requests)
 			wake_up(&q->wait_for_requests[rw]);
 	}
 }

_

Tested all of them and some combinations:
patch 1 alone: still mouse hangs
patch 2 alone: still mouse hangs
patch 3 alone: no hangs, but I get some zombie process (starting a lot of
xterms results in zombie xterms, not noticed with vanilla and the other patches)
patch 1+2: no mouse hangs
patch 1+2+3: no mouse hangs, no zombies

Quoted 8 lines · Matthias Mueller

Tested all of them and some combinations:
patch 1 alone: still mouse hangs
patch 2 alone: still mouse hangs
patch 3 alone: no hangs, but I get some zombie process (starting a lot of
xterms results in zombie xterms, not noticed with vanilla and the other patches)
patch 1+2: no mouse hangs
patch 1+2+3: no mouse hangs, no zombies

Forgot to mention: no zombies with patch 1 or 2

Quoted 12 lines · Matthias Mueller

Quoted 8 lines · Matthias Mueller

Tested all of them and some combinations:
patch 1 alone: still mouse hangs
patch 2 alone: still mouse hangs
patch 3 alone: no hangs, but I get some zombie process (starting a lot of
xterms results in zombie xterms, not noticed with vanilla and the other patches)
patch 1+2: no mouse hangs
patch 1+2+3: no mouse hangs, no zombies

Quoted 1 line

Forgot to mention: no zombies with patch 1 or 2

So 1+2 gives you zombies?

Quoted 5 lines · Carl-Daniel Hailfinger

Quoted 3 lines · Matthias Mueller

On Wed, May 28, 2003 at 02:10:40PM +0200, Matthias Mueller wrote:

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Tested all of them and some combinations:

patch 1 alone:    hangs, no zombies
patch 2 alone:    hangs, no zombies
patch 3 alone: no hangs,    zombies

patch 1+2: no hangs, no zombies
patch 1+2+3: no hangs, no zombies

Quoted 7 lines · Matthias Mueller

Quoted 5 lines · Carl-Daniel Hailfinger

Quoted 3 lines · Matthias Mueller

On Wed, May 28, 2003 at 02:10:40PM +0200, Matthias Mueller wrote:

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Tested all of them and some combinations:

patch 1 alone:    hangs, no zombies
patch 2 alone:    hangs, no zombies
patch 3 alone: no hangs,    zombies

patch 1+2: no hangs, no zombies
patch 1+2+3: no hangs, no zombies

Right?

Tested all of them and some combinations:
patch 1 alone: still mouse hangs
patch 2 alone: still mouse hangs
patch 3 alone: no hangs, but I get some zombie process (starting a lot of
xterms results in zombie xterms, not noticed with vanilla and the other patches)
patch 1+2: no mouse hangs
patch 1+2+3: no mouse hangs, no zombies

Quoted 8 lines · Matthias Mueller

Tested all of them and some combinations:
patch 1 alone: still mouse hangs
patch 2 alone: still mouse hangs
patch 3 alone: no hangs, but I get some zombie process (starting a lot of
xterms results in zombie xterms, not noticed with vanilla and the other patches)
patch 1+2: no mouse hangs
patch 1+2+3: no mouse hangs, no zombies

I can't find a sense in the zombie thing, how can you generate zombie at all from xterms? That sounds like your userspace is terribly broken and it may have race conditions or whatever. In no way those patches can generate or not-generate zombies from xterms. I never ever seen a zombie xterm in my whole linux experience.

Quoted 10 lines · Andrea Arcangeli

Quoted 8 lines · Matthias Mueller

Tested all of them and some combinations:
patch 1 alone: still mouse hangs
patch 2 alone: still mouse hangs
patch 3 alone: no hangs, but I get some zombie process (starting a lot of
xterms results in zombie xterms, not noticed with vanilla and the other patches)
patch 1+2: no mouse hangs
patch 1+2+3: no mouse hangs, no zombies

Quoted 5 lines

I can't find a sense in the zombie thing, how can you generate zombie at all from xterms? That sounds like your userspace is terribly broken and it may have race conditions or whatever. In no way those patches can generate or not-generate zombies from xterms. I never ever seen a zombie xterm in my whole linux experience.

I rechecked everything an noticed, that it wasn't a xterm, but a wrapper script, that executed rxvt. I changed that to plain xterm and the zombies were gone. So I think there was probably a bug in rxvt triggered there. After that I redid the tests, with the same result (and no zombies). I can feel no difference between 1+2 or 1+2+3.

Could you please work out which change caused it? Go back to stock 2.4 and then apply this:

then this:

Then this (totally unlikely, don't bother):

Quoted 3 lines · Andrew Morton

Could you please work out which change caused it? Go back to stock 2.4 and then apply this:

[snip] 1

Quoted 1 line

then this:

[snip] 2

Quoted 1 line

Then this (totally unlikely, don't bother):

[snip] 3

Ok patch combination final score for me is as follows in the presence of a large continuous write:
1 No change
2 No change
3 improvement++; minor hangs with reads
1+2 improvement+++; minor pauses with switching applications 1+2+3 improvement++++; no pauses

Applications may start up slowly that's fine. The mouse cursor keeps spinning and responding at all times though with 1+2+3 which it hasn't done in 2.4 for

a year or so.

Con

Hello !

I've done a few tests with -rc6 on my dev machine (dual xp 1.5G, 512 MB, scsi). It's the *FIRST* time I have ever seen my mouse cursor hang (just a little bit however, and totally acceptable) ! Usually, my kernel include -aa VM and lowlat patches, and I've never encountered this behaviour on this machine with such a configuration. However, with stock kernel, I admit that during the 2 minutes it takes to write the 2G file, I see the mouse stick two or three times during about 1 second, which is quite acceptable IMHO. Opening an xterm may take 10s to get to the prompt (more annoying). Same to launch 'ps'.

I use a fairly simple window manager (ctwm), which doesn't access the disk once it's launched. It never gets stuck during all the operation if I disable the swap. If I enable the swap, it sometimes takes one or two seconds to draw a menu. The swap is used up to about 4 MB.

I then tried -rc6 with ll_rw_blk from -rc5, and it's worse, even with swap disabled. The hangs happen more often, but are about the same durations. So I confirm that -rc6 is better here than -rc5.

I retried with rc4aa1, and everything went very smooth again ; it takes at most 1 second to get an xterm with the prompt ready, and ps responds immediately. So I think that there are two things here:
- those who experience very long hangs may use a heavy window manager which does continuous disk accesses (I mean it accesses the disk for
any simple operation).
- a hungry WM may also be swapped during such operations, rendering it totally unusable, particularly if the swap is on the same physical disk as the file being written to.

So, could the people who report long hangs retry with swap disabled ? Can we limit the amount of memory consummed by the cache during such a write ?

Hello !

I've done a few tests with -rc6 on my dev machine (dual xp 1.5G, 512 MB, scsi). It's the *FIRST* time I have ever seen my mouse cursor hang (just a little bit however, and totally acceptable) ! Usually, my kernel include -aa VM and lowlat patches, and I've never encountered this behaviour on this machine with such a configuration. However, with stock kernel, I admit that during the 2 minutes it takes to write the 2G file, I see the mouse stick two or three times during about 1 second, which is quite acceptable IMHO. Opening an xterm may take 10s to get to the prompt (more annoying). Same to launch 'ps'.

I use a fairly simple window manager (ctwm), which doesn't access the disk once it's launched. It never gets stuck during all the operation if I disable the swap. If I enable the swap, it sometimes takes one or two seconds to draw a menu. The swap is used up to about 4 MB.

I then tried -rc6 with ll_rw_blk from -rc5, and it's worse, even with swap disabled. The hangs happen more often, but are about the same durations. So I confirm that -rc6 is better here than -rc5.

I retried with rc4aa1, and everything went very smooth again ; it takes at most 1 second to get an xterm with the prompt ready, and ps responds immediately. So I think that there are two things here:
- those who experience very long hangs may use a heavy window manager which does continuous disk accesses (I mean it accesses the disk for any simple operation).
- a hungry WM may also be swapped during such operations, rendering it totally unusable, particularly if the swap is on the same physical disk as the file being written to.

So, could the people who report long hangs retry with swap disabled ? Can we limit the amount of memory consummed by the cache during such a write ?

I run fluxbox, not a very heavy window manager, but I installed ctwm and tried again with vanilla 2.4.20. If I disabled swap the short hangs (1s) are gone, but the long mouse hangs (10s) are still there.

I've done a few tests with -rc6 on my dev machine (dual xp 1.5G, 512 MB, scsi). It's the *FIRST* time I have ever seen my mouse cursor hang (just a little bit however, and totally acceptable) ! Usually, my kernel include -aa VM and lowlat patches, and I've never encountered this behaviour on this machine with such a configuration. However, with stock kernel, I admit that during the 2 minutes it takes to write the 2G file, I see the mouse stick two or three times during about 1 second, which is quite acceptable IMHO.

Opening an xterm may take 10s to get to the prompt (more annoying). Same to launch 'ps'.

I retried with rc4aa1, and everything went very smooth again ; it takes at most 1 second to get an xterm with the prompt ready, and ps responds immediately. So I think that there are two things here:
- those who experience very long hangs may use a heavy window manager which does continuous disk accesses (I mean it accesses the disk for any simple operation).
- a hungry WM may also be swapped during such operations, rendering it totally unusable, particularly if the swap is on the same physical disk as the file being written to.

So, could the people who report long hangs retry with swap disabled ?

Can we limit the amount of memory consummed by the cache during such a write ?

Quoted 3 lines

machine with such a configuration. However, with stock kernel, I admit that during the 2 minutes it takes to write the 2G file, I see the mouse stick two or three times during about 1 second, which is quite acceptable IMHO.

WRONG. A mouse stick is not acceptable in _any_ way. Other OS' can handle this

Quoted 2 lines

Opening an xterm may take 10s to get to the prompt (more annoying). Same to launch 'ps'.

ACK!

The pauses/stops occurs no matter of what WindowManager (KDE2/3, WindowMaker, fvwm, gnome etc. foobar). The point why you are not seeing such things with -aa is his Lowlatency Elevator and lowlatency-fixes and some important fixes which are not in stock kernel yet.

I reproduced mouse sticks and keyboard does not accept anything problems for $seconds with _every_ kernel which is based on 2.4.19/2.4.20/2.4.21*. This also includes -AA (well, not that braindead bad like mainline did before the fix) but this is because of lowlat elevator from Andrea. And as I told yesterday (or 2 days ago? dunno) lowlat elevator drops throughput (Andrea, it _does_ ;).

It's not just only mouse hangs (as I've reported tons of times) but also keyboard does not accept any input (delay varies between 1 to 15 seconds) and this also applies if you don't run X at all.

- Start a screen session, not running X at all.
- Trash your HD with tons of writes.
- Press Ctrl-A-C for a new screen session.

You will see, it takes as long as, you wrote above, with starting up an Xterm or calling ps. It does _not_ happen with 2.4.18!

Quoted 1 line

So, could the people who report long hangs retry with swap disabled ?

It's somewhat better but not acceptable.

Quoted 2 lines

Can we limit the amount of memory consummed by the cache during such a write ?

I ask for such a feature since years ;)

I also confirm it does ; it takes 122 seconds to write this file in -rc6, and 142 seconds in -aa. But I don't think that desktop people would notice anyway.

btw, were you running parallel reads or writes at the same time? (i.e. launching xterms or ps etc.. in parallel?) I ask because if xterm startups quick is because the write workload is getting more seeks in its way.

I'd be very interested if you can measure a bonnie performance change in contigous reads and writes on a otherwise completely idle machine, the size of the queue has to be big enough to keep the I/O pipeline full during contigous writes at full speed.

you can also try with:

	echo 20 500 0 0 500 3000 30 10 >/proc/sys/vm/bdflush

So, could the people who report long hangs retry with swap disabled ? Can we limit the amount of memory consummed by the cache during such a write ?

Quoted 2 lines · Matthias Mueller

Works fine on my notebook. Good throughput and no mouse hangs anymore.

Interesting.

Could you please work out which change caused it? Go back to stock 2.4 and then apply this:

diff -puN drivers/block/ll_rw_blk.c~1 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~1 +1 -1

@@ -590,10 +590,10 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	generic_unplug_device(q);
 	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
+		generic_unplug_device(q);
 		if (q->rq[rw].count == 0)
 			schedule();
 		spin_lock_irq(&io_request_lock);

then this:

diff -puN drivers/block/ll_rw_blk.c~2 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~2 +1 -1

@@ -590,7 +590,7 @@ static struct request *__get_request_wai
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	add_wait_queue_exclusive(&q->wait_for_requests[rw], &wait);
+	add_wait_queue(&q->wait_for_requests[rw], &wait);
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		generic_unplug_device(q);

Then this (totally unlikely, don't bother):

diff -puN drivers/block/ll_rw_blk.c~3 drivers/block/ll_rw_blk.c

drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c~3 +1 -2

@@ -829,8 +829,7 @@ void blkdev_release_request(struct reque
 	 */
 	if (q) {
 		list_add(&req->queue, &q->rq[rw].free);
-		if (++q->rq[rw].count >= q->batch_requests &&
-				waitqueue_active(&q->wait_for_requests[rw]))
+		if (++q->rq[rw].count >= q->batch_requests)
 			wake_up(&q->wait_for_requests[rw]);
 	}
 }

_

Quoted 1 line

It'd be interesting if any of these changes make a difference.

Works fine on my notebook. Good throughput and no mouse hangs anymore.

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/:
speak _NOW_ please, doesn't matter who you are!

Quoted 4 lines · Marc-Christian Petersen

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/:
speak _NOW_ please, doesn't matter who you are!

Ok, add my box to the list. Variety of post 2.4.18 kernels, -ac's, -rc's, etc... all demonstrate it to one degree or another.

Lately it's gotten REALLY bad.

Currently I'm using 21-rc2-ac2 and it freezes for upwards of 15 sec regularly when I'm exercising the HD (three simultaneous brag threads downloading from various newsgroups). The mouse moves, but other than that, X is entirely unresponsive. An xterm with continually scrolling text, for example, will appear to stop scrolling until the kernel comes back.

The HD light is on solid the whole time.

21-rc2 does it too. I haven't tried anything later than that yet. Well, I tried 20-ck7 and it ate my RAID0 due to a DMA-ism and I've not tested anything else since. :(

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/:
speak _NOW_ please, doesn't matter who you are!

Quoted 3 lines · Marc-Christian Petersen

ALL: Anyone who has this kind of pauses/stops/mouse is dead/keyboard is dead/:
speak _NOW_ please, doesn't matter who you are!

FWIW, me too.

Actually it just happens in the fixing stage when burning prebuilt iso images from the hard disk (same IDE channel as the burner, 2.4.20) Having a completely frozen machine under X was quite panic inducing ;)

A friend told me they also get regular "pauses" when quitting from vmware.

Actually it just happens in the fixing stage when burning prebuilt iso images from the hard disk (same IDE channel as the burner, 2.4.20) Having a completely frozen machine under X was quite panic inducing ;)

A friend told me they also get regular "pauses" when quitting from vmware.

On Wednesday 28 May 2003 11:36, Ragnar Hojland Espinosa wrote:

Hi Ragnar,

Quoted 3 lines

Actually it just happens in the fixing stage when burning prebuilt iso images from the hard disk (same IDE channel as the burner, 2.4.20) Having a completely frozen machine under X was quite panic inducing ;)

That's a problem of IDE itself. I still say IDE is broken by design ;-)

Actually it just happens in the fixing stage when burning prebuilt iso images from the hard disk (same IDE channel as the burner, 2.4.20) Having a completely frozen machine under X was quite panic inducing ;)

Quoted 3 lines · Ragnar Hojland Espinosa

Actually it just happens in the fixing stage when burning prebuilt iso images from the hard disk (same IDE channel as the burner, 2.4.20) Having a completely frozen machine under X was quite panic inducing ;)

If you have a disk and the burner ont he same channel this is quite normal. The fixate is a single ATAPI command and like all ATA commands locks the bus to both master/slave for its duration of execution.

Its an IDE limitation