Degraded I/O performance, since 2.5.41

When I run a certain large webserver benchmark, I prefer to warm the pagecache up with the file set first, to cheat a little :) I grep through 20 different 500MB file sets in parallel to do this. It is a _lot_ slower in the BK snapshot than in plain 2.5.41.

And, no, these numbers aren't inflated, I have a lot of fast disks. I
_can_ do 50MB/sec :)

A little snipped from vmstat (I cut out the boring columns):
good kernel: 2.5.41: vmstat 4
Cached bi bo bi cs us s id

389280     53284     7 1625  3235 12 88  0
600580     53489    19 1599  3264 11 89  0
813428     53891     0 1587  3256 12 88  0
1027260    54093     0 1609  3239 12 88  0
1241448    54183     0 1611  3251 11 89  0
1454036    53790     0 1618  3267 12 88  0

doing the entire 10GB grep takes 192 seconds.
a dd produces: ~48000 bi/sec

exact same grep operation on kernel: 2.5.41+yesterday's bk: vmstat 4
Cached bi bo bi cs us sy id

4855948    9697     1 1408   846 20 80  0
4890464    8745     0 1398   800 18 82  0
4922392    8077    55 1364   676 21 79  0
4959164    9296     1 1399   798 18 82  0
4995936    9315     0 1407   830 19 81  0
5027208    7931     0 1351   638 22 78  0
5066256    9855     9 1416   856 19 81  0

I was too impatient to wait on the greps to complete. a dd produces: ~37800 bi/sec

So, bi/sec goes from 54,000 in 2.5.41, to ~8700 in yesterday's snapshot. It goes from around 50MB/sec to about 8MB/sec.

Although vmstat shows 0% idle time, the profilers show lots of idle time, 98%! I tried oprofile and readprofile. Is the 2.0.9 vmstat still broken? I'm using idle=poll if that makes any difference.

Dave Hansen
[email protected]