2.6 vm/elevator loading down disks where 2.4 does not

Sorry for the long email. I pruned it as much as possible...

The problem:

When we upgraded one of our production boxes (details below) to 2.6.6, we noticed an immediate loss of 5 - 15 percent efficiency. While these boxes usually had less than 0.5% variation through out the day, this box was consistently doing 10% fewer searches than the others.

Upon investigation, we saw that the 2.6 box was reading from the disk about 5 times as much as 2.4. Iin 2.4 we can almost completely saturate the CPUs; they'll get to 90% of the real CPU's, and 15% of the virtual CPUs. With 2.6, they never get above 60/10 because they are in io-wait state constantly (which, under 2.4, is reported as idle IIRC). I have not done extensive testing of the anticipatory elevator, but it did appear slower than deadline in early tests.

The vmstat runs at the bottom of this email were done in parallel on two machines, receiving mostly identical amounts of real traffic. Traffic is load balanced by mod_backhand to these machines, and is directly responsive to system load with a granularity of 2 seconds, so really, the 2.6.6 box was actually getting somewhat *less* traffic. Notice how much higher the 'bi' numbers are, for blocks in. As I said before, expected variation is less tham 0.5%.

This behavior is consistent and has been observed for over a month in production now. I'm just looking for reasons this is happening and maybe what needs to be profiled/tuned/fixed in order to find out.

If you're still interested by this point, here is the Background:

We have a bunch of identical Dual P4 Xeon 2.8Ghz machines. Each has an Intel i865 chipset, 1GB of DDR RAM, and 2xWDC 40GB 8MB cache drives. All are running RedHat 8.0 with security patches from fedoralegacy.org. All are running vanilla kernel 2.4.23 or later, except one, that runs 2.6.6. The 2.6.6 kernel was built directly from one of the other kernel's .config with 'make oldconfig'. Two new options were selected.. CONFIG_4KSTACKS, and (on the cmdline) elevator=deadline. The two disks are setup in software RAID1. There is no swap configured.

These machines run text searches from web requests using a proprietary file-based database (called Texis, http://www.thunderstone.com). The data access patterns are generally "search through index files in a tree-walking type of manner, then seek to data records in data files." The index files are less than 300MB, and constantly accessed. The data files total 3GB, but the data being read is very small... at most 40kB at a time.

And now for the real details:

---------------------VMSTAT 2.4.23-------------------------

$ free -m ; uptime ; vmstat 5 5 
             total       used       free     shared    buffers    
cached 
Mem:          1007        983         24          0         12       
812 
-/+ buffers/cache:        157        850 
Swap:            0          0          0 
 14:51:06 up 113 days, 22:22,  1 user,  load average: 0.70, 0.59, 0.64 
procs -----------memory---------- ---swap-- -----io---- --system-- 
----cpu---- 
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us 
sy id wa 
 0  0      0  25284  13284 832500    0    0     1     1    1     1 12 
2 86  0 
 1  0      0  20748  13312 832688    0    0    37    39  139    92  3 
0 97  0 
 3  0      0  16720  13348 833020    0    0    63    80  189   222 12 
2 86  0 
 0  0      0  23672  13376 833184    0    0    31    57  166   142  7 
1 92  0 
 1  0      0  16572  13412 833288    0    0    20    51  155   137  5 
1 93  0 

------------------END VMSTAT 2.4.23-------------------------

---------------------VMSTAT 2.6.6-------------------------

$ free -m ; uptime ; vmstat 5 5 
             total       used       free     shared    buffers    
cached 
Mem:          1010        990         20          0         27       
867 
-/+ buffers/cache:         95        915 
Swap:            0          0          0 
 14:51:05 up 7 days,  1:17,  1 user,  load average: 0.59, 0.66, 0.76 
procs -----------memory---------- ---swap-- -----io---- --system-- 
----cpu---- 
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us 
sy id wa 
 0  1      0  20732  28220 888556    0    0    11    12   11    14  9 
2 88  2 
 0  0      0  27552  28260 883416    0    0   223   107  198   232 19 
3 76  2 
 0  0      0  26452  28276 884420    0    0   226    33  192   217  6 
1 87  6 
 0  0      0  26388  28308 884660    0    0    36    56  154   136  4 
0 95  0 
 0  0      0  25536  28344 885236    0    0   114    62  173   186  8 
2 89  1

------------------END VMSTAT 2.6.6-------------------------

Possibly a memory zone problem. Could you try booting with "mem=896m" on the kernel command line, see how that affects things?

This took a while longer to try, as I didn't want to unfairly test it against a box with 1G of RAM. So I rebooted my 2.4.23 and 2.6.7-rc3 test boxes with mem=896m. No change in the 5:1 ratio when comparing 2.6's disk reads to 2.4's. Of course, both boxes ended up reading from the disk more often, as they had less RAM for cache. I was unable to run a long test as I did before, but I'm confident the 3 hours I did run tests for show that this isn't a memory zone problem.

I still think this behavior is happening because useful pages are being removed from the page cache too soon. Maybe this is happening because of excessive readahead?

-cb

I don't think this will account for the entire change in disk activity, but 2.6.7-pre? contains a fix for the read ahead code to prevent it from reading extra sectors when unneeded.

The fix applies to 'seeky database type loads' which...

...sounds like it may apply to you.

So, if you and your server have some time, you might try 2.6.7rc3 and see if it changes any of the numbers.

Ray Lee

I updated my 2.6 box to 2.6.7-rc3 a few hours ago. While its not really fair to pass judgement during such low-traffic times, things look about the same. Where i see the 2.4 box not even touching the disks for a minute or longer, the 2.6 box will hit it every 10 seconds at least, and hit it with 40 blocks or more. 3 hours should be plenty of time to get the indexes and data files mostly cached.

It almost seems to me like 2.6 is caching too much with each read, and therefore having to free other pages that really should have been left alone. This might even explain why 2.4 is maintaining a lot more free memory (75-80MB versus 2.6 having only 15-20MB free) and less cache. Oddly enough, I noticed that the blockdev command reports 256 as the readahead in 2.6.x, but 1024 in 2.4.x. I tried mucking with that value but it didn't make a whole lot of difference.

Might it help to have some swap available, if for nothing else but to make the algorithms work better? Really the box should never use it; there are no daemons that go unused for longer than 10 minutes ... and when I have turned on swap in the past, it uses less than 2MB of it. This machine, for all intents and purposes, should spend most of its time searching an index and database that are already cached.. 90% of the searches are on similar terms.

That brings up an interesting point... is there a system wide stat that tells me how effective the file cache is? I guess majfaults/s fits that bill to some degree.

/proc/vmstat should log global major/minor fault counters (actually summed on the fly per-cpu counters). I fixed those to properly report major and minor faults for 2.6. The analogous numbers where they are present are completely and utterly meaningless gibberish in 2.4.

-- wli