Skip to contentSkip to conversation

IO performance problems with 2.6.9

DISCUSSION Peter Zaitsev 2 participants 2 messages Oct 22, 2004

lore
Share & tools
lore
#0 lore raw

Hi,

I was testing Linux performance on 80 drives disk array today (8 hardware RAID volumes stripped by Linux RAID0 256K stripe)

I've initially spotted this by very poor performance with DBT2 benchmark, while later isolated test to "SysBench"
http://sysbench.sourceforge.net/, running random read/write tests on single 40G file.

The system is dual Opteron 2.4 with 8G of memory.

ReiserFS file system (data=writeback,noatime,notail) was used.

Results I'm getting in IOs/Sec with 256 concurrent threads are:

READ WRITE

Buffered IO 16K     6185     2496 
O_DIRECT 16K        5169      471
Buffered IO 4K      7091     4758
O_DIRECT    4K      7676      571 

Note I had only 1Gb FC connection so 16K reads saturated about 5xxx/sec

For Comparison I also did the run with SUSE SLES Kernel (2.6.5-7.97SMP)

                    READ     MIXED (R/W)
Buffered IO 16K     2375     2520
O_DIRECT 16K        5391     1052
Buffered IO 4K      9576     5736
O_DIRECT    4K      8063     1041 

Sorry I did not do Exactly the same run due to lack of time accessing hardware.

The issues with these results are:

1) SuSE kernel does "Buffered" reads in 4K reads instead of 16K reads. This can be seen in "iostat -x". This seems to be fixed in 2.6.9

2) 2.6.9 does writes in 4K requests instead of 16K. This also can be seen in "iostat -x"

3) O_DIRECT writes are simply broken :)
One can see in vmstat only 1 thread will be in "b" if this mode is used and only one outstanding request will be submitted to SCSI controller at the same time. Interesting enough if I use 128 files totaling the same size O_DIRECT shows good performance. I thought inode locking problem was removed in 2.6 :(

4) Unrelated but still unfortunate. 2.6.9 kernel seems to behave weird if swap is disabled. I was running with 8G of memory allocating 6G for MySQL buffers which left about 1.5G for kernel, 1G of which was used
for file cache. During IO intensive run, using buffered IO I got
"kswapd" running like a crazy taking 90% of CPU time on one of CPUs. What for is it running if there is no swap files enabled ?

Signature · 2 lines
Peter Zaitsev, Senior Support Engineer
MySQL AB, www.mysql.com
#1 lore raw
Quoted 6 lines · Peter Zaitsev

4) Unrelated but still unfortunate. 2.6.9 kernel seems to behave weird
if swap is disabled. I was running with 8G of memory allocating 6G for
MySQL buffers which left about 1.5G for kernel, 1G of which was used
for file cache. During IO intensive run, using buffered IO I got
"kswapd" running like a crazy taking 90% of CPU time on one of CPUs. What for is it running if there is no swap files enabled ?

Can you boot with profile=2, and get a profile of about 30 seconds while kswapd is going crazy please?

Also capture /proc/vmstat before and after that interaval.

Can you also capture a the output of vmstat 1 while this is happening

Thanks