2.6.29.1 vs 2.6.27.21 ext4 performance problem
I have a server that is busy reading and writing several disks with ext4 fs. After booting 2.6.29.1 kernel I have noticed significant increase in read rate. I measure the read rate using a script which reads sysfs byte and access counts and calculates average per second.
Here is the output of the script from 2.6.29.1 kernel:
sda: read-rate=58932 (r=4163)
sdb: read-rate=2570440 (r=40763)
sdc: read-rate=3320403 (r=52756)
sdd: read-rate=3199639 (r=52885)
SUM: read-rate=9149414 (r=150567)
And this is output from 2.6.27.21:
sda: read-rate=32768 (r=4096)
sdb: read-rate=204800 (r=24576)
sdc: read-rate=204800 (r=22528)
sdd: read-rate=704512 (r=45056)
SUM: read-rate=1146880 (r=96256)
The rate is in bytes per second, "r" is accesses per second.
One can notice that the access rate is similar (the work load on the server is basically the same), but byte rate is much higher.
I don't yet know what causes this reading increase. Any ideas?
Signature · 1 line
Alexander.
Quoted 5 lines · Alexander V. Lukyanov
I have a server that is busy reading and writing several disks with ext4 fs. After booting 2.6.29.1 kernel I have noticed significant increase in read rate. I measure the read rate using a script which reads sysfs byte and access counts and calculates average per second.
Without knowing exactly what your workload is, it's hard to say for sure. One of the things which did go in after 2.6.27 was inode table readahead. The intent is to actually improve performance, although at least initially it might mean a bit more I/O activity.
Does the differences go away if you do echo 0 into /sys/fs/ext4/<dev>/inode_readahead_blks?
Are you seeing an increase in time, or decrease in transactions per second, or just that there is is an increase in read rates?
- Ted
Quoted 2 lines · Theodore Tso
Does the differences go away if you do echo 0 into /sys/fs/ext4/<dev>/inode_readahead_blks?
Yes! Thanks for the tip. I think the problem here is that there are lots of files (2M per file system) and the access is random.
Quoted 2 lines
Are you seeing an increase in time, or decrease in transactions per second, or just that there is is an increase in read rates?
Load average goes up from 6 to 30, and request latency increases.
Signature · 1 line
Alexander..