Multi disk performance (8 disks), limit 230 MB/s
Hi all,
I have been doing some benchmarking experiments on kernel 2.4.19 with 8 IDE disks. Due to poor performance of 2 disks at 1 IDE channels we have bought 4 Promise Ultra 100 TX2 (32-bit 66 Mhz) controllers and to avoid bus saturation Supermicro P4PDE motherboard with multiple PCI buses (64-bit 66 Mhz) and 2-Xeons. I submitted already PCI slot placing problems to the mailing list. But theoretically I can live with the current IDE condrollers->PCI slots assighnment.
The assignment is the following: 3 IDE controllers are connected to the one PCI 64-bit Hub with bandwidth 1 GByte/s and 4th controller is on another hub with the same characteristics.
Theoretically with 6 IBM disks (47 MB/s from the first cylinders) I should achieve a number about 266 MB/s (32 bit X 66 Mhz) < 6*47. AND 2*47 = 94 MB/s < 266 MB/s from the last two disks. Thus the rate should be 94 + 266 = 360 MB/s.
BUT no matter from which set of the disks I read or write I have got the
following parallel read/write rates (raw access):
write (MB/s) read (MB/s) systime (top) real/user/sys(time) (s)
1 disk : 48 45 3 % 3.0 / 0.1 / 0.4 2 disks: 83 94 10 % 3.5 / 0.1 / 0.6 4 disks: 131 189 21 % 4.3 / 0.4 / 2.8 5 disks: 172 233 4.5 / 0.5 / 4.5 6 disks: 197 234 ? 30 % 5.2 / 0.6 / 6.6 7 disks: 209 ? 230 ? 5.9 / 0.6 / 8.8 8 disks: 214 ? 229 ? 40 % 6.7 / 0.8 /10.8
The method of the measurement:
1. Prespawn thread for every disk
2. signal threads to start write of 64 Mb of data
3. wait until all threads finished, measure time
4. signal threads to start read of 64 Mb of data
5. wait until all threads finished, measure time
All disks are in UDMA 100 mode
Has anyone else seen similar problems?
What limits the performance: IDE disk driver, bottleneck in the kernel I/O subsystem?
How to improve?
Roman
Quoted 26 lines · Roman Dementiev
Hi all,
I have been doing some benchmarking experiments on kernel 2.4.19 with 8 IDE disks. Due to poor performance of 2 disks at 1 IDE channels we have bought 4 Promise Ultra 100 TX2 (32-bit 66 Mhz) controllers and to avoid bus saturation Supermicro P4PDE motherboard with multiple PCI buses (64-bit 66 Mhz) and 2-Xeons. I submitted already PCI slot placing problems to the mailing list. But theoretically I can live with the current IDE
condrollers->PCIslots assighnment.The assignment is the following: 3 IDE controllers are connected to the one PCI 64-bit Hub with bandwidth 1 GByte/s and 4th controller is on another hub with the same characteristics.
Theoretically with 6 IBM disks (47 MB/s from the first cylinders) I should achieve a number about 266 MB/s (32 bit X 66 Mhz) < 6*47. AND 2*47 = 94 MB/s < 266 MB/s from the last two disks. Thus the rate should be 94 + 266 = 360 MB/s.
BUT no matter from which set of the disks I read or write I have got the
following parallel read/write rates (raw access):
write (MB/s) read (MB/s) systime (top) real/user/sys(time) (s)
Quoted 8 lines
1 disk : 48 45 3 % 3.0 / 0.1 / 0.4 2 disks: 83 94 10 % 3.5 / 0.1 / 0.6 4 disks: 131 189 21 % 4.3 / 0.4 / 2.8 5 disks: 172 233 4.5 / 0.5 / 4.5 6 disks: 197 234 ? 30 % 5.2 / 0.6 / 6.6 7 disks: 209 ? 230 ? 5.9 / 0.6 / 8.8 8 disks: 214 ? 229 ? 40 % 6.7 / 0.8 /10.8
raw access in 2.4 isn't very good - it uses 512-byte chunks. If you can hunt down the `rawvary' patch that might help, but I don't know if it works against IDE.
Testing 2.5 would be interesting ;)
Try the same test with O_DIRECT reads or writes against ext2 filesystems. That will use 4k blocks.
Be sceptical about the `top' figures. They're just statistical and are subject to gross errors under some circumstances - you may have run out of CPU (unlikely with direct IO).
direct IO is synchronous: the kernel waits for completion of each IO request before submitting the next. You _must_ submit large IO's. That means passing 128k or more to each invokation of the
read and write system calls. See the bandwidth-versus-request size
numbers I measured for O_DIRECT ext3:
http://www.geocrawler.com/lists/3/SourceForge/7493/0/9329947/
Quoted 10 lines · Andrew Morton
Quoted 3 lines · Roman Dementiev
8 disks: 214 ? 229 ? 40 % 6.7 / 0.8 /10.8
raw access in 2.4 isn't very good - it uses 512-byte chunks. If you can hunt down the `rawvary' patch that might help, but I don't know if it works against IDE.
Testing 2.5 would be interesting ;)
I still can't run it, 'interrupt lost' ...
Quoted 4 lines
Try the same test with
O_DIRECTreads or writes against ext2 filesystems. That will use 4k blocks.
Thanx, it works. I have got 375 MB/s.
Roman