mpt performance differences

Hi there!

We have two machines, both with LSI MPT Fusion SCSI cards in them. They have the same CPUs, memory, etc. Their disk performance is quite different however.

The two machines are "crusaders" and "minsk". They're both dual Operton servers, with six 10k 72G SCA disks in them, arranged as software RAID-10. RAM in both is the same, 4G. Both are running Debian sarge AMD64.

Initally we dismissed the huge difference in hdparm -tT numbers as just hdparm not really being a useful test, so we've run bonnie with default settings, and there's still quite a difference between them:

Minsk
=====
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
minsk            8G 54409  98 147492  32 18438   5 25201  44 32090   6 601.4  1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  5569  99 +++++ +++ +++++ +++  5819  99 +++++ +++ 20239  99

/dev/md3:
Timing cached reads: 3432 MB in 2.00 seconds = 1715.40 MB/sec
Timing buffered disk reads: 224 MB in 3.02 seconds = 74.13 MB/sec

Crusaders
=========
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
crusaders        8G 52445  95 104509  19 21752   6 30754  54 86996  17 520.5  1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  5824  99 +++++ +++ +++++ +++  5917  99 +++++ +++ 19196  99

/dev/md1:
Timing cached reads: 3836 MB in 2.00 seconds = 1916.37 MB/sec
Timing buffered disk reads: 418 MB in 3.00 seconds = 139.17 MB/sec

The bit we're concerned about is the read speeds, block is way lower on minsk than crusaders.

Kernel used is 2.6.12-rc3 with 3.03.00 patches from lsil.com applied. I've tried various combinations of kernel, without any improvement. We've also tried the kernel on crusaders, but performance was even worse then (as in, around 3M/s, instead of 30 we get now..)

We suspect it's caused by a quirk of a newer version of the silicon on the controller, they're slightly different in lspci:

Minsk
=====
0000:03:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)
Subsystem: LSI Logic / Symbios Logic: Unknown device 1060
Flags: bus master, 66MHz, medium devsel, latency 128, IRQ 30 I/O ports at 3000 [size=256]
Memory at fe120000 (64-bit, non-prefetchable) [size=128K] Memory at fe100000 (64-bit, non-prefetchable) [size=128K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
Capabilities: [68] PCI-X non-bridge device.

Crusaders
=========
0000:03:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
Subsystem: LSI Logic / Symbios Logic: Unknown device 1060
Flags: bus master, 66MHz, medium devsel, latency 128, IRQ 30 I/O ports at 4000 [size=256]
Memory at fe120000 (64-bit, non-prefetchable) [size=128K] Memory at fe100000 (64-bit, non-prefetchable) [size=128K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
Capabilities: [68] PCI-X non-bridge device.

Has anyone else seen this effect, or aware of a possible cause?

Thanks!

David Zanetti <[email protected]>
Team Leader, Systems Administration
Catalyst IT Limited
+64-4-8032233 +64-21-402260