Skip to contentSkip to conversation

3Ware Hard Bus Hang 2.4.18 > 220 MB/S

DISCUSSION Jeff V. Merkey 2 participants 4 messages Feb 27, 2002

lore
Share & tools
lore
#0 lore raw

Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am seeing about 216-224 MB/S total throughput on writes to local arrays on 2.4.18.

The system is also running an Intel Gigabit Ethernet Card at 116-122 MB/S with full network traffic and writing this traffic to the 3Ware arrays. All this hardware is running on a Serverworks HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII processors.

After running for about 3 hours, the system will hard hang and die. Using debugging tools, I have isolated to the hang to the 3Ware adapters. If I remove all but a single 3Ware adapter, the system will run reliably for days at these data rates. The moment I add more than one 3Ware 7810 adapter, the system will lock up. Recent testing reveals that the hang is in the 3Ware card itself (all the LEDs go on at once and stay on). Attempts by the system to reset the adapter fail until the system is power cycled.

3Ware dfriver version is .16 from the 2.4.18 tree. Firmware is the 48 bit LBA version.

Please advise.

Jeff

#1 lore raw
Quoted 1 line · Jeff V. Merkey

More info. I put in some trace code to determine how many io buffer heads were being fed to each adapter. When the number of bh's reaches numbers above 4244+- buffer heads outstanding at one time, then I see the cards lockup.

Jeff

Quoted 29 lines

Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am seeing about 216-224 MB/S total throughput on writes to local arrays on 2.4.18.

The system is also running an Intel Gigabit Ethernet Card at 116-122 MB/S with full network traffic and writing this traffic to the 3Ware arrays. All this hardware is running on a Serverworks HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII processors.

After running for about 3 hours, the system will hard hang and die. Using debugging tools, I have isolated to the hang to the 3Ware adapters. If I remove all but a single 3Ware adapter, the system will run reliably for days at these data rates. The moment I add more than one 3Ware 7810 adapter, the system will lock up. Recent testing reveals that the hang is in the 3Ware card itself (all the LEDs go on at once and stay on). Attempts by the system to reset the adapter fail until the system is power cycled.

3Ware dfriver version is .16 from the 2.4.18 tree. Firmware is the 48 bit LBA version.

Please advise.

Jeff

#2 lore raw
In reply to Jeff V. Merkey
Quoted 5 lines · Jeff V. Merkey

Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am seeing about 216-224 MB/S total throughput on writes to local arrays on 2.4.18.

Have you done any kind of variations on this configuration to see when or where the maximum throughput happens? Daniel and I were speculating about where the 3ware limits are. Specs say 100MB/s per adapter (for both 6000 and 7000 series), you would probably hit max bandwidth with 2 adapters. The drives themselves are not a limiting factor, unless you are down to striping across only 2 drives instead of all 8. I take it you are using the hardware RAID instead of software MD RAID?

Quoted 5 lines

The system is also running an Intel Gigabit Ethernet Card at 116-122 MB/S with full network traffic and writing this traffic to the 3Ware arrays. All this hardware is running on a Serverworks HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII processors.

Does this board have multiple PCI busses? Is the GigE card on a different bus than the 3ware cards?

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#3 lore raw
In reply to Jeff V. Merkey
Quoted 7 lines · Andreas Dilger
Quoted 5 lines · Jeff V. Merkey

Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am seeing about 216-224 MB/S total throughput on writes to local arrays on 2.4.18.

Quoted 7 lines

Have you done any kind of variations on this configuration to see when or where the maximum throughput happens? Daniel and I were speculating about where the 3ware limits are. Specs say 100MB/s per adapter (for both 6000 and 7000 series), you would probably hit max bandwidth with 2 adapters. The drives themselves are not a limiting factor, unless you are down to striping across only 2 drives instead of all 8. I take it you are using the hardware RAID instead of software MD RAID?

Single adapter can push 130 MB/S with 8 drives striped at RAID 0 (hardware RAID not software). With 2 adapters with 8 drives, the max rate of 216 MB/S is achieved. Adding additional adapters on the same PCI bus hits the wall at 238+- MB/S. I have not tried putting the two other 3Ware's on the 66 Mhz bus, but in theory, this would result in 350 MB/S total throughput from what I am seeing with the Gigabit ethernet adapter.

Quoted 9 lines
Quoted 6 lines

The system is also running an Intel Gigabit Ethernet Card at 116-122 MB/S with full network traffic and writing this traffic to the 3Ware arrays. All this hardware is running on a Serverworks HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII processors.

Does this board have multiple PCI busses? Is the GigE card on a different bus than the 3ware cards?

The Gigabit adapter is running on the 66 Mhz bus. When combined with SCI I am able to push 238 MB/S through the SCI to the 3Ware adapters and out to disk. At present, I have modified the SCI drivers and I am DMA'ing directly into Linus buffer cache from the SCI disk data.

Very very fast. The 3Ware adapters are barfing when the BH count gets too high. I have no idea why.

Jeff

Quoted 6 lines

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/