DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset
The error :
Curious - OSB4 thinks the DMA is still running.
OSB4 wait exit
appears to occur during the "rewrite" phase of Bonnie++ every time we run it. The setup is a Tyan 2515 with a Seagate ST310211A drive. Bonnie ran for ~48 hours on 2.4.10 without slowdown (I assume this is because the extra check in -ac isn't present) but there was some file system corruption after the 1st 24 hours:
EXT2-fs error (device ide0(3,6)): ext2_free_blocks: Freeing blocks not in datazone - block = 2138996092, count = 1
Here's the lspci:
00:00.0 Host bridge: ServerWorks CNB20LE (rev 06)
Flags: bus master, medium devsel, latency 32
00:00.1 Host bridge: ServerWorks CNB20LE (rev 06)
Flags: bus master, medium devsel, latency 16
[snip]
00:0f.0 ISA bridge: ServerWorks OSB4 (rev 50)
Subsystem: ServerWorks OSB4
Flags: bus master, medium devsel, latency 0
00:0f.1 IDE interface: ServerWorks: Unknown device 0211 (prog-if 8a [Master SecP PriP])
Flags: bus master, medium devsel, latency 64 I/O ports at ffa0 [size=16]
Signature · 6 lines
Marvin Justice Software Developer BOXX Technologies, Inc. www.boxxtech.com 512-235-6318 (V) 412-835-0434 (F)
Quoted 2 lines
Curious - OSB4 thinks the DMA is still running.
OSB4 wait exit
This is an OSB4 trap for a problem I've seen with seagate drivers on some boxes. Under very high load with UDMA seagate drives the OSB4 returns to us with the chip reporting DMA in progress. The next I/O after this will cause disk corruption as it DMA's the last 4 bytes of the previous write, then the data shifted up by 4 bytes until the end of that I/O. I need to confirm nobody else is seeing this with other working workloads, then I'll swap it for a panic - better to die than kill the disc contents.
Serverworks have been looking at the problem but have yet to duplicate it. As far as we can tell (and Red Hat have been working with a customer on this directly) your choices are
1. Use multiword DMA not UDMA
2. Use non seagate disks with that controller
I am hopeful that serverworks will figure out what is up, but not every box sees it - and indeed they've yet to be able to reproduce it.
Alan
Alright. Until the situation clarifies you might tell RH turn off OSB4 by default in future releases. It caused us no end of grief with 2.4.3-12.
Signature · 6 lines
Marvin Justice Software Developer BOXX Technologies, Inc. www.boxxtech.com 512-235-6318 (V) 512-835-0434 (F)
Quoted 23 lines · Alan Cox
Quoted 2 lines
Curious - OSB4 thinks the DMA is still running.
OSB4 wait exitThis is an OSB4 trap for a problem I've seen with seagate drivers on some boxes. Under very high load with UDMA seagate drives the OSB4 returns to us with the chip reporting DMA in progress. The next I/O after this will cause disk corruption as it DMA's the last 4 bytes of the previous write, then the data shifted up by 4 bytes until the end of that I/O. I need to confirm nobody else is seeing this with other working workloads, then I'll swap it for a panic - better to die than kill the disc contents.
Serverworks have been looking at the problem but have yet to duplicate it. As far as we can tell (and Red Hat have been working with a customer on this directly) your choices are
1. Use multiword DMA not UDMA
2. Use non seagate disks with that controllerI am hopeful that serverworks will figure out what is up, but not every box sees it - and indeed they've yet to be able to reproduce it.
Alan
Quoted 2 lines
Alright. Until the situation clarifies you might tell RH turn off OSB4 by default in future releases. It caused us no end of grief with 2.4.3-12.
Blacklisting seagate drives on the OSB4 is already on the todo list - its taken a long time to figure out what was going on - it was only because someone had > 100 boxes and knew they all did this that we got anywhere
Quoted 42 lines · Marvin Justice
The error :
Curious - OSB4 thinks the DMA is still running.
OSB4 wait exitappears to occur during the "rewrite" phase of Bonnie++ every time we run it. The setup is a Tyan 2515 with a Seagate ST310211A drive. Bonnie ran for ~48 hours on 2.4.10 without slowdown (I assume this is because the extra check in -ac isn't present) but there was some file system corruption after the 1st 24 hours:
EXT2-fs error (device
ide0(3,6)):ext2_free_blocks: Freeing blocks not in datazone - block = 2138996092, count = 1Here's the lspci:
00:00.0 Host bridge: ServerWorks CNB20LE (rev 06)
Flags: bus master, medium devsel, latency 3200:00.1 Host bridge: ServerWorks CNB20LE (rev 06)
Flags: bus master, medium devsel, latency 16[snip]
00:0f.0 ISA bridge: ServerWorks OSB4 (rev 50)
Subsystem: ServerWorks OSB4
Flags: bus master, medium devsel, latency 000:0f.1 IDE interface: ServerWorks: Unknown device 0211 (prog-if 8a [Master SecP PriP])
Flags: bus master, medium devsel, latency 64 I/O ports at ffa0 [size=16]--
Marvin Justice
Software Developer
BOXX Technologies, Inc.
www.boxxtech.com
512-235-6318 (V)
412-835-0434 (F)
As mentioned earlier on this list, we saw similar corruption, detectable by the fact that if you do a ls -R on the file system there will be input/output errors due to inodes that are no longer there, eventually.
We began with a system with a Seagate system drive (master, ide0), same
model as in Marvin's post.. IDE cd rom
as slave on the ide0 bus, and two IBM data drives on the ide1 bus.
The vendor eventually swapped out all the Seagate drives for Western Digital ones, which does not make the problem go away, but makes it much less frequent and harder to reproduce, but at least there is something in /var/log/messages to show for it whenever it does happen. It is wrong to think this problem happens only with Seagate drives.
Steve
------------------------------------------------------------------
Steven C. Timm (630) 840-8525 [email protected] http://home.fnal.gov/~timm/
Fermilab Computing Division/Operating Systems Support Scientific Computing Support Group--Computing Farms Operations
Quoted 10 lines · Alan Cox
1. Use multiword DMA not UDMA
2. Use non seagate disks with that controllerI am hopeful that serverworks will figure out what is up, but not every box sees it - and indeed they've yet to be able to reproduce it.
Alan
Quoted 3 lines
We began with a system with a Seagate system drive (master, ide0), same
model as in Marvin's post.. IDE cd rom
as slave on the ide0 bus, and two IBM data drives on the ide1 bus.
...
Quoted 2 lines
whenever it does happen. It is wrong to think this problem happens only with Seagate drives.
indeed, it seems that the most common trigger for this problem is simply putting two devices from different vendors on the same channel. Seagate drives are fairly notorious for not getting along with others.
it's best to think of IDE as a point-to-point link.
regards, mark hahn.
------------------------------------------------------------------
Steven C. Timm (630) 840-8525 [email protected] http://home.fnal.gov/~timm/
Fermilab Computing Division/Operating Systems Support Scientific Computing Support Group--Computing Farms Operations
Quoted 4 lines · Mark Hahn
Quoted 3 lines
We began with a system with a Seagate system drive (master, ide0), same
model as in Marvin's post.. IDE cd rom
as slave on the ide0 bus, and two IBM data drives on the ide1 bus....
Quoted 8 lines
Quoted 2 lines
whenever it does happen. It is wrong to think this problem happens only with Seagate drives.
indeed, it seems that the most common trigger for this problem is simply putting two devices from different vendors on the same channel. Seagate drives are fairly notorious for not getting along with others.
it's best to think of IDE as a point-to-point link.
True... however, we were still able to reproduce these problems with only one drive on the bus.
Steve
Quoted 4 lines
regards, mark hahn.