readX_check() performance evaluation

Hi all,

We are planning recovery from the PCI bus intermittent errors. PCI-X standard describes it as "5.4 Error Handling and Fault Tolerance" in PCI-X 1.0b. There were several discussions in lkml, how to recover from PCI errors.

Seto posted "[RFC] How drivers notice a HW error?" (readX_check() I/F)
http://marc.theaimsgroup.com/?l=linux-kernel&m6992207709400&w=2

Grant will show his idea near future,
http://marc.theaimsgroup.com/?l=linux-kernel&m7453681120603&w=2

I made a readX_check() prototype which Seto proposed to measure performance disadvantage of this kind of I/F. And I made a performance evaluation of Disk I/O with this prototype.
Comments are welcome.

Conclusion:
Performance disadvantage of readX_check() is a very small. I'd like you to understand that such a function will not cause severe performance disadvantage as you imagine.

This patch:
- is for Fusion MPT driver.
- has no error recovery code yet, sorry.
- currently supports ia64 only. But I believe that some other CPU(such as SPARC, PPC, PA-RISC) can also support this kind of I/F.
I know, unfortunately, that i386 can't support this kind of I/F, because it can't recover from machine check state.

How to use this patch:
- Apply to vanilla 2.6.1.
- Rename drivers/message/fusion/mptbase.c to mptbase_main.c (Because we make mptbase.ko from mptbase.c and read_check.S, so source file name has to be renamed. Though I know read_check.S should go under the architecture directory, because this patch is only for performance evaluation, forgive me. )

Evaluation Environment:
Kernel:
vanilla 2.6.1 and 2.6.1+readX_check patch
Platform: Intel Tiger-4 (1-CPU)
processor : 0
vendor : GenuineIntel
arch : IA-64
family : Itanium 2
model : 1
revision : 5
archrev : 0
features : branchlong
cpu number : 0
cpu regs : 4
cpu MHz : 1296.473997
itc MHz : 1296.473997
BogoMIPS : 1941.96
SCSI HBA/driver:
onboard LSI Logic 53C1030(FwRev
030600h, MaxQ%5) Fusion MPT driver(kernel 2.6.1)
Disks
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: FUJITSU Model: MAP3367NC Rev: 5207
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: FUJITSU Model: MAP3367NC Rev: 5207
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: FUJITSU Model: MAP3367NC Rev: 5207
Type: Direct-Access ANSI SCSI revision: 03

Test tool:
rawread 1.0.3
http://www-124.ibm.com/developerworks/opensource/linuxperf/rawread/rawread.html

Results:
To avoid buffer cache, we measured performance by O_DIRECT.

1-1) ./rawread -p 1 -d 1 -s 512 -n 131072 -x -z

     (1 disk, 512-bytes/1-read,)
                                        avg. sys(%) 
                    MB/s        IOPS    of vmstat
---------------------------------------------------
patched 2.6.1       5.245       10741   16.2
vanilla 2.6.1       5.269       10790   15.7

---------------------------------------------------
patched/vanilla 0.995 1.032

1-2) ./rawread -p 2 -d 1 -s 512 -n 131072 -x -z

     (2 disks, 512-bytes/1-read,)
                                        avg. sys(%) 
                    MB/s        IOPS    of vmstat
---------------------------------------------------
patched 2.6.1       10.473      21448   30.6
vanilla 2.6.1       10.548      21602   30.6

---------------------------------------------------
patched/vanilla 0.993 1.000

1-3) ./rawread -p 3 -d 1 -s 512 -n 131072 -x -z

     (3 disks, 512-bytes/1-read,)
                                        avg. sys(%) 
                    MB/s        IOPS    of vmstat
---------------------------------------------------
patched 2.6.1      11.267       23074   29.9
vanilla 2.6.1      11.251       23042   30.5

---------------------------------------------------
patched/vanilla 1.001 0.980

2-1) ./rawread -p 1 -d 1 -s 4096 -n 131072 -x -z

     (1 disk, 4096-bytes/1-read,)
                                        avg. sys(%) 
                    MB/s        IOPS    of vmstat
---------------------------------------------------
patched 2.6.1      39.422       10092   14.1
vanilla 2.6.1      39.389       10083   14.0

---------------------------------------------------
patched/vanilla 1.001 1.007

2-2) ./rawread -p 2 -d 1 -s 4096 -n 131072 -x -z

     (2 disks, 4096-bytes/1-read,)
                                        avg. sys(%) 
                    MB/s        IOPS    of vmstat
---------------------------------------------------
patched 2.6.1      70.438       18032   24.1
vanilla 2.6.1      70.390       18019   24.1

---------------------------------------------------
patched/vanilla 1.001 1.000

2-3) ./rawread -p 3 -d 1 -s 4096 -n 131072 -x -z

     (3 disks, 4096-bytes/1-read,)
                                        avg. sys(%) 
                    MB/s        IOPS    of vmstat
---------------------------------------------------
patched 2.6.1      70.588       18070   24.5
vanilla 2.6.1      70.861       18140   24.4

---------------------------------------------------
patched/vanilla 0.996 1.004

3-1) ./rawread -p 1 -d 1 -s 32768 -n 131072 -x -z

     (1 disk, 32768-bytes/1-read,)
                                        avg. sys(%) 
                    MB/s        IOPS    of vmstat
---------------------------------------------------
patched 2.6.1      69.226       2215     3.5
vanilla 2.6.1      68.546       2190     3.4

---------------------------------------------------
patched/vanilla 1.010 1.029

3-2) ./rawread -p 2 -d 1 -s 32768 -n 131072 -x -z

     (2 disk, 32768-bytes/1-read,)
                                        avg. sys(%) 
                    MB/s        IOPS    of vmstat
---------------------------------------------------
patched 2.6.1     139.315       4458     7.0
vanilla 2.6.1     139.188       4454     6.6

---------------------------------------------------
patched/vanilla 1.010 1.029

3-3) ./rawread -p 3 -d 1 -s 32768 -n 131072 -x -z

     (3 disks, 32768-bytes/1-read,)
                                        avg. sys(%) 
                    MB/s        IOPS    of vmstat
---------------------------------------------------
patched 2.6.1     208.883       6684    10.0
vanilla 2.6.1     209.193       6694    10.2

---------------------------------------------------
patched/vanilla 0.999 0.980

-------
Thanks,
Hironobu Ishii

I don't work on error recovery full time and that's really a full time job. In a nutshell, I'd like to treat IO errors as exceptions and hide most of the support in the regular readX() macros. Arch support controls readX/writeX implementations and CONFIG_* options can be used to pick which behavior someone wants. I'd expect drivers which support error recovery to register a error recovery callback and "fake" value to hand back for PIO reads until recovery is complete.

I could be wrong. Exception handling is ugly. But my hope is that by putting all the exception handling in one place in the driver, the driver will be forced to be methodical in being "deterministic" WRT to driver state and can return to a known state by calling one routine. This will keep the drivers maintainable by "part-time hackers" who don't care about error recovery.

This is no surprise. The cost of PIO reads is far greater (100x) than the extra cost to check for errors.
Eg PIO read on 1GHz HP rx2600 is ~1000-1100 CPU cycles and it's in the same order of magnitude for all architectures.

Error recovyer code is the hard part. Find all the locations in the code and writing instance specific error recovery code. The HPUX driver I first worked on is amazingly similar to MPT. And it had error recovery support (for "Host Powerfail") and truly was a PITA to support.

yes - probably a few others as well.

I think i386 could. The method to check for errors will be different and the types of errors which are detectable are fewer.
I'm not sure it would be recoverable though. But it should be able to shutdown a misbehaving driver instance/device before the box crashed. (well, assuming there is no memory corruption).

thanks,
grant

One big problem is how to get rid of the spinlocks after the exception though (hardware access usually happens inside a spinlock)

I presume you could return a magic value (all ones), but then you still have to make sure the driver doesn't break when that happens. That would likely require testing for that value on every read access and make the code similarly ugly and difficult to write as with Linus' explicit checking model.

But there may be no other choice, see below...

Yes, there are often magic bits in northbridges and chipsets. Problem is that they're sometimes buggy (because not well tested) and give random errors.

Also enabling them tends to trigger a *lot* of bugs in random drivers.

They usually give an MCE, but it is not exact for writes (happens sometime later) and may not even be for reads.

The only sane way to handle them would be a global call back per pci_dev, but then you run into problems with the locking again.

Also in my experience from AMD64 which originally was a bit aggressive on enabling MCEs: enabling MCEs increases your kernel support load a lot.

Many people have slightly buggy systems which still happen to work mostly. If you report every problem you as kernel maintainer will be flooded with reports about things you can nothing to do about. So I don't think it would make sense to enable it by default.

One idea I played with was to only enable it for driver debugging, but it is hard to educate driver developers about it (most just don't know about it and we have no way to pass information to them). In the end I removed it because it was too much hazzle. In short this stuff probably only makes sense when you're a system vendor who sells support contracts for whole systems including hardware support. For the normal linux model where software is independent from hardware (and hardware is usually crappy) it just doesn't work very well.

-Andi

Andi> Also in my experience from AMD64 which originally was a bit Andi> aggressive on enabling MCEs: enabling MCEs increases your Andi> kernel support load a lot.

Andi> Many people have slightly buggy systems which still happen to Andi> work mostly. If you report every problem you as kernel Andi> maintainer will be flooded with reports about things you can Andi> nothing to do about.

I find this comment interesting. Can you elaborate what you mean by "slightly buggy systems"?

	--david

e.g. one bit ECC errors in memory are quite common. And with ECC memory they are not really fatal. Similar with drivers. A lot of drivers do bus aborts and other things regularly, but there is not necessarily data corruption.

-Andi

Andi> e.g. one bit ECC errors in memory are quite common. And with Andi> ECC memory they are not really fatal.

Yet they are a good indicator that something is wrong (not performing properly) or may be failing soon. I don't think putting on blinders for such problems is a good idea. Though I agree that the question of how to report such things without needlessly alerting Joe Clueless is an interesting challenge.

	--david

Most server class hardware should log it somewhere and allow to read the event log in the firmware. This even works for unhandleable errors unlike what the OS could do.

But when printed in Linux they will report it to the linux maintainer or their distribution vendor. "My Linux is buggy and giving these weird messages" And they are both in no position at all to do something about it.

I toyed with the idea of printking a disclaimer of
"This is likely not a software bug. Report it to your hardware vendor." But I doubt this would help much. Even when you say clearly in the message that the hardware failed the user sees a weird message and thinks it is Linux's fault.

You could enable it with CONFIG_I_HAVE_A_HARDWARE_SUPPORT_CONTRACT_OR_I_WRITE_DRIVERS Or just make it a kernel command line option with off by default.

-Andi

Andi> Most server class hardware should log it somewhere and allow Andi> to read the event log in the firmware. This even works for Andi> unhandleable errors unlike what the OS could do.

And you'd want to reboot your server just so you can check on the soft failure rate? ;-)

	--david

Yep, I reboot my machines all the time ;-)

Seriously you can count it somewhere and present it in sysfs or /proc. Or log it somewhere else and supply a special utility to show them that makes it clear that the events are hardware and not software related. I suppose if your server vendor is serious they will supply a tool to read the firmware log from a running system.

But printks enabled by default are a bad idea (and a bug too BTW - printk called from MCE handlers can randomly deadlock)

-Andi

Andi> Seriously you can count it somewhere and present it in sysfs Andi> or /proc. Or log it somewhere else and supply a special Andi> utility to show them that makes it clear that the events are Andi> hardware and not software related. I suppose if your server Andi> vendor is serious they will supply a tool to read the firmware Andi> log from a running system.

Andi> But printks enabled by default are a bad idea (and a bug too Andi> BTW - printk called from MCE handlers can randomly deadlock)

No argument here. I didn't get/see the earlier part of this discussion so I didn't realize you were complaining about printks only. Never mind.

	--david

Hi!

What about new KERN_HWBUG loglevel?
Pavel

When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]

We have done a rather large study with DIMMs that had SBEs and have found no evidence that a SBE turns into a UCE, i.e. the fact that a SBE is reported, is no indication that the device might fail soon.

As a matter of fact the soft error rates increases while parts use smaller process technologies and lower supply voltages. Cosmic rays are one source for soft errors. Another source are alpha particles emitted by the solder.

Still I think it's important to log SBEs, but you probably will need a treshhold in case you hit a hard SBE. Also scrubbing the memory location (and re-read the location to check if the error was transient or not) might be a good idea if the memory controller supports this. If it is a true, hard SBE it should be reported. It also might be a good idea to mark the page, so it does not get re-allocated.

Thanks

Matthias Fouquet-Lapar  Core Platform Software    [email protected]  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127

Matthias> We have done a rather large study with DIMMs that had SBEs Matthias> and have found no evidence that a SBE turns into a UCE, Matthias> i.e. the fact that a SBE is reported, is no indication Matthias> that the device might fail soon.

Matthias> As a matter of fact the soft error rates increases while Matthias> parts use smaller process technologies and lower supply Matthias> voltages. Cosmic rays are one source for soft Matthias> errors. Another source are alpha particles emitted by the Matthias> solder.

Ehh, wait a second: you're saying that your study proved that if the device isn't failing, it isn't failing. ;-) Of course you'll get noise and perhaps even lots of it due to cosmic rays but this doesn't say anything about the error pattern you when a device _is_ failing (e.g., due to overheating, over-clocking, or wrong voltage). Or did your study cover the cases where a system is operated under "out-of-spec" situation?

Matthias> Still I think it's important to log SBEs, but you probably Matthias> will need a treshhold in case you hit a hard SBE. Also Matthias> scrubbing the memory location (and re-read the location to Matthias> check if the error was transient or not) might be a good Matthias> idea if the memory controller supports this. If it is a Matthias> true, hard SBE it should be reported. It also might be a Matthias> good idea to mark the page, so it does not get Matthias> re-allocated.

Yes. And once I finally received Andi's earlier mails (guess I have to thank MyDoom for that... ;-( ), it was clear that nobody argued for turning off the error reporting. The issue was only whether or not to log a message via printk() (which, in this case, clearly isn't a good idea). So I think we're all in violent agreement.

	--david

I should have been more precice. We used field returned parts which had reported SBEs and had been exchanged in the field. Our goal was to see if any of these parts "de-generate" over time. Most of these parts had hard single bit failures in one or more locations. As I said, we didn't find evidence that even hard SBEs turn into a multiple bit error. Of course the chances of getting a UCE are higher when a "soft" SBE occurs in a memory location which already has a hard SBE.

Thanks

Matthias Fouquet-Lapar  Core Platform Software    [email protected]  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127

Matthias> We have done a rather large study with DIMMs that had SBEs Matthias> I should have been more precice. We used field returned Matthias> parts which had reported SBEs and had been exchanged in Matthias> the field. Our goal was to see if any of these parts Matthias> "de-generate" over time. Most of these parts had hard Matthias> single bit failures in one or more locations.

Ah, that's more interesting, agreed.

Matthias> As I said, we didn't find evidence that even hard SBEs Matthias> turn into a multiple bit error.

But you were changing the operating environment of the chip, so I wouldn't draw too strong of a conclusion. Or was the reason for the hard SBEs known and it was determined that the operating environment was not a factor in triggering them?

	--david

That is a very good point and one of my favourite subjects. I think a lot of error checking has to be done in-flight, i.e. at the time of the error check if the error is transient or can be reproduced, if possible log environmental information (temp and VDD) with the error etc. And then have a small EEPROM on standard DIMMs and save this error information, so we don't rely on paper tags. Or maybe include the DIMM serial number in the error message. But I'm getting carried away :)

As for the test environment, a fair amount of DIMMs was put through environmental stress ("shake & bake") as well as extended voltage margins. (I remain impressed with these chambers where you can dial down from +60C to
-40C within a few minutes while the system is vibrating with a couple of G's)
We actually exceeded the DIMM manufacturers specifications for limits, again no sign of increased failure rate for DIMMs with SBEs.

Some failure modes are very complex and data pattern sensitive.

As someone pointed out quite correctly, there is a fine line how much information should be logged and potentialy ring a bell for the customer to place a service call to replace a part which potentially will never fail again (again there is a difference between a hard and a soft error).

One option might be to have a separate error log, so the console is not overflowed with messages and then use some tool to diagnose the errors and potentially warn the user, i.e. turn on the "check engine" light. We should keep the average user in mind

Thanks

Matthias Fouquet-Lapar  Core Platform Software    [email protected]  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127

...

yes - any proposal is going to require reviewing all PIO reads and how the read return value is consumed (or discarded).

yeah. My hope was it would be less invasive. But more changes are probably needed than I expected.

...

While ia64/parisc platforms have HW support for this, I totally agree it won't work well for most (x86) platforms. I'd like to reduce the burden on the driver writers for common drivers (eg MPT) used on "vanilla" x86.

And like I pointed out before, linux kernel needs to review panic() calls to see if some of them could easily be eliminated. The general robustness issues (eg pci_map_single() panics on failure) aren't prerequisites for IO error checking, but the latter seems less useful with out the former.

I'd like to defend the pci_map_single() interface. It was designed to reduce the complexity at the cost of robustness.
I think it was a fair trade off at the time and it sounds like the time has come for a different trade off.

thanks,
grant

It would probably a good idea to implement it for i386 on chipsets that support it reliably and try to educate driver writers to enable it when they are testing drivers. This would likely improve the quality of linux drivers long term and make your job as maintainer of an "anal IO error" platform easier.

Just it should not be enabled by default in production kernels. And finding out where it works reliably will be some work.

There is no reason pci_map_single() has to panic on overflow. On x86-64 it returns an unmapped address that is guaranteed to cause an bus abort for 128KB. And you have an macro to test for it (pci_dma_error()). I believe ppc64 has adopted it too. Of course most drivers don't use it yet.

Still panic on overflow is useful for testing and it is kept as an kernel command line option.

-Andi

yup. The key drivers we deal with reached that point last year. Drivers could always be better. But those issues have been discussed and presented:
o LinuxTag 2002 keynote by Alan Cox, "Submitting new Kernel drivers"

	(http://gsyprf11.external.hp.com/porting_zx1/mgp/Code.mgp)

o OLS 2002 talk by Arjan van de Ven, "How not to write kernel drivers"
o OLS 2002 talk by Greg K-H, "Documentation/CodingStyle and Beyond"
o OLS 2002 talk by myself, "Porting Drivers to HP ZX1"

It helps to "enforce" driver quality through "anal IO Error containment" but it's too late when it happens on a customer box.

agreed.

parisc and ia64 will also bus abort. And then HPMC/MCA respectively. We could reserve a "safe page" and then hand that back I guess. But that sounds like a very broken error containment strategy to me. (ie outbound data will be garbage).

This really isn't an issue for HP ZX1/IA64 since most drivers (64-bit) can bypass the IOMMU and directly address memory. parisc-linux still isn't commercially interesting.

I didn't know about pci_dma_error.
Google found two references: One is:

	http://www.x86-64.org/lists/discuss/msg03841.html

<search 2.6.2-rc2 source tree>
grundler <502>find -name '*.[chS]' | xargs fgrep pci_dma_error ./include/asm-x86_64/pci.h:#define pci_dma_error(x) ((x) == bad_dma_address)
grundler <503>

That explains why most drivers don't use it yet.
It's only supported on one arch.
Maybe propose this to linux-pci mailing list?

grant

It was discussed on linux-arch and ppc64 at least agreed on it. The other architectures can get it via a comptibility #define that is always 0.

There was a patch for that somewhere, but apparently it was never merged or not merged yet.

Anton, what was the state of that?

-Andi