I/O MCA recovery
Background: in an effort to allow option ROM emulation on ia64 (via the X int10+x86 emulator), I've had to look at doing I/O error recovery since many option ROMs expect to do legacy I/O port reads and writes to ports that may or may not respond (one particular ROM that I've looked at continuously polls a register in legacy I/O space until it returns a value). On sn2, when a device doesn't respond to an I/O (legacy space or otherwise), a PCI master abort is generated, which generally causes an MCA.
Recovering from such an event requires reprogramming chipset and bridge registers (some to just clear error state and others to re-arm error detection) and as such is very platform specific. Another issue is that the MCA event may arrive after the processor has switched to a task completely unrelated to the I/O. The approach I've taken thus far is to register the I/O address range that a process mmaps in /proc/bus/pci (in pci_mmap_page_range), along with its associated PID. When an MCA occurs, an I/O error recovery routine checks the target identifier value against the linked list of I/O ranges and recovers appropriately (the PID is there so that we can send a SIGBUS or somesuch in the future). This allows us to avoid calling PAL_MC_DRAIN on every interrupt to try and flush out errors (which I'm guessing would be very expensive), but may have other problems.
Ultimately, this involves adding a machine vector for I/O error recovery and a linked list of I/O regions and their PIDs. The I/O error handler could optionally be extended to look for any PCI resource range and call a per-device error handling callback or shutdown routine.
Thoughts? Does this approach sound reasonable?
Thanks,
Jesse
Quoted 5 lines · Jesse Barnes
Another issue is that the
MCA event may arrive after the processor has switched to a task completely unrelated to the I/O. The approach I've taken thus far is to register the I/O address range that a process mmaps in/proc/bus/pci(inpci_mmap_page_range), along with its associated PID.
Why not use the existing resource map?
The PCI bus data structures are hierarchial and resources are well defined in that. Seems like the linked list of I/O ranges could (1) get very long and (2) just replicates what's already there.
Quoted 6 lines
Ultimately, this involves adding a machine vector for I/O error recovery and a linked list of I/O regions and their PIDs. The I/O error handler could optionally be extended to look for any PCI resource range and call a per-device error handling callback or shutdown routine.
Thoughts? Does this approach sound reasonable?
Seems to fit nicely with previous pci_read_check() support proposed earlier.
grant
Quoted 11 lines · Grant Grundler
Quoted 5 lines · Jesse Barnes
Another issue is that the
MCA event may arrive after the processor has switched to a task completely unrelated to the I/O. The approach I've taken thus far is to register the I/O address range that a process mmaps in/proc/bus/pci(inpci_mmap_page_range), along with its associated PID.Why not use the existing resource map?
The PCI bus data structures are hierarchial and resources are well defined in that. Seems like the linked list of I/O ranges could (1) get very long and (2) just replicates what's already there.
You mean just use request_region at pci_mmap_page_range time instead? That would prevent multiple processes from accessing the same region, but we'd still want to know the PID of the process that had that range reserved (the gfx guys really want to get a signal when an I/O error occurs so they can recover in userspace; other userspace drivers probably want the same). Another problem is that legacy I/O space isn't listed in any of the PCI resource maps (at least as far as I know), so there would be no way to track that region, which is the one that I'm *really* interested in. :)
Quoted 9 lines
Quoted 6 lines
Ultimately, this involves adding a machine vector for I/O error recovery and a linked list of I/O regions and their PIDs. The I/O error handler could optionally be extended to look for any PCI resource range and call a per-device error handling callback or shutdown routine.
Thoughts? Does this approach sound reasonable?
Seems to fit nicely with previous
pci_read_check()support proposed earlier.
Yeah, it could be tied in with that interface.
Jesse
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Jesse> Background: in an effort to allow option ROM emulation on Jesse> ia64 (via the X int10+x86 emulator), I've had to look at Jesse> doing I/O error recovery since many option ROMs expect to do Jesse> legacy I/O port reads and writes to ports that may or may not Jesse> respond (one particular ROM that I've looked at continuously Jesse> polls a register in legacy I/O space until it returns a Jesse> value). On sn2, when a device doesn't respond to an I/O Jesse> (legacy space or otherwise), a PCI master abort is generated, Jesse> which generally causes an MCA.
Jesse> Recovering from such an event requires reprogramming chipset Jesse> and bridge registers (some to just clear error state and Jesse> others to re-arm error detection) and as such is very Jesse> platform specific. Another issue is that the MCA event may Jesse> arrive after the processor has switched to a task completely Jesse> unrelated to the I/O. The approach I've taken thus far is to Jesse> register the I/O address range that a process mmaps in Jesse> /proc/bus/pci (in pci_mmap_page_range), along with its Jesse> associated PID. When an MCA occurs, an I/O error recovery Jesse> routine checks the target identifier value against the linked Jesse> list of I/O ranges and recovers appropriately (the PID is Jesse> there so that we can send a SIGBUS or somesuch in the Jesse> future). This allows us to avoid calling PAL_MC_DRAIN on Jesse> every interrupt to try and flush out errors (which I'm Jesse> guessing would be very expensive), but may have other Jesse> problems.
Jesse> Ultimately, this involves adding a machine vector for I/O Jesse> error recovery and a linked list of I/O regions and their Jesse> PIDs. The I/O error handler could optionally be extended to Jesse> look for any PCI resource range and call a per-device error Jesse> handling callback or shutdown routine.
Jesse> Thoughts? Does this approach sound reasonable?
Eh, I/O space is required to soft-fail, isn't it?
Why can't you hide this in the platform-specific inX/outX routines? I suppose it would be very slow to drain MCAs after every inX/outX, but you'd have to do the slow part only once, until you know whether or not the given I/O address is safe.
--david
Quoted 7 lines · Jesse Barnes
Quoted 4 lines · Grant Grundler
Why not use the existing resource map?
The PCI bus data structures are hierarchial and resources are well defined in that. Seems like the linked list of I/O ranges could (1) get very long and (2) just replicates what's already there.You mean just use
request_regionatpci_mmap_page_rangetime instead?
No - directly walk the either the PCI bus/device tree or walk the ioport space resource tree and lookup the owner.
Quoted 5 lines
That
would prevent multiple processes from accessing the same region, but we'd still want to know the PID of the process that had that range reserved (the gfx guys really want to get a signal when an I/O error occurs so they can recover in userspace; other userspace drivers probably want the same).
hrm...ic. And the process doesn't need to "mmap" the region/file for IO Port space like it would for MMIO accesses. :^(
Isn't working with the vendor to NOT do this sort of crap also an option? At least a few vendors offer EFI drivers for video cards... (ie avoid the problem of x86 BIOS in the first place)
Linux really needs a driver/card that can provide HW acceleration without first having BIOS initializing it. parisc-linux port (and probably a few others) could use such a driver/card too.
Quoted 3 lines
Another problem is that legacy I/O space isn't listed in any of the PCI resource maps (at least as far as I know), so there would be no way to track that region, which is the one that I'm *really* interested in. :)
Well, if code is randomly poking around without registering with request_region, then your proposal is as good as any.
Quoted 1 line
Yeah, it could be tied in with that interface.
cool.
thanks,
grant
Quoted 2 lines · Grant Grundler
hrm...ic. And the process doesn't need to "mmap" the region/file for IO Port space like it would for MMIO accesses. :^(
Right, that's even worse. I'm not sure how to deal with that (sn2 doesn't even support port I/O in the ia64 architected sense).
Quoted 3 lines
Isn't working with the vendor to NOT do this sort of crap also an option? At least a few vendors offer EFI drivers for video cards... (ie avoid the problem of x86 BIOS in the first place)
Yeah, that's the Way Of The Future (tm), but there's a lot of legacy stuff out there...
Quoted 3 lines
Linux really needs a driver/card that can provide HW acceleration without first having BIOS initializing it. parisc-linux port (and probably a few others) could use such a driver/card too.
The parisc port should be able to use the int10 emulator too, as long as you can recover from any errors that might be generated...
Quoted 6 lines
Quoted 3 lines
Another problem is that legacy I/O space isn't listed in any of the PCI resource maps (at least as far as I know), so there would be no way to track that region, which is the one that I'm *really* interested in. :)
Well, if code is randomly poking around without registering with
request_region, then your proposal is as good as any.
Ok, I'll keep hacking on it then and post a patch when I have something presentable.
Thanks,
Jesse
Quoted 1 line · David Mosberger
Eh, I/O space is required to soft-fail, isn't it?
I thought so too, but I haven't been able to find the spec that contains that requirement.
Quoted 4 lines
Why can't you hide this in the platform-specific inX/outX routines? I suppose it would be very slow to drain MCAs after every inX/outX, but you'd have to do the slow part only once, until you know whether or not the given I/O address is safe.
This is I/O initiated by userspace loads/stores, so unless I wrap every in/out with some sort of ioctl or something, those won't help me. Also, with this scheme, we could potentially recover from regular read/writes too.
Jesse
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Quoted 1 line · David Mosberger
Quoted 1 line
Eh, I/O space is required to soft-fail, isn't it?
Jesse> I thought so too, but I haven't been able to find the spec Jesse> that contains that requirement.
It's certainly implied.
Quoted 4 lines
Quoted 4 lines
Why can't you hide this in the platform-specific inX/outX routines? I suppose it would be very slow to drain MCAs after every inX/outX, but you'd have to do the slow part only once, until you know whether or not the given I/O address is safe.
Jesse> This is I/O initiated by userspace loads/stores, so unless I Jesse> wrap every in/out with some sort of ioctl or something, those Jesse> won't help me.
User-level accesses are mapped via the MMU so you could always intercept the page-faults.
Jesse> Also, with this scheme, we could potentially recover from Jesse> regular read/writes too.
_If_ there is an infrastructure what you can hook into, fine. But I'm highly suspicious of using broken platforms as a justification for new infrastructure.
--david
Quoted 6 lines · David Mosberger
Jesse> This is I/O initiated by userspace loads/stores, so unless I Jesse> wrap every in/out with some sort of ioctl or something, those Jesse> won't help me.
User-level accesses are mapped via the MMU so you could always intercept the page-faults.
Wouldn't that mean that on every I/O access we'd have to page fault, do the I/O, and then invalidate the mapping? That seems like a lot of overhead.
Quoted 6 lines
Jesse> Also, with this scheme, we could potentially recover from Jesse> regular read/writes too.
_If_ there is an infrastructure what you can hook into, fine. But I'm highly suspicious of using broken platforms as a justification for new infrastructure.
Are you describing ia64 as a broken platform here? The problem I'm trying to solve isn't sn2 specific (though part of the X stuff I have to do will be driven by sn2 requirements), it's a generic way to deal with hard fails on PIO reads, which afaik, affects all ia64 platforms. Correct me if I'm wrong here...
Thanks,
Jesse
Quoted 3 lines · Grant Grundler
Linux really needs a driver/card that can provide HW acceleration without first having BIOS initializing it. parisc-linux port (and probably a few others) could use such a driver/card too.
Modern hardware is complex. The (closed) McApple firmware has firmware to init various video cards and stufff AFAIK. Not sure if this is any better than BIOS emulation though.
--cw
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
User-level accesses are mapped via the MMU so you could always
intercept the page-faults.
Jesse> Wouldn't that mean that on every I/O access we'd have to page Jesse> fault, do the I/O, and then invalidate the mapping? That Jesse> seems like a lot of overhead.
Yes. I doubt it would be an issue for inX/outX emulation in the int10 module.
Jesse> Also, with this scheme, we could potentially recover from Jesse> regular read/writes too.
Quoted 3 lines
Quoted 3 lines
_If_ there is an infrastructure what you can hook into, fine.
But I'm highly suspicious of using broken platforms as a justification for new infrastructure.
Jesse> Are you describing ia64 as a broken platform here?
Hardly.
Jesse> The problem I'm trying to solve isn't sn2 specific (though Jesse> part of the X stuff I have to do will be driven by sn2 Jesse> requirements)
I was talking about hard-failure of inX/outX. If SN2 does that, it's broken and I'm not terribly sympathetic (but see below).
Jesse> it's a generic way to deal with hard fails on PIO reads, Jesse> which afaik, affects all ia64 platforms. Correct me if I'm Jesse> wrong here...
Let me try to say it differently: inX/outX must soft-fail. How you achieve that on SN2, I don't really care. If, for other reasons, there happens to be an infrastructure you can hook into to facility implementation of soft-fail inX/outX on SN2, that's certainly fine by me. But don't try to use inX/outX soft-fail as a reason to justify the infrastructure. Better?
--david
Quoted 2 lines · David Mosberger
Yes. I doubt it would be an issue for inX/outX emulation in the int10 module.
True.
Quoted 2 lines
I was talking about hard-failure of inX/outX. If SN2 does that, it's broken and I'm not terribly sympathetic (but see below).
But my point was: doesn't in/out hard fail on other ia64 platforms too? If so, then it makes sense to deal with it generically.
Quoted 10 lines
Jesse> it's a generic way to deal with hard fails on PIO reads, Jesse> which afaik, affects all ia64 platforms. Correct me if I'm Jesse> wrong here...
Let me try to say it differently: inX/outX must soft-fail. How you achieve that on SN2, I don't really care. If, for other reasons, there happens to be an infrastructure you can hook into to facility implementation of soft-fail inX/outX on SN2, that's certainly fine by me. But don't try to use inX/outX soft-fail as a reason to justify the infrastructure. Better?
Sure, that makes sense. The other part of the implementation was to deal with regular MMIO accesses though--userspace drivers want to get signalled when an error occurs, would you propose the page fault mechanism to detect that as well, or is an MCA handler a better way to go?
Thanks,
Jesse
Quoted 5 lines · Jesse Barnes
Are you describing ia64 as a broken platform here? The problem I'm trying to solve isn't sn2 specific (though part of the X stuff I have to do will be driven by sn2 requirements), it's a generic way to deal with hard fails on PIO reads, which afaik, affects all ia64 platforms. Correct me if I'm wrong here...
hardfail vs softfail is a chipset, not arch issue. Intel IA32/IA64 chipsets will softfail on MMIO reads and IO port access that master abort (timeout).
HP chipsets (ZX1/SX1000) will hardfail - it's one of the differences I point out in the "Porting drivers to ZX1" OLS2002 paper I wrote. Sounds like SGI chipsets behave the same way.
ISTR only config space accesses are always required to softfail on master aborts. I don't recall if IO Port space is required to as well and it looks like this issue is outside the scope of PCI spec.
grant
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Jesse> But my point was: doesn't in/out hard fail on other ia64 Jesse> platforms too?
No. AFAIK, inX/outX is always supposed to soft-fail. On zx1-based machines, firmware initializes the chipset such that memory-mapped I/O will hard-fail. Intel chipsets will always softfail, even for memory-mapped I/O. Anybody knows what IBM's chipset does in this regard?
Jesse> Sure, that makes sense. The other part of the implementation Jesse> was to deal with regular MMIO accesses though--userspace Jesse> drivers want to get signalled when an error occurs, would you Jesse> propose the page fault mechanism to detect that as well, or Jesse> is an MCA handler a better way to go?
I don't have a strong opinion on this particular issue.
--david
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Grant Grundler
Grant> HP chipsets (ZX1/SX1000) will hardfail - it's one of the Grant> differences I point out in the "Porting drivers to ZX1" Grant> OLS2002 paper I wrote. Sounds like SGI chipsets behave the Grant> same way.
Hmmh, I thought zx1 soft-fails on inX/outX?
--david
Quoted 10 lines · Grant Grundler
Quoted 5 lines · Jesse Barnes
Are you describing ia64 as a broken platform here? The problem I'm trying to solve isn't sn2 specific (though part of the X stuff I have to do will be driven by sn2 requirements), it's a generic way to deal with hard fails on PIO reads, which afaik, affects all ia64 platforms. Correct me if I'm wrong here...
hardfail vs softfail is a chipset, not arch issue. Intel IA32/IA64 chipsets will softfail on MMIO reads and IO port access that master abort (timeout).
That's nice (that's why I wanted a pointer to a spec--I can hand it to our hw guys who have been asking for it).
Quoted 3 lines
HP chipsets (ZX1/SX1000) will hardfail - it's one of the differences I point out in the "Porting drivers to ZX1" OLS2002 paper I wrote. Sounds like SGI chipsets behave the same way.
Yep.
Quoted 3 lines
ISTR only config space accesses are always required to softfail on master aborts. I don't recall if IO Port space is required to as well and it looks like this issue is outside the scope of PCI spec.
Alex says that zx1 can soft fail on IO port accesses if configured to do so. We don't have that option.
And of course, the more general case of MMIO accesses by user level drivers remains...
Thanks,
Jesse
Quoted 8 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Grant Grundler
Grant> HP chipsets (ZX1/SX1000) will hardfail - it's one of the Grant> differences I point out in the "Porting drivers to ZX1" Grant> OLS2002 paper I wrote. Sounds like SGI chipsets behave the Grant> same way.
Hmmh, I thought zx1 soft-fails on inX/outX?
I'll have to try it...none of the drivers I care use IO port space or randomly attempt to poke around.
ISTR the graphics folks agonizing over this issue when bringing up the radeon cards as console - the BIOS emulation in firmware was has issues with this same problem. Alex, remember more details?
grant
Quoted 7 lines · Jesse Barnes
Quoted 3 lines
ISTR only config space accesses are always required to softfail on master aborts. I don't recall if IO Port space is required to as well and it looks like this issue is outside the scope of PCI spec.
Alex says that zx1 can soft fail on IO port accesses if configured to do so. We don't have that option.
And the default configuration for zx1 is I/O port space in soft fail,
MMIO space in hardfail. The current zx1 "smarts" in X can reconfigure the bridge w/ VGA to softfail MMIO for probing/int10 support. It's ugly, but it works.
Alex
Signature · 1 line
Alex Williamson HP Linux & Open Source Lab
Quoted 2 lines · Jesse Barnes
That's nice (that's why I wanted a pointer to a spec--I can hand it to our hw guys who have been asking for it).
Well, ask Intel. Nicely. :^)
Quoted 1 line
Alex says that zx1 can soft fail on IO port accesses if configured to do so.
Ok. Then IO Port and MMIO accesses will by default hardfail on ZX1. I believe there is only one bit that can be twiddled regarding this.
grant
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Jesse> That's nice (that's why I wanted a pointer to a spec--I can Jesse> hand it to our hw guys who have been asking for it).
I'm not sure you'll find an explicit statement to that effect. In my opinion, it's implied because the x86 IN/OUT instructions do not (and should not) cause MCAs.
--david
Quoted 2 lines · Grant Grundler
Ok. Then IO Port and MMIO accesses will by default hardfail on ZX1. I believe there is only one bit that can be twiddled regarding this.
I'm wrong.
later chip revs added two more bits to "adjust" I/O port failure mode. (thanks alex)
grant
Quoted 2 lines · Jesse Barnes
Ok, I'll keep hacking on it then and post a patch when I have something presentable.
Well, maybe this isn't quite presentable, but it seems to work for my small test cases. The basic idea is that when a user land process mmaps something in /proc/bus/pci, it'll get added to a list which is checked by the MCA handler when a bus check occurs. If there's a match, the process is SIGBUS'd.
There are still lots of holes I think, especially since I'm not sure about all of the MCA specific bits yet. In particular, is looking at the current MCA record ok? Should I be indexing into ->proc_err.info any further than I do? There are also locking problems, since at the moment an MCA could occur on multiple processors, but I think the MCA code in general doesn't handle that case...
Anyway, I'd appreciate any comments. Maybe something like this is ok for the mainline? I haven't dealt with read_check here, but I think the pseudocode would be something like this:
read_check(addr) {
val = *addr;
PAL_MC_DRAIN;
if (dev->error_flag)
return error;
return val;
}
Obviously, error_flag would be set in the MCA code as well based on walking the PCI resource structures looking for a match with the target identifier that caused the MCA...
Jesse
===== arch/ia64/kernel/mca.c 1.60 vs edited =====
arch/ia64/kernel/mca.c +39 -1
@@ -797,13 +797,51 @@void ia64_mca_ucmc_handler (void ) { +struct io_range *range; +unsigned long io_addr =0 ;pal_processor_state_info_t *psp = (pal_processor_state_info_t *) &ia64_sal_to_os_handoff_state.proc_state_param; -int recover = psp->tc && !(psp->cc || psp->bc || psp->rc || psp->uc); +int recover =0 ; +ia64_err_rec_t *curr_record;/* Get the MCA error record and log it */ ia64_mca_log_sal_error_record (SAL_INFO_TYPE_MCA ); +/* TLB errors are fixed up before we get here, so recover */ +if (psp->tc) { + recover =1 ; + goto return_to_sal; + } + +/* +* If it's not a bus check with a valid target identifier, +* we don't have a chance. +*/ +if (!psp->bc) {// || !curr_record->proc_err.info->valid.target_identifier) { + recover =0 ; + goto return_to_sal; + } + + curr_record =IA64_LOG_CURR_BUFFER (SAL_INFO_TYPE_MCA ); + io_addr = curr_record->proc_err.info->target_identifier; + +/* +* See if an I/O error occured in a previously registered range +*/ +list_for_each_entry (range, &pci_io_ranges, range_list) { +if (range->start <= io_addr && io_addr <= range->end) { +struct siginfo siginfo; + recover =1 ; + siginfo.si_signo =SIGBUS ; + siginfo.si_code =BUS_ADRERR ; + siginfo.si_addr = (void *) io_addr; +force_sig_info (SIGBUS , &siginfo, +find_task_by_pid (range->owner)); +break ; + } + } + +return_to_sal:/* * Wakeup all the processors which are spinning in the rendezvous * loop.
===== arch/ia64/pci/pci.c 1.48 vs edited =====
arch/ia64/pci/pci.c +29 -0
@@ -20,6 +20,7 @@#include <linux/slab.h> #include <linux/smp_lock.h> #include <linux/spinlock.h> +#include <linux/slab.h> #include <asm/machvec.h> #include <asm/page.h>
@@ -48,6 +49,9 @@struct pci_fixup pcibios_fixups[1 ]; +LIST_HEAD (pci_io_ranges); +spinlock_t io_range_list_lock =SPIN_LOCK_UNLOCKED ; +/* * Low-level SAL-based PCI configuration access functions. Note that SAL * calls are already serialized (via sal_lock), so we don't need another
@@ -437,6 +441,8 @@ pci_mmap_page_range (struct pci_dev *dev, struct vm_area_struct *vma, enum pci_mmap_state mmap_state, int write_combine) { + struct io_range *new_range; +/* * I/O space cannot be accessed via normal processor loads and stores on this * platform.
@@ -465,6 +471,29 @@ vma->vm_end - vma->vm_start, vma->vm_page_prot))return -EAGAIN ; + new_range =kmalloc (sizeof (struct io_range ),GFP_KERNEL ); +if (!new_range) { +printk (KERN_WARNING "%s: cannot allocate io_range, " +"I/O errors for 0x%016lx-0x%016lx will be fatal" , + __FUNCTION__, vma->vm_start, vma->vm_end); + goto out; + } + +/* +* Track this range and its associated process for use by the +* MCA handler. +*/ + new_range->start =__pa (vma->vm_pgoff <<PAGE_SHIFT ); + new_range->end = new_range->start + (vma->vm_end - vma->vm_start); + new_range->owner = current->pid; + +spin_lock (&io_range_list_lock); +list_add (&new_range->range_list, &pci_io_ranges); +spin_unlock (&io_range_list_lock); + +printk ("I/O range 0x%016lx-0x%016lx registered\n" , + new_range->start, new_range->end); + out:return 0 ; }
===== drivers/pci/proc.c 1.38 vs edited =====
drivers/pci/proc.c +14 -0
@@ -279,8 +279,22 @@static int proc_bus_pci_release (struct inode *inode,struct file *file) { +struct io_range *range; +kfree (file->private_data); file->private_data =NULL ; + +spin_lock (&io_range_list_lock); +list_for_each_entry (range, &pci_io_ranges, range_list) { +if (range->owner == current->pid) { +list_del (&range->range_list); +printk ("I/O range 0x%016lx-0x%016lx de-registered\n" , + range->start, range->end); +kfree (range); +break ; + } + } +spin_unlock (&io_range_list_lock);return 0 ; }
===== include/asm-ia64/io.h 1.19 vs edited =====
include/asm-ia64/io.h +16 -0
@@ -1,6 +1,8 @@#ifndef _ASM_IA64_IO_H#define _ASM_IA64_IO_H +#include <linux/list.h> +/* * This file contains the definitions for the emulated IO instructions * inb/inw/inl/outb/outw/outl and the "string versions" of the same
@@ -50,12 +52,26 @@extern struct io_space io_space[];extern unsigned int num_io_spaces; +/* +* Simple I/O range object with owner (if there is one) +*/ +struct io_range { +unsigned long start, end; +struct list_head range_list; +pid_t owner; +}; + +extern struct list_head pci_io_ranges; +# ifdef __KERNEL__ +#include <linux/spinlock.h> #include <asm/intrinsics.h> #include <asm/machvec.h> #include <asm/page.h> #include <asm/system.h> + +extern spinlock_t io_range_list_lock;/* * Change virtual addresses to physical addresses and vv.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Jesse> Anyway, I'd appreciate any comments. Maybe something like Jesse> this is ok for the mainline? I haven't dealt with read_check Jesse> here, but I think the pseudocode would be something like Jesse> this:
So if multiple processes map the same I/O range then _all_ of them will be terminated? What if stuff gets remapped etc? This all just seems incredibly fragile to me.
--david
Quoted 2 lines · David Mosberger
So if multiple processes map the same I/O range then _all_ of them will be terminated?
As it stands, only the first PID that has a matching range will get a SIGBUS, but that could be changed.
Quoted 1 line
What if stuff gets remapped etc?
Then the machine takes an unrecoverable MCA, just like it does now.
Quoted 1 line
This all just seems incredibly fragile to me.
I think it's useful for at least a subset of the I/O MCAs that people see, and with the addition of the read_check stuff, may be more generally useful. Obviously though, this doesn't recover from 100% of I/O errors, something that's probably not practical anyway...
Jesse
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
This all just seems incredibly fragile to me.
Jesse> I think it's useful for at least a subset of the I/O MCAs Jesse> that people see, and with the addition of the read_check Jesse> stuff, may be more generally useful. Obviously though, this Jesse> doesn't recover from 100% of I/O errors, something that's Jesse> probably not practical anyway...
It definitely falls in the experimental category. That isn't bad but I don't think it's mainline material. Certainly not for 2.6.
--david
Quoted 2 lines · David Mosberger
It definitely falls in the experimental category. That isn't bad but I don't think it's mainline material. Certainly not for 2.6.
What would you like to see before it hits the mainline? The read_check support implemented? For better or worse, that requires a lot of buy in on lkml. Just looking for other suggestions...
Thanks,
Jesse
Quoted 3 lines · Jesse Barnes
Quoted 1 line
What if stuff gets remapped etc?
Then the machine takes an unrecoverable MCA, just like it does now.
Oh, actually I'm wrong here. Since I'm looking at the physical address in the target identifier, a virtual remap will have no effect, so it'll still get caught. Of course, that only addresses one of your concerns...
Jesse
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Quoted 2 lines · David Mosberger
Quoted 2 lines
It definitely falls in the experimental category. That isn't bad but I don't think it's mainline material. Certainly not for 2.6.
Jesse> What would you like to see before it hits the mainline? The Jesse> read_check support implemented? For better or worse, that Jesse> requires a lot of buy in on lkml. Just looking for other Jesse> suggestions...
I just don't see a lot of demand for this kind of feature at this point, so why bloat the kernel? Plus the solution so far seems rather incomplete and fragile. I suggest to play with this in a separate tree and if users start to really like the patch and can't live without it, well, I'm sure I'll hear about it.
--david
Quoted 5 lines · David Mosberger
I just don't see a lot of demand for this kind of feature at this point, so why bloat the kernel? Plus the solution so far seems rather incomplete and fragile. I suggest to play with this in a separate tree and if users start to really like the patch and can't live without it, well, I'm sure I'll hear about it.
Sure, ok. I'll see how useful it is in real world situations (e.g. our gfx folks have been wanting this for awhile) and come back when I have compelling evidence of its utility.
Thanks,
Jesse
Quoted 3 lines
There are also locking problems, since at the moment an MCA could occur on multiple processors, but I think the MCA code in general doesn't handle that case...
At the moment the MCA code serializes simulaneous MCA on multiple processors (see the hand-crafted spinlock in mca_asm.S at the ia64_os_mca_spin label).
-Tony
Quoted 7 lines · Tony
Quoted 3 lines
There are also locking problems, since at the moment an MCA could occur on multiple processors, but I think the MCA code in general doesn't handle that case...
At the moment the MCA code serializes simulaneous MCA on multiple processors (see the hand-crafted spinlock in
mca_asm.Sat theia64_os_mca_spinlabel).
Thanks Tony, I hadn't looked at that code in awhile. I guess the I/O error recovery code should try to acquire the io_range_list_lock before looking through the list. If it can't get the lock, we just have to give up and make the error unrecoverable since we don't know if another CPU will take an MCA while holding that lock, leaving the list in a bad state...
I don't *think* that doing unconditional rendezvous in the PROM will help this situation either, but maybe someone else has good ideas about how to handle that?
Thanks,
Jesse
Quoted 3 lines
Thanks Tony, I hadn't looked at that code in awhile. I guess the I/O error recovery code should try to acquire the
io_range_list_lockbefore looking through the list.
Yes ... I wouldn't attempt anything tricker than a "trylock" in code that's going to run as part of an MCA handler. The async delivery of MCA means that there's always a risk of deadlocking a cpu if it tries to grab a lock that it already holds at the time the MCA is delivered.
-Tony
Quoted 9 lines · Jesse Barnes
Quoted 7 lines · Tony
Quoted 3 lines
There are also locking problems, since at the moment an MCA could occur on multiple processors, but I think the MCA code in general doesn't handle that case...
At the moment the MCA code serializes simulaneous MCA on multiple processors (see the hand-crafted spinlock in
mca_asm.Sat theia64_os_mca_spinlabel).
Quoted 5 lines
Thanks Tony, I hadn't looked at that code in awhile. I guess the I/O error recovery code should try to acquire the
io_range_list_lockbefore looking through the list. If it can't get the lock, we just have to give up and make the error unrecoverable since we don't know if another CPU will take an MCA while holding that lock, leaving the list in a bad state...
Seems like spinning on a trylock for a short period would be reasonable. It everything is OK, the process with the lock will let go quickly. Otherwise, we're probably dead anyway.
Quoted 3 lines
I don't *think* that doing unconditional rendezvous in the PROM will help this situation either, but maybe someone else has good ideas about how to handle that?
In general, I suggest avoiding rendezvous unless there is a really obvious reason to do so. In this case, I think you're right.
Signature · 2 lines
Russ Anderson, OS RAS/Partitioning Project Lead SGI - Silicon Graphics Inc [email protected]
Quoted 4 lines · Russ Anderson
Seems like spinning on a trylock for a short period would be reasonable. It everything is OK, the process with the lock will let go quickly. Otherwise, we're probably dead anyway.
Here's the latest (untested!) version that tries to do that.
Quoted 6 lines
Quoted 3 lines
I don't *think* that doing unconditional rendezvous in the PROM will help this situation either, but maybe someone else has good ideas about how to handle that?
In general, I suggest avoiding rendezvous unless there is a really obvious reason to do so. In this case, I think you're right.
I'm coming to the same conclusion, even though throwing a whole new kernel execution context into the mix (MCA context) really makes this confusing...
Jesse
===== arch/ia64/kernel/mca.c 1.60 vs edited =====
arch/ia64/kernel/mca.c +58 -1
@@ -797,13 +797,70 @@void ia64_mca_ucmc_handler (void ) { +struct io_range *range; +unsigned long io_addr =0 ;pal_processor_state_info_t *psp = (pal_processor_state_info_t *) &ia64_sal_to_os_handoff_state.proc_state_param; -int recover = psp->tc && !(psp->cc || psp->bc || psp->rc || psp->uc); +int recover =0 ; +ia64_err_rec_t *curr_record;/* Get the MCA error record and log it */ ia64_mca_log_sal_error_record (SAL_INFO_TYPE_MCA ); +/* TLB errors are fixed up before we get here, so recover */ +if (psp->tc) { + recover =1 ; + goto return_to_sal; + } + +/* +* If it's not a bus check with a valid target identifier, +* we don't have a chance. +*/ +if (!psp->bc) { + recover =0 ; + goto return_to_sal; + } + +/* +* If we can't get this lock, we can't safely look at the list, +* so give up. +*/ +if (!spin_trylock (&io_range_list_lock)) { + recover =0 ; + goto return_to_sal; + } + + curr_record =IA64_LOG_CURR_BUFFER (SAL_INFO_TYPE_MCA ); + io_addr = curr_record->proc_err.info->target_identifier; + +/* +* See if an I/O error occured in a previously registered range +*/ +list_for_each_entry (range, &pci_io_ranges, range_list) { +if (range->start <= io_addr && io_addr <= range->end) { +struct siginfo siginfo; +struct task_struct *owner =NULL ; + recover =1 ; + siginfo.si_signo =SIGBUS ; + siginfo.si_code =BUS_ADRERR ; + siginfo.si_addr = (void *) io_addr; + owner =find_task_by_pid (range->owner); +if (owner) +force_sig_info (SIGBUS , &siginfo, owner); +else { +/* +* need to free memory too, is that safe +* here? +*/ +list_del (&range->range_list); + } +break ; + } + } +spin_unlock (&io_range_list_lock); + +return_to_sal:/* * Wakeup all the processors which are spinning in the rendezvous * loop.
===== arch/ia64/pci/pci.c 1.48 vs edited =====
arch/ia64/pci/pci.c +29 -0
@@ -20,6 +20,7 @@#include <linux/slab.h> #include <linux/smp_lock.h> #include <linux/spinlock.h> +#include <linux/slab.h> #include <asm/machvec.h> #include <asm/page.h>
@@ -48,6 +49,9 @@struct pci_fixup pcibios_fixups[1 ]; +LIST_HEAD (pci_io_ranges); +spinlock_t io_range_list_lock =SPIN_LOCK_UNLOCKED ; +/* * Low-level SAL-based PCI configuration access functions. Note that SAL * calls are already serialized (via sal_lock), so we don't need another
@@ -437,6 +441,8 @@ pci_mmap_page_range (struct pci_dev *dev, struct vm_area_struct *vma, enum pci_mmap_state mmap_state, int write_combine) { + struct io_range *new_range; +/* * I/O space cannot be accessed via normal processor loads and stores on this * platform.
@@ -465,6 +471,29 @@ vma->vm_end - vma->vm_start, vma->vm_page_prot))return -EAGAIN ; + new_range =kmalloc (sizeof (struct io_range ),GFP_KERNEL ); +if (!new_range) { +printk (KERN_WARNING "%s: cannot allocate io_range, " +"I/O errors for 0x%016lx-0x%016lx will be fatal" , + __FUNCTION__, vma->vm_start, vma->vm_end); + goto out; + } + +/* +* Track this range and its associated process for use by the +* MCA handler. +*/ + new_range->start =__pa (vma->vm_pgoff <<PAGE_SHIFT ); + new_range->end = new_range->start + (vma->vm_end - vma->vm_start); + new_range->owner = current->pid; + +spin_lock (&io_range_list_lock); +list_add (&new_range->range_list, &pci_io_ranges); +spin_unlock (&io_range_list_lock); + +printk ("I/O range 0x%016lx-0x%016lx registered\n" , + new_range->start, new_range->end); + out:return 0 ; }
===== drivers/pci/proc.c 1.38 vs edited =====
drivers/pci/proc.c +14 -0
@@ -279,8 +279,22 @@static int proc_bus_pci_release (struct inode *inode,struct file *file) { +struct io_range *range; +kfree (file->private_data); file->private_data =NULL ; + +spin_lock (&io_range_list_lock); +list_for_each_entry (range, &pci_io_ranges, range_list) { +if (range->owner == current->pid) { +list_del (&range->range_list); +printk ("I/O range 0x%016lx-0x%016lx de-registered\n" , + range->start, range->end); +kfree (range); +break ; + } + } +spin_unlock (&io_range_list_lock);return 0 ; }
===== include/asm-ia64/io.h 1.19 vs edited =====
include/asm-ia64/io.h +16 -0
@@ -1,6 +1,8 @@#ifndef _ASM_IA64_IO_H#define _ASM_IA64_IO_H +#include <linux/list.h> +/* * This file contains the definitions for the emulated IO instructions * inb/inw/inl/outb/outw/outl and the "string versions" of the same
@@ -50,12 +52,26 @@extern struct io_space io_space[];extern unsigned int num_io_spaces; +/* +* Simple I/O range object with owner (if there is one) +*/ +struct io_range { +unsigned long start, end; +struct list_head range_list; +pid_t owner; +}; + +extern struct list_head pci_io_ranges; +# ifdef __KERNEL__ +#include <linux/spinlock.h> #include <asm/intrinsics.h> #include <asm/machvec.h> #include <asm/page.h> #include <asm/system.h> + +extern spinlock_t io_range_list_lock;/* * Change virtual addresses to physical addresses and vv.