Problems with 2.6.15-mm1 and mm2.
OK, so on -mm1 we get:
Unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip: c01fccd2 *pde = 0042d001 *pte = 00000000 Oops: 0000 [#1] SMP last sysfs file: Modules linked in: CPU: 0 EIP: 0060:[<c01fccd2>] Not tainted VLI EFLAGS: 00010286 (2.6.15-mm1-autokern1)
EIP is at pci_call_probe+0x1a/0xa1
eax: 00000000 ebx: ffffffed ecx: e748a030 edx: c03a4680
esi: e7767800 edi: c03a4680 ebp: ffffffff esp: e749fef0
ds: 007b es: 007b ss: 0068
Process swapper (pid: 1, threadinfo=e749e000 task=e748a030)
Stack: <0>ffffffed e7767800 c03a4680 c03a46ac c01fcd8c c03a4680 e7767800 c03a441c
<0>c03a4680 e7767800 c03a46ac c01fcdbf c03a4680 e7767800
e7767800 00000000
<0>e7767848 c021c265 e7767848 e7767848 00000000 c021c311
c021c34a c03a46ac
Call Trace:
[<c01fcd8c>] __pci_device_probe+0x33/0x47
[<c01fcdbf>] pci_device_probe+0x1f/0x34
[<c021c265>] driver_probe_device+0x3a/0x84
[<c021c311>] __driver_attach+0x0/0x60
[<c021c34a>] __driver_attach+0x39/0x60
[<c021ba18>] bus_for_each_dev+0x47/0x6d
[<c01f4d5a>] kobject_add+0x76/0x95
[<c021c385>] driver_attach+0x14/0x18
[<c021c311>] __driver_attach+0x0/0x60
[<c021be30>] bus_add_driver+0x54/0x87
[<c021c72c>] driver_register+0x3b/0x3e
[<c01fcfa6>] __pci_register_driver+0x7e/0x8c
[<c03f646d>] qla1280_init+0xc/0xf
[<c03e4789>] do_initcalls+0x4b/0x99
[<c0100393>] init+0x98/0x195
[<c01002fb>] init+0x0/0x195
[<c0100ea9>] kernel_thread_helper+0x5/0xbfrom the NUMA-Q. http://test.kernel.org/19793/debug/console.log
------------------------------------------------------------------
on -mm2 I get the x86_64 seems to lock up (NFI why ... looking at it), the NUMA-Q and x440 panic (very similar to the above).
I think Andy figured out what was causing those panics. Can we drop those patches until they're fixed?
Thanks,
M.
Quoted 45 lines · Martin Bligh
OK, so on -mm1 we get:
Unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip: c01fccd2 *pde = 0042d001 *pte = 00000000 Oops: 0000 [#1] SMP last sysfs file: Modules linked in: CPU: 0 EIP: 0060:[<c01fccd2>] Not tainted VLI EFLAGS: 00010286 (2.6.15-mm1-autokern1)EIP is at
pci_call_probe+0x1a/0xa1
eax: 00000000 ebx: ffffffed ecx:e748a030edx:c03a4680
esi:e7767800edi:c03a4680ebp: ffffffff esp:e749fef0
ds: 007b es: 007b ss: 0068
Process swapper (pid: 1, threadinfo=e749e000 task=e748a030)Stack: <0>ffffffed e7767800 c03a4680 c03a46ac c01fcd8c c03a4680 e7767800 c03a441c <0>c03a4680 e7767800 c03a46ac c01fcdbf c03a4680 e7767800 e7767800 00000000 <0>e7767848 c021c265 e7767848 e7767848 00000000 c021c311 c021c34a c03a46ac Call Trace: [<c01fcd8c>] __pci_device_probe+0x33/0x47 [<c01fcdbf>] pci_device_probe+0x1f/0x34 [<c021c265>] driver_probe_device+0x3a/0x84 [<c021c311>] __driver_attach+0x0/0x60 [<c021c34a>] __driver_attach+0x39/0x60 [<c021ba18>] bus_for_each_dev+0x47/0x6d [<c01f4d5a>] kobject_add+0x76/0x95 [<c021c385>] driver_attach+0x14/0x18 [<c021c311>] __driver_attach+0x0/0x60 [<c021be30>] bus_add_driver+0x54/0x87 [<c021c72c>] driver_register+0x3b/0x3e [<c01fcfa6>] __pci_register_driver+0x7e/0x8c [<c03f646d>] qla1280_init+0xc/0xf [<c03e4789>] do_initcalls+0x4b/0x99 [<c0100393>] init+0x98/0x195 [<c01002fb>] init+0x0/0x195 [<c0100ea9>] kernel_thread_helper+0x5/0xb
Quoted 2 lines
from the NUMA-Q. http://test.kernel.org/19793/debug/console.log
Yes, I asked Greg about that - we don't know what's causing it yet. I have a bad feeling that this bug will go into Linus's tree if we don't fix it quick.
I had problems with gregkh-pci-x86-pci-domain-support-the-meat.patch. It might be worth reverting that.
Quoted 7 lines
on -mm2 I get the
x86_64seems to lock up (NFI why ... looking at it), the NUMA-Q and x440 panic (very similar to the above).I think Andy figured out what was causing those panics. Can we drop those patches until they're fixed?
I'm not aware of any buggy x86_64 patches in -mm2 :(
I guess you don't have the time to sit down and do a bisection search. It'd take a solid few hours...
Quoted 11 lines
Quoted 3 lines
from the NUMA-Q. http://test.kernel.org/19793/debug/console.log
Yes, I asked Greg about that - we don't know what's causing it yet. I have a bad feeling that this bug will go into Linus's tree if we don't fix it quick.
I had problems with gregkh-pci-x86-pci-domain-support-the-meat.patch. It might be worth reverting that.
Andy figured out what caused it.
Quoted 9 lines
Quoted 4 lines
on -mm2 I get the
x86_64seems to lock up (NFI why ... looking at it), the NUMA-Q and x440 panic (very similar to the above).I think Andy figured out what was causing those panics. Can we drop those patches until they're fixed?
I'm not aware of any buggy
x86_64patches in -mm2 :(I guess you don't have the time to sit down and do a bisection search. It'd take a solid few hours...
It's more that I don't have direct access to the machines in question
any more. The x86_64 one might just be a machine issue ... trying to confirm that still ... but the PCI one was real.
M.
Below is cut & pasted, so not applyable, and maybe it shouldn't be applied. But ... it worked before whatever is in -mm ... so my personal feeling is that if we don't have a fix for whatever is currently in -mm, it should get dropped until we do ? Going backwards = bad.
Perhaps I'm in a time loop, and just confused. but I don't think so ?
----------------------------------------------
From apw:
pci device ensure sysdata initialised
[Ok, here is a patch to ensure sysdata is valid for all busses.]
We have been seeing panic's on NUMA systems in pci_call_probe() in 2.6.15-rc5-mm2 and -mm3. It seems that some changes have occured to the meaning of the 'sysdata' for a device such that it is no longer just an integer containing the node, it is now a structure containing the node and other data. However, it seems that we do not always initialise this sysdata before we probe the device.
Below are three examples from a boot with this checked for. The attached patch ensures that we supply a valid sysdata for system busses. Currently we take no account of the node for this bus for no ACPI configured systems. This is unchanged from the -mm1 code.
Intel(R) PRO/1000 Network Driver - version 6.1.16-k2 Copyright (c) 1999-2005 Intel Corporation. pci_call_probe: starting drv<c03d4be0> dev<dfd16800> id<c03d4734> pci_call_probe: dev->bus<dfce6800> pci_call_probe: dev->bus->sysdata<00000000> pci_call_probe: node<-1> e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection pci_call_probe: starting drv<c03ef220> dev<dfd17400> id<c03eed00> pci_call_probe: dev->bus<dfce6800> pci_call_probe: dev->bus->sysdata<00000000> pci_call_probe: node<-1> Linux Tulip driver version 1.1.13 (December 15, 2004) input: AT Translated Set 2 keyboard as /class/input/input0 tulip0: EEPROM default media type Autosense. tulip0: Index #0 - Media 10baseT (#0) described by a 21140 non-MII (0) block. tulip0: Index #1 - Media 100baseTx (#3) described by a 21140 non-MII (0) block. tulip0: Index #2 - Media 10baseT-FDX (#4) described by a 21140 non-MII (0) block. tulip0: Index #3 - Media 100baseTx-FDX (#5) described by a 21140 non-MII (0) block. eth1: Digital DS21140 Tulip rev 33 at 0001fc00, 00:00:BC:0F:08:96, IRQ 28. pci_call_probe: starting drv<c040a360> dev<dfd14400> id<c040a0fc> pci_call_probe: dev->bus<dfce6600> pci_call_probe: dev->bus->sysdata<dfffafa0> pci_call_probe: node<0> qla1280: QLA1040 found on PCI bus 0, dev 11
--- arch/i386/pci/common.c | 2 ++ arch/i386/pci/fixup.c | 8 +++++--- arch/i386/pci/legacy.c | 3 ++- arch/i386/pci/numa.c | 8 +++++--- arch/i386/pci/visws.c | 4 ++-- include/asm-i386/pci.h | 1 + 6 files changed, 17 insertions(+), 9 deletions(-)
arch/i386/pci/common.c +2 -0
@@ -29,6 +29,8 @@ unsigned long pirq_table_addr;struct pci_bus *pci_root_bus;struct pci_raw_ops *raw_pci_ops; +struct pci_sysdata pci_default_sysdata = { .node =-1 }; + static int pci_read(struct pci_bus *bus, unsigned int devfn, int
where, int size, u32 *value)
{
return raw_pci_ops->read(pci_domain_nr(bus), bus->number,
arch/i386/pci/fixup.c +4 -2
@@ -25,9 +25,11 @@ static void __devinit pci_fixup_i450nx(spci_read_config_byte (d, reg++, &subb);DBG ("i450NX PXB %d: %02x/%02x/%02x\n" , pxb, busno, suba, subb);if (busno) -pci_scan_bus (busno, &pci_root_ops,NULL );/* Bus A */ +pci_scan_bus (busno, &pci_root_ops, + &pci_default_sysdata);/* Bus A */ if (suba < subb) -pci_scan_bus (suba+1 , &pci_root_ops,NULL );/* Bus B */ +pci_scan_bus (suba+1 , &pci_root_ops, + &pci_default_sysdata);/* Bus B */ } pcibios_last_bus =-1 ; }
@@ -42,7 +44,7 @@ static void __devinit pci_fixup_i450gx(su8 busno;pci_read_config_byte (d,0x4a , &busno); printk(KERN_INFO "PCI: i440KX/GX host bridge %s: secondary bus
%02x\n", pci_name(d), busno);
- pci_scan_bus(busno, &pci_root_ops, NULL);
+ pci_scan_bus(busno, &pci_root_ops, &pci_default_sysdata);
pcibios_last_bus = -1;
}
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL,PCI_DEVICE_ID_INTEL_82454GX, pci_fixup_i450gx);
arch/i386/pci/legacy.c +2 -1
@@ -26,7 +26,8 @@ static void __devinit pcibios_fixup_peer l != 0x0000 && l != 0xffff) {DBG ("Found device at %02x:%02x [%04x]\n" , n, devfn, l);printk (KERN_INFO "PCI: Discovered peer bus %02x\n" , n); -pci_scan_bus (n, &pci_root_ops,NULL ); +pci_scan_bus (n, &pci_root_ops, + &pci_default_sysdata);break ; } }
arch/i386/pci/numa.c +5 -3
@@ -97,9 +97,11 @@ static void __devinit pci_fixup_i450nx(spci_read_config_byte (d, reg++, &subb);DBG ("i450NX PXB %d: %02x/%02x/%02x\n" , pxb, busno, suba, subb);if (busno) -pci_scan_bus (QUADLOCAL2BUS (quad,busno), &pci_root_ops,NULL );/* Bus A */ +pci_scan_bus (QUADLOCAL2BUS (quad,busno), &pci_root_ops, + &pci_default_sysdata);/* Bus A */ if (suba < subb) -pci_scan_bus (QUADLOCAL2BUS (quad,suba+1 ), &pci_root_ops,NULL );/* Bus B */ +pci_scan_bus (QUADLOCAL2BUS (quad,suba+1 ), &pci_root_ops, + &pci_default_sysdata);/* Bus B */ } pcibios_last_bus =-1 ; }
@@ -124,7 +126,7 @@ static int __init pci_numa_init(void)printk ("Scanning PCI bus %d for quad %d\n" ,QUADLOCAL2BUS (quad,0 ), quad);pci_scan_bus (QUADLOCAL2BUS (quad,0 ), - &pci_root_ops,NULL ); + &pci_root_ops, &pci_default_sysdata); }return 0 ; }
arch/i386/pci/visws.c +2 -2
@@ -102,8 +102,8 @@ static int __init pcibios_init(void) "bridge B (PIIX4) bus: %u\n", pci_bus1, pci_bus0); raw_pci_ops = &pci_direct_conf1; -pci_scan_bus (pci_bus0, &pci_root_ops,NULL ); -pci_scan_bus (pci_bus1, &pci_root_ops,NULL ); +pci_scan_bus (pci_bus0, &pci_root_ops, &pci_default_sysdata); +pci_scan_bus (pci_bus1, &pci_root_ops, &pci_default_sysdata);pci_fixup_irqs (visws_swizzle, visws_map_irq);pcibios_resource_survey ();return 0 ;
include/asm-i386/pci.h +1 -0
@@ -9,6 +9,7 @@ struct pci_sysdata {int domain;/* PCI domain */ int node;/* NUMA node */ }; +extern struct pci_sysdata pci_default_sysdata;#ifdef CONFIG_PCI_DOMAINS static inline int pci_domain_nr (struct pci_bus *bus)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 31 lines · Martin Bligh
Quoted 11 lines
Quoted 3 lines
from the NUMA-Q. http://test.kernel.org/19793/debug/console.log
Yes, I asked Greg about that - we don't know what's causing it yet. I have a bad feeling that this bug will go into Linus's tree if we don't fix it quick.
I had problems with gregkh-pci-x86-pci-domain-support-the-meat.patch. It might be worth reverting that.
Andy figured out what caused it.
Quoted 8 lines
Quoted 6 lines
on -mm2 I get the
x86_64seems to lock up (NFI why ... looking at it), the NUMA-Q and x440 panic (very similar to the above).I think Andy figured out what was causing those panics. Can we drop those patches until they're fixed?
Quoted 4 lines
I'm not aware of any buggy
x86_64patches in -mm2 :(I guess you don't have the time to sit down and do a bisection search. It'd take a solid few hours...
It's more that I don't have direct access to the machines in question any more. The
x86_64one might just be a machine issue ... trying to confirm that still ... but the PCI one was real.
Quoted 8 lines
M.
Below is cut & pasted, so not applyable, and maybe it shouldn't be applied. But ... it worked before whatever is in -mm ... so my personal feeling is that if we don't have a fix for whatever is currently in -mm, it should get dropped until we do ? Going backwards = bad.
Perhaps I'm in a time loop, and just confused. but I don't think so ?
This isn't at all clear, sorry.
Does the patch you sent fix things in 2.6.15-mm2? On NUMAQ and on x86_64? Does it fix a bug which was introduced in a patch which in in 2.6.15-mm2? If so, which one?
etc ;)
Quoted 8 lines
Quoted 3 lines
Below is cut & pasted, so not applyable, and maybe it shouldn't be applied. But ... it worked before whatever is in -mm ... so my personal feeling is that if we don't have a fix for whatever is currently in -mm, it should get dropped until we do ? Going backwards = bad.
Perhaps I'm in a time loop, and just confused. but I don't think so ?
This isn't at all clear, sorry.
Does the patch you sent fix things in 2.6.15-mm2? On NUMAQ and on
x86_64? Does it fix a bug which was introduced in a patch which in in 2.6.15-mm2? If so, which one?
It fixes the symptom, yes. Greg complained it's papering over the problem and not a real fix, which is fair enough, but ...
yes, it did seem to be a newly introduced problem in -mm1 or -mm2. To my mind, if we don't have a proper fix, we ought to drop the patch that caused the problem in the first place? Andy .. can you finger which patch that was, I forget ...
M.
Quoted 7 lines
It fixes the symptom, yes. Greg complained it's papering over the problem and not a real fix, which is fair enough, but ...
yes, it did seem to be a newly introduced problem in -mm1 or -mm2. To my mind, if we don't have a proper fix, we ought to drop the patch that caused the problem in the first place? Andy .. can you finger which patch that was, I forget ...
The fix I proposed (also attached to this email) fixes a bug introduced in the pci domain patches in greg's tree, specifically this patch:
gregkh-pci-x86-pci-domain-support-struct-pci_sysdata.patch
I believe that my fix is complete in the context of those changes, in that it returns the original semantics. As greg points out this exposes a couple of places where we do have proper domain and node information where it is not supplied. I have a patch or two waiting to be tested which tries to fill in some of the gaps exposed, but the boxes I was testing on have dropped off the network and am waiting for that to be fixed before sending them in. That said I think that if the pci domain patches are accepted then this minimal fix should be applied with them.
-apw
pci device ensure sysdata initialised
[Ok, here is a patch to ensure sysdata is valid for all busses.]
We have been seeing panic's on NUMA systems in pci_call_probe() in 2.6.15-rc5-mm2 and -mm3. It seems that some changes have occured to the meaning of the 'sysdata' for a device such that it is no longer just an integer containing the node, it is now a structure containing the node and other data. However, it seems that we do not always initialise this sysdata before we probe the device.
Below are three examples from a boot with this checked for. The attached patch ensures that we supply a valid sysdata for system busses. Currently we take no account of the node for this bus for no ACPI configured systems. This is unchanged from the -mm1 code.
Intel(R) PRO/1000 Network Driver - version 6.1.16-k2 Copyright (c) 1999-2005 Intel Corporation. pci_call_probe: starting drv<c03d4be0> dev<dfd16800> id<c03d4734> pci_call_probe: dev->bus<dfce6800> pci_call_probe: dev->bus->sysdata<00000000> pci_call_probe: node<-1> e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection pci_call_probe: starting drv<c03ef220> dev<dfd17400> id<c03eed00> pci_call_probe: dev->bus<dfce6800> pci_call_probe: dev->bus->sysdata<00000000> pci_call_probe: node<-1> Linux Tulip driver version 1.1.13 (December 15, 2004) input: AT Translated Set 2 keyboard as /class/input/input0 tulip0: EEPROM default media type Autosense. tulip0: Index #0 - Media 10baseT (#0) described by a 21140 non-MII (0) block. tulip0: Index #1 - Media 100baseTx (#3) described by a 21140 non-MII (0) block. tulip0: Index #2 - Media 10baseT-FDX (#4) described by a 21140 non-MII (0) block. tulip0: Index #3 - Media 100baseTx-FDX (#5) described by a 21140 non-MII (0) block. eth1: Digital DS21140 Tulip rev 33 at 0001fc00, 00:00:BC:0F:08:96, IRQ 28. pci_call_probe: starting drv<c040a360> dev<dfd14400> id<c040a0fc> pci_call_probe: dev->bus<dfce6600> pci_call_probe: dev->bus->sysdata<dfffafa0> pci_call_probe: node<0> qla1280: QLA1040 found on PCI bus 0, dev 11
--- arch/i386/pci/common.c | 2 ++ arch/i386/pci/fixup.c | 8 +++++--- arch/i386/pci/legacy.c | 3 ++- arch/i386/pci/numa.c | 8 +++++--- arch/i386/pci/visws.c | 4 ++-- include/asm-i386/pci.h | 1 + 6 files changed, 17 insertions(+), 9 deletions(-)
arch/i386/pci/common.c +2 -0
@@ -29,6 +29,8 @@ unsigned long pirq_table_addr;struct pci_bus *pci_root_bus;struct pci_raw_ops *raw_pci_ops; +struct pci_sysdata pci_default_sysdata = { .node =-1 }; +static int pci_read (struct pci_bus *bus,unsigned int devfn,int where,int size,u32 *value) { return raw_pci_ops->read(pci_domain_nr(bus), bus->number,
arch/i386/pci/fixup.c +5 -3
@@ -25,9 +25,11 @@ static void __devinit pci_fixup_i450nx(spci_read_config_byte (d, reg++, &subb);DBG ("i450NX PXB %d: %02x/%02x/%02x\n" , pxb, busno, suba, subb);if (busno) -pci_scan_bus (busno, &pci_root_ops,NULL );/* Bus A */ +pci_scan_bus (busno, &pci_root_ops, + &pci_default_sysdata);/* Bus A */ if (suba < subb) -pci_scan_bus (suba+1 , &pci_root_ops,NULL );/* Bus B */ +pci_scan_bus (suba+1 , &pci_root_ops, + &pci_default_sysdata);/* Bus B */ } pcibios_last_bus =-1 ; }
@@ -42,7 +44,7 @@ static void __devinit pci_fixup_i450gx(su8 busno;pci_read_config_byte (d,0x4a , &busno);printk (KERN_INFO "PCI: i440KX/GX host bridge %s: secondary bus %02x\n" ,pci_name (d), busno); -pci_scan_bus (busno, &pci_root_ops,NULL ); +pci_scan_bus (busno, &pci_root_ops, &pci_default_sysdata); pcibios_last_bus =-1 ; }DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_INTEL ,PCI_DEVICE_ID_INTEL_82454GX , pci_fixup_i450gx);
arch/i386/pci/legacy.c +2 -1
@@ -26,7 +26,8 @@ static void __devinit pcibios_fixup_peer l != 0x0000 && l != 0xffff) {DBG ("Found device at %02x:%02x [%04x]\n" , n, devfn, l);printk (KERN_INFO "PCI: Discovered peer bus %02x\n" , n); -pci_scan_bus (n, &pci_root_ops,NULL ); +pci_scan_bus (n, &pci_root_ops, + &pci_default_sysdata);break ; } }
arch/i386/pci/numa.c +5 -3
@@ -97,9 +97,11 @@ static void __devinit pci_fixup_i450nx(spci_read_config_byte (d, reg++, &subb);DBG ("i450NX PXB %d: %02x/%02x/%02x\n" , pxb, busno, suba, subb);if (busno) -pci_scan_bus (QUADLOCAL2BUS (quad,busno), &pci_root_ops,NULL );/* Bus A */ +pci_scan_bus (QUADLOCAL2BUS (quad,busno), &pci_root_ops, + &pci_default_sysdata);/* Bus A */ if (suba < subb) -pci_scan_bus (QUADLOCAL2BUS (quad,suba+1 ), &pci_root_ops,NULL );/* Bus B */ +pci_scan_bus (QUADLOCAL2BUS (quad,suba+1 ), &pci_root_ops, + &pci_default_sysdata);/* Bus B */ } pcibios_last_bus =-1 ; }
@@ -124,7 +126,7 @@ static int __init pci_numa_init(void)printk ("Scanning PCI bus %d for quad %d\n" ,QUADLOCAL2BUS (quad,0 ), quad);pci_scan_bus (QUADLOCAL2BUS (quad,0 ), - &pci_root_ops,NULL ); + &pci_root_ops, &pci_default_sysdata); }return 0 ; }
arch/i386/pci/visws.c +2 -2
@@ -102,8 +102,8 @@ static int __init pcibios_init(void) "bridge B (PIIX4) bus: %u\n", pci_bus1, pci_bus0); raw_pci_ops = &pci_direct_conf1; -pci_scan_bus (pci_bus0, &pci_root_ops,NULL ); -pci_scan_bus (pci_bus1, &pci_root_ops,NULL ); +pci_scan_bus (pci_bus0, &pci_root_ops, &pci_default_sysdata); +pci_scan_bus (pci_bus1, &pci_root_ops, &pci_default_sysdata);pci_fixup_irqs (visws_swizzle, visws_map_irq);pcibios_resource_survey ();return 0 ;
include/asm-i386/pci.h +1 -0
@@ -9,6 +9,7 @@ struct pci_sysdata {int domain;/* PCI domain */ int node;/* NUMA node */ }; +extern struct pci_sysdata pci_default_sysdata;#ifdef CONFIG_PCI_DOMAINS static inline int pci_domain_nr (struct pci_bus *bus)