Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3
Not sure if you've seen this already ... catching up on test results.
This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this problem.
Full logs:
mm2 - http://test.kernel.org/abat/50727/debug/console.log
mm3 - http://test.kernel.org/abat/51442/debug/console.log
config - http://test.kernel.org/abat/51442/build/dotconfig
I'm guessing from the 00000004 that the pcibus_to_node(dev->bus) is failing because bus->sysdata is NULL. The disassembly and structure offsets seem to line up for that.
#define pcibus_to_node(bus) ( (struct pci_sysdata *)((bus)->sysdata))->node
struct pci_sysdata {
int domain; /* PCI domain */
int node; /* NUMA node */};
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip: c02060d4 *pde = 0042c001 *pte = 00000000 Oops: 0000 [#1] SMP last sysfs file: Modules linked in: CPU: 2 EIP: 0060:[<c02060d4>] Not tainted VLI EFLAGS: 00010286 (2.6.18-mm2-autokern1 #1)
EIP is at pci_call_probe+0x19/0xb5
eax: 00000000 ebx: e778b400 ecx: e7400030 edx: c03b89a0
esi: e778b400 edi: 0000ffff ebp: e69a2fa0 esp: e740dea4
ds: 007b es: 007b ss: 0068
Process swapper (pid: 1, ti=e740c000 task=e7400030 task.ti=e740c000)
Stack: ffffffed e778b400 c03b89a0 c02061a3 c03b89a0 e778b400 c03b873c c03b89a0
e778b400 c03b89d4 c02061d6 c03b89a0 e778b400 e778b400 c03b89d4
e778b448
c0224800 e778b448 c03b89d4 e778b448 00000000 c03b89d4 c0224936
e69a2fa0
Call Trace:
[<c02061a3>] __pci_device_probe+0x33/0x47
[<c02061d6>] pci_device_probe+0x1f/0x34
[<c0224800>] really_probe+0x31/0xb9
[<c0224936>] driver_probe_device+0x93/0x9c
[<c02249b5>] __driver_attach+0x0/0x7c
[<c02249fc>] __driver_attach+0x47/0x7c
[<c0223e98>] bus_for_each_dev+0x47/0x6d
[<c01fd05a>] kobject_add+0xa9/0xf2
[<c0224a45>] driver_attach+0x14/0x18
[<c02249b5>] __driver_attach+0x0/0x7c
[<c022437b>] bus_add_driver+0x53/0xd0
[<c0224d99>] driver_register+0x74/0x77
[<c02063ea>] __pci_register_driver+0x6b/0x7a
[<c04146c3>] qla1280_init+0xc/0xf
[<c04007ff>] do_initcalls+0x55/0xe8
[<c0184095>] proc_mkdir+0x12/0x16
[<c0135136>] init_irq_proc+0x21/0x2f
[<c01003b8>] init+0x0/0x148
[<c010040d>] init+0x55/0x148
[<c01033c7>] kernel_thread_helper+0x7/0x10Quoted 1 line · On Sun, 08 Oct 2006 00:02:07 -0700 "Martin J. Bligh" <[email protected]> wrote:
Not sure if you've seen this already ... catching up on test results.
Nope.
Quoted 17 lines
This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this problem.
Full logs:
mm2 - http://test.kernel.org/abat/50727/debug/console.log
mm3 - http://test.kernel.org/abat/51442/debug/console.logconfig - http://test.kernel.org/abat/51442/build/dotconfig
I'm guessing from the 00000004 that the
pcibus_to_node(dev->bus)is failing becausebus->sysdatais NULL. The disassembly and structure offsets seem to line up for that.#define pcibus_to_node(bus) ( (struct pci_sysdata *)((bus)->sysdata))->node
Quoted 8 lines
struct
pci_sysdata{int domain; /* PCI domain */ int node; /* NUMA node */};
BUG: unable to handle kernel NULL pointer dereference at virtual address00000004
I don't see anything in the mm1->mm2 additions which might have caused this. Don't know, sorry. Bisection time?
Quoted 43 lines
printing eip:
c02060d4
*pde =0042c001
*pte = 00000000Oops: 0000 [#1] SMP last sysfs file: Modules linked in: CPU: 2 EIP: 0060:[<c02060d4>] Not tainted VLI EFLAGS: 00010286 (2.6.18-mm2-autokern1 #1)EIP is at
pci_call_probe+0x19/0xb5
eax: 00000000 ebx:e778b400ecx:e7400030edx:c03b89a0
esi:e778b400edi:0000ffffebp:e69a2fa0esp:e740dea4
ds: 007b es: 007b ss: 0068
Process swapper (pid: 1, ti=e740c000 task=e7400030 task.ti=e740c000)Stack: ffffffed e778b400 c03b89a0 c02061a3 c03b89a0 e778b400 c03b873c c03b89a0 e778b400 c03b89d4 c02061d6 c03b89a0 e778b400 e778b400 c03b89d4 e778b448 c0224800 e778b448 c03b89d4 e778b448 00000000 c03b89d4 c0224936 e69a2fa0 Call Trace: [<c02061a3>] __pci_device_probe+0x33/0x47 [<c02061d6>] pci_device_probe+0x1f/0x34 [<c0224800>] really_probe+0x31/0xb9 [<c0224936>] driver_probe_device+0x93/0x9c [<c02249b5>] __driver_attach+0x0/0x7c [<c02249fc>] __driver_attach+0x47/0x7c [<c0223e98>] bus_for_each_dev+0x47/0x6d [<c01fd05a>] kobject_add+0xa9/0xf2 [<c0224a45>] driver_attach+0x14/0x18 [<c02249b5>] __driver_attach+0x0/0x7c [<c022437b>] bus_add_driver+0x53/0xd0 [<c0224d99>] driver_register+0x74/0x77 [<c02063ea>] __pci_register_driver+0x6b/0x7a [<c04146c3>] qla1280_init+0xc/0xf [<c04007ff>] do_initcalls+0x55/0xe8 [<c0184095>] proc_mkdir+0x12/0x16 [<c0135136>] init_irq_proc+0x21/0x2f [<c01003b8>] init+0x0/0x148 [<c010040d>] init+0x55/0x148 [<c01033c7>] kernel_thread_helper+0x7/0x10
Quoted 16 lines · Martin J. Bligh
Not sure if you've seen this already ... catching up on test results.
This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this problem.
Full logs:
mm2 - http://test.kernel.org/abat/50727/debug/console.log
mm3 - http://test.kernel.org/abat/51442/debug/console.logconfig - http://test.kernel.org/abat/51442/build/dotconfig
I'm guessing from the 00000004 that the
pcibus_to_node(dev->bus)is failing becausebus->sysdatais NULL. The disassembly and structure offsets seem to line up for that.
Quoted 8 lines
#define pcibus_to_node(bus) ( (struct pci_sysdata *)((bus)->sysdata))->nodestruct
pci_sysdata{int domain; /* PCI domain */ int node; /* NUMA node */};
Martin,
Jeff moved "node" to a proper field in sysdata, instead of overloading sysdata itself. I think this is causing the problem. I guess we could end up with sysdata = NULL in some cases ? Since you are the NUMA-Q expert, where does sysdata gets set for NUMA-Q ? :)
-mm2 changed:
#define pcibus_to_node(bus) ((long) (bus)->sysdata)
to
#define pcibus_to_node(bus) ((struct pci_sysdata *)((bus)->sysdata))-
Quoted 1 line
node
Thanks,
Badari
Quoted 50 lines
BUG: unable to handle kernel NULL pointer dereference at virtual address00000004
printing eip:c02060d4
*pde =0042c001
*pte = 00000000Oops: 0000 [#1] SMP last sysfs file: Modules linked in: CPU: 2 EIP: 0060:[<c02060d4>] Not tainted VLI EFLAGS: 00010286 (2.6.18-mm2-autokern1 #1)EIP is at
pci_call_probe+0x19/0xb5
eax: 00000000 ebx:e778b400ecx:e7400030edx:c03b89a0
esi:e778b400edi:0000ffffebp:e69a2fa0esp:e740dea4
ds: 007b es: 007b ss: 0068
Process swapper (pid: 1, ti=e740c000 task=e7400030 task.ti=e740c000)Stack: ffffffed e778b400 c03b89a0 c02061a3 c03b89a0 e778b400 c03b873c c03b89a0 e778b400 c03b89d4 c02061d6 c03b89a0 e778b400 e778b400 c03b89d4 e778b448 c0224800 e778b448 c03b89d4 e778b448 00000000 c03b89d4 c0224936 e69a2fa0 Call Trace: [<c02061a3>] __pci_device_probe+0x33/0x47 [<c02061d6>] pci_device_probe+0x1f/0x34 [<c0224800>] really_probe+0x31/0xb9 [<c0224936>] driver_probe_device+0x93/0x9c [<c02249b5>] __driver_attach+0x0/0x7c [<c02249fc>] __driver_attach+0x47/0x7c [<c0223e98>] bus_for_each_dev+0x47/0x6d [<c01fd05a>] kobject_add+0xa9/0xf2 [<c0224a45>] driver_attach+0x14/0x18 [<c02249b5>] __driver_attach+0x0/0x7c [<c022437b>] bus_add_driver+0x53/0xd0 [<c0224d99>] driver_register+0x74/0x77 [<c02063ea>] __pci_register_driver+0x6b/0x7a [<c04146c3>] qla1280_init+0xc/0xf [<c04007ff>] do_initcalls+0x55/0xe8 [<c0184095>] proc_mkdir+0x12/0x16 [<c0135136>] init_irq_proc+0x21/0x2f [<c01003b8>] init+0x0/0x148 [<c010040d>] init+0x55/0x148 [<c01033c7>] kernel_thread_helper+0x7/0x10-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 35 lines · Badari Pulavarty
Quoted 16 lines · Martin J. Bligh
Not sure if you've seen this already ... catching up on test results.
This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this problem.
Full logs:
mm2 - http://test.kernel.org/abat/50727/debug/console.log
mm3 - http://test.kernel.org/abat/51442/debug/console.logconfig - http://test.kernel.org/abat/51442/build/dotconfig
I'm guessing from the 00000004 that the
pcibus_to_node(dev->bus)is failing becausebus->sysdatais NULL. The disassembly and structure offsets seem to line up for that.Quoted 8 lines
#define pcibus_to_node(bus) ( (struct pci_sysdata *)((bus)->sysdata))->nodestruct
pci_sysdata{int domain; /* PCI domain */ int node; /* NUMA node */};
Martin,
Jeff moved "node" to a proper field in sysdata, instead
of overloading sysdata itself. I think this is causing the
problem. I guess we could end up with sysdata = NULL in some
cases ? Since you are the NUMA-Q expert, where does sysdata gets set for NUMA-Q ? :)
Quoted 9 lines
-mm2 changed:
#define
pcibus_to_node(bus)((long) (bus)->sysdata)to
#define
pcibus_to_node(bus)((structpci_sysdata*)((bus)->sysdata))-Quoted 1 line
node
Buggered if I know, that's some strange pci thing ;-)
But can we revert whatever patch that was until it gets fixed, please?
Thanks,
M.
Quoted 40 lines · Martin Bligh
Quoted 39 lines · Badari Pulavarty
Quoted 16 lines · Martin J. Bligh
Not sure if you've seen this already ... catching up on test results.
This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this problem.
Full logs:
mm2 - http://test.kernel.org/abat/50727/debug/console.log
mm3 - http://test.kernel.org/abat/51442/debug/console.logconfig - http://test.kernel.org/abat/51442/build/dotconfig
I'm guessing from the 00000004 that the
pcibus_to_node(dev->bus)is failing becausebus->sysdatais NULL. The disassembly and structure offsets seem to line up for that.Quoted 8 lines
#define pcibus_to_node(bus) ( (struct pci_sysdata *)((bus)->sysdata))->nodestruct
pci_sysdata{int domain; /* PCI domain */ int node; /* NUMA node */};
Martin,
Jeff moved "node" to a proper field in sysdata, instead
of overloading sysdata itself. I think this is causing the
problem. I guess we could end up with sysdata = NULL in some
cases ? Since you are the NUMA-Q expert, where does sysdata gets set for NUMA-Q ? :)-mm2 changed:
#define
pcibus_to_node(bus)((long) (bus)->sysdata)
Quoted 8 lines
Quoted 4 lines
to
#definepcibus_to_node(bus)((structpci_sysdata*)((bus)->sysdata))-Quoted 1 line
node
Buggered if I know, that's some strange pci thing ;-)
But can we revert whatever patch that was until it gets fixed, please?
It needs to get fixed, otherwise whose buses of PCI devices disappear on some machines.
Can you turn on PCI debugging?
Jeff
Jeff Garzik [[email protected]] wrote:
| Martin Bligh wrote: | >Badari Pulavarty wrote: | >>On Sun, 2006-10-08 at 00:02 -0700, Martin J. Bligh wrote: | >> | >>>Not sure if you've seen this already ... catching up on test results. | >>> | >>>This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this | >>>problem. | >>> | >>>Full logs: | >>> | >>>mm2 - http://test.kernel.org/abat/50727/debug/console.log | >>>mm3 - http://test.kernel.org/abat/51442/debug/console.log | >>> | >>>config - http://test.kernel.org/abat/51442/build/dotconfig | >>> | >>>I'm guessing from the 00000004 that the pcibus_to_node(dev->bus) | >>>is failing because bus->sysdata is NULL. The disassembly and | >>>structure offsets seem to line up for that. | >>> | >>>#define pcibus_to_node(bus) ( | >>> (struct pci_sysdata *)((bus)->sysdata))->node | >>> | >>>struct pci_sysdata { | >>> int domain; /* PCI domain */ | >>> int node; /* NUMA node */ | >>>}; | >>> | >> | >> | >>Martin, | >> | >>Jeff moved "node" to a proper field in sysdata, instead | >>of overloading sysdata itself. I think this is causing the | >>problem. I guess we could end up with sysdata = NULL in some | >>cases ? Since you are the NUMA-Q expert, where does sysdata gets set | >>for NUMA-Q ? :) | >> | >>-mm2 changed: | >> | >>#define pcibus_to_node(bus) ((long) (bus)->sysdata) | >> | >>to | >>#define pcibus_to_node(bus) ((struct pci_sysdata *)((bus)->sysdata))- | >> | >>>node | > | >Buggered if I know, that's some strange pci thing ;-) | > | >But can we revert whatever patch that was until it gets fixed, please? | | It needs to get fixed, otherwise whose buses of PCI devices disappear on | some machines. | | Can you turn on PCI debugging? | | Jeff
I turned on CONFIG_PCI_DEBUG. already had CONFIG_KERNEL_DEBUG and CONFIG_DEBUG_INFO. Booted with "debug" parameter.
Here are the last few messages. Complete dmesg attached.
Let me know if there are other config tokens or boot options that may provide more info.
Suka
---
PCI: Calling quirk c024da00 for 0000:00:0a.0
PCI: Calling quirk c031ce30 for 0000:00:0a.0
PCI: Calling quirk c024da00 for 0000:00:0b.0
PCI: Calling quirk c031ce30 for 0000:00:0b.0
PCI: Calling quirk c024da00 for 0000:00:0e.0
PCI: Calling quirk c031ce30 for 0000:00:0e.0
PCI: Calling quirk c024da00 for 0000:00:0e.1
PCI: Calling quirk c031ce30 for 0000:00:0e.1
PCI: Calling quirk c024da00 for 0000:00:0e.2
PCI: Calling quirk c031ce30 for 0000:00:0e.2
PCI: Calling quirk c024da00 for 0000:00:0e.3
PCI: Calling quirk c031ce30 for 0000:00:0e.3
PCI: Calling quirk c024da00 for 0000:00:10.0
PCI: Calling quirk c031ce30 for 0000:00:10.0
PCI: Calling quirk c024da00 for 0000:00:12.0
PCI: Calling quirk c05093b0 for 0000:00:12.0
PCI: Calling quirk c031ce30 for 0000:00:12.0
PCI: Calling quirk c024da00 for 0000:00:14.0
PCI: Calling quirk c05093b0 for 0000:00:14.0
PCI: Calling quirk c031ce30 for 0000:00:14.0
PCI: Calling quirk c024da00 for 0000:01:0c.0
PCI: Calling quirk c031ce30 for 0000:01:0c.0
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
loop: loaded (max 8 devices)
Intel(R) PRO/1000 Network Driver - version 7.2.9-k2
Copyright (c) 1999-2006 Intel Corporation.
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip: c024e92c *pde = 00529001 *pte = 00000000 Oops: 0000 [#1]
SMP
last sysfs file:
Modules linked in: CPU: 0 EIP: 0060:[<c024e92c>] Not tainted VLI EFLAGS: 00010286 (2.6.18-mm3 #3)
EIP is at pci_call_probe+0x1c/0xe0
eax: 00000000 ebx: dfe63c00 ecx: c10fca90 edx: c048ad60
esi: dfe63c00 edi: 0000000f ebp: dfc3fd00 esp: c10ffe70
ds: 007b es: 007b ss: 0068
Process swapper (pid: 1, ti=c10fe000 task=c10fca90 task.ti=c10fe000)
Stack: dfe63c00 ffffffed ffffffed dfe63c00 c048b1c0 c024ea55 c048b1c0 dfe63c00
c048ad60 c048b1c0 dfe63c00 c048b1f4 c024ea9f c048b1c0 dfe63c00 dfe63c48
dfc3fd00 c02761f9 dfe63c48 00000286 dfff9060 000000d0 c048b1f4 dfc3fd00
Call Trace:
[<c024ea55>] __pci_device_probe+0x65/0x80
[<c024ea9f>] pci_device_probe+0x2f/0x50
[<c02761f9>] really_probe+0xf9/0x100
[<c02762e8>] driver_probe_device+0xc8/0xe0
[<c03ae92d>] klist_next+0x5d/0xa0
[<c02763a0>] __driver_attach+0x0/0xa0
[<c0276430>] __driver_attach+0x90/0xa0
[<c0275409>] bus_for_each_dev+0x69/0x80
[<c0276465>] driver_attach+0x25/0x30
[<c02763a0>] __driver_attach+0x0/0xa0
[<c0275ad3>] bus_add_driver+0x73/0x140
[<c024edc4>] __pci_register_driver+0x74/0x90
[<c050c6f9>] tulip_init+0x29/0x30
[<c04f2a62>] do_initcalls+0x42/0x140
[<c01433eb>] register_irq_proc+0xab/0xd0
[<c01003f0>] init+0x0/0x1a0
[<c0143479>] init_irq_proc+0x39/0x50
[<c01003f0>] init+0x0/0x1a0
[<c0100451>] init+0x61/0x1a0
[<c0103c0b>] kernel_thread_helper+0x7/0x1c
=======================
Code: 74 92 eb 8e 8d 74 26 00 8d bc 27 00 00 00 00 57 56 53 83 ec 08 8b 5c 24 1c 89 e0 25 00 e0 ff ff 8b 08 8b 43 10 8b 79 5c 8b 40 44 <8b> 50 04 85 d2 78 11 0f a3 15 c0 9f 4e c0 19 c0 85 c0 0f 85 8cBIOS-provided physical RAM map:
sanitize start
sanitize endcopy_e820_map() start: 0000000000000000 size: 000000000009fc00 end: 000000000009fc00 type: 1 copy_e820_map() type is E820_RAMcopy_e820_map() start: 0000000000100000 size: 000000003ff00000 end: 0000000040000000 type: 1 copy_e820_map() type is E820_RAMcopy_e820_map() start: 00000000fec00000 size: 0000000000009000 end: 00000000fec09000 type: 2 copy_e820_map() start: 00000000ffe80000 size: 0000000000180000 end: 0000000100000000 type: 2
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 0000000000100000 - 0000000040000000 (usable)
BIOS-e820: 00000000fec00000 - 00000000fec09000 (reserved)
BIOS-e820: 00000000ffe80000 - 0000000100000000 (reserved)
Node: 0, start_pfn: 0, end_pfn: 159
Setting physnode_map array to node 0 for pfns:
0
Node: 0, start_pfn: 256, end_pfn: 262144
Setting physnode_map array to node 0 for pfns:
256 65792 131328 196864
Node: 0, start_pfn: 0, end_pfn: 262144
Setting physnode_map array to node 0 for pfns:
0 65536 131072 196608
Reserving 2560 pages of KVA for lmem_map of node 0
Shrinking node 0 from 262144 pages to 259584 pages
Reserving total of 2560 pages for numa KVA remapkva_start_pfn ~ 226816 find_max_low_pfn() ~ 229376max_pfn = 262144
128MB HIGHMEM available.
896MB LOWMEM available.min_low_pfn = 1385, max_low_pfn = 229376, highstart_pfn = 229376
Low memory ends at vaddr f8000000
node 0 will remap to vaddr f7600000 - f8a00000
High memory starts at vaddr f8000000
found SMP MP-table at 000f6040
Entering add_active_range(0, 0, 259584) 0 entries of 256 used
Zone PFN ranges:
DMA 0 -> 4096
Normal 4096 -> 229376
HighMem 229376 -> 262144
early_node_map[1] active PFN ranges
0: 0 -> 259584
On node 0 totalpages: 259584
DMA zone: 32 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 4064 pages, LIFO batch:0
Normal zone: 1760 pages used for memmap
Normal zone: 223520 pages, LIFO batch:31 HighMem zone: 236 pages used for memmap
HighMem zone: 29972 pages, LIFO batch:7
DMI not present or invalid.
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: IBM NUMA Product ID: SBB <6>Found an OEM MPC table at 1fff82a8- parsing it ...
Translation: record 0, type 1, quad 0, global 3, local 3
Translation: record 1, type 1, quad 0, global 1, local 1
Translation: record 2, type 1, quad 0, global 1, local 1
Translation: record 3, type 1, quad 0, global 1, local 1
Translation: record 4, type 3, quad 0, global 0, local 0
Translation: record 5, type 3, quad 0, global 1, local 1
Translation: record 6, type 4, quad 0, global 2, local 18
Translation: record 7, type 2, quad 0, global 13, local 14
Translation: record 8, type 2, quad 0, global 14, local 13 APIC at: 0xFEC08000
Processor #0 6:10 APIC version 17 (quad 0, apic 1) Processor #4 6:10 APIC version 17 (quad 0, apic 8) Processor #1 6:10 APIC version 17 (quad 0, apic 2) Processor #2 6:10 APIC version 17 (quad 0, apic 4)
Bus #0 is PCI (node 0) Bus #1 is PCI (node 0) Bus #2 is EISA (node 0)
I/O APIC #13 Version 17 at 0xFEC00000.
I/O APIC #14 Version 17 at 0xFEC01000.
Enabling APIC mode: NUMA-Q. Using 2 I/O APICs
Processors: 4
Allocating PCI resources starting at 50000000 (gap: 40000000:bec00000)
Detected 700.054 MHz processor.
Built 1 zonelists. Total pages: 257556
Kernel command line: root=/dev/sda1 console=ttyS0,57600 reboot=bios debug
mapped APIC to ffffd000 (fec08000)
mapped IOAPIC to ffffc000 (fec00000)
mapped IOAPIC to ffffb000 (fec01000)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Initializing HighMem for node 0 (00038000:0003f600)
Memory: 1022216k/1048576k available (2757k kernel code, 15732k reserved, 1252k data, 216k init, 120832k highmem) virtual kernel memory layout:
fixmap : 0xffe1c000 - 0xfffff000 (1932 kB)
pkmap : 0xffc00000 - 0xffe00000 (2048 kB)
vmalloc : 0xf8800000 - 0xffbfe000 ( 115 MB)
lowmem : 0xc0000000 - 0xf8000000 ( 896 MB)
.init : 0xc04f2000 - 0xc0528000 ( 216 kB)
.data : 0xc03b16ac - 0xc04eaa4c (1252 kB)
.text : 0xc0100000 - 0xc03b16ac (2757 kB)Checking if this processor honours the WP bit even in supervisor mode... Ok. Calibrating delay using timer specific routine.. 1401.31 BogoMIPS (lpj=2802636) Mount-cache hash table entries: 512
CPU: After generic identify, caps: 0387fbff 00000000 00000000 00000000 0000000000000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU serial number disabled.
CPU: After all inits, caps: 0383fbff 00000000 00000000 00000040 00000000 00000000 00000000 Compat vDSO mapped to ffffe000.
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 16k freed
CPU0: Intel Pentium III (Cascades) stepping 00
Leaving ESR disabled.
Mapping cpu 0 to node 0
Booting processor 1/2 eip 2000
Storing NMI vector
Initializing CPU#1
Leaving ESR disabled.
Mapping cpu 1 to node 0
Calibrating delay using timer specific routine.. 1400.08 BogoMIPS (lpj=2800175)
CPU: After generic identify, caps: 0387fbff 00000000 00000000 00000000 0000000000000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU serial number disabled.
CPU: After all inits, caps: 0383fbff 00000000 00000000 00000040 00000000 00000000 00000000
CPU1: Intel Pentium III (Cascades) stepping 00
Booting processor 2/4 eip 2000
Storing NMI vector
Initializing CPU#2
Leaving ESR disabled.
Mapping cpu 2 to node 0
Calibrating delay using timer specific routine.. 1400.06 BogoMIPS (lpj=2800120)
CPU: After generic identify, caps: 0387fbff 00000000 00000000 00000000 0000000000000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU serial number disabled.
CPU: After all inits, caps: 0383fbff 00000000 00000000 00000040 00000000 00000000 00000000
CPU2: Intel Pentium III (Cascades) stepping 00
Booting processor 3/8 eip 2000
Storing NMI vector
Initializing CPU#3
Leaving ESR disabled.
Mapping cpu 3 to node 0
Calibrating delay using timer specific routine.. 1400.06 BogoMIPS (lpj=2800124)
CPU: After generic identify, caps: 0387fbff 00000000 00000000 00000000 0000000000000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU serial number disabled.
CPU: After all inits, caps: 0383fbff 00000000 00000000 00000040 00000000 00000000 00000000
CPU3: Intel Pentium III (Cascades) stepping 00
Total of 4 processors activated (5601.52 BogoMIPS).
ExtINT not setup in hardware but reported by MP table
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=0 pin2=0
..MP-BIOS bug: 8254 timer not connected to IO-APIC
...trying to set up timer (IRQ0) through the 8259A ...
..... (found pin 0) ... failed.
...trying to set up timer as Virtual Wire IRQ... works.
checking TSC synchronization across 4 CPUs: passed.
Brought up 4 CPUs
migration_cost=12000
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfd231, last bus=1
PCI: Using configuration type 1
Setting up standard PCI resources
SCSI subsystem initialized
PCI: Probing PCI hardware (bus 00)
PCI: Scanning bus 0000:00
PCI: Found 0000:00:0a.0 [1077/1020] 000100 00
PCI: Calling quirk c024db60 for 0000:00:0a.0
PCI: Calling quirk c024e240 for 0000:00:0a.0
PCI: Found 0000:00:0b.0 [1002/4750] 000300 00
PCI: Calling quirk c024db60 for 0000:00:0b.0
PCI: Calling quirk c024e240 for 0000:00:0b.0
Boot video device is 0000:00:0b.0
PCI: Found 0000:00:0e.0 [8086/7110] 000601 00
PCI: Calling quirk c024e050 for 0000:00:0e.0
PCI: Calling quirk c024db60 for 0000:00:0e.0
PCI: Calling quirk c024e240 for 0000:00:0e.0
PCI: Found 0000:00:0e.1 [8086/7111] 000101 00
PCI: Calling quirk c024e050 for 0000:00:0e.1
PCI: Calling quirk c024db60 for 0000:00:0e.1
PCI: Calling quirk c024e240 for 0000:00:0e.1
PCI: Found 0000:00:0e.2 [8086/7112] 000c03 00
PCI: Calling quirk c024e050 for 0000:00:0e.2
PCI: Calling quirk c024db60 for 0000:00:0e.2
PCI: Calling quirk c024e240 for 0000:00:0e.2
PCI: Found 0000:00:0e.3 [8086/7113] 000680 00
PCI: Calling quirk c024e050 for 0000:00:0e.3
PCI: Calling quirk c024d150 for 0000:00:0e.3
PCI quirk: region 0c00-0c3f claimed by PIIX4 ACPI
PIIX4 devres C PIO at 007c-007c
PCI: Calling quirk c024db60 for 0000:00:0e.3
PCI: Calling quirk c024e240 for 0000:00:0e.3
PCI: Found 0000:00:10.0 [8086/84ca] 000600 00
PCI: Calling quirk c024e050 for 0000:00:10.0
PCI: Calling quirk c024db60 for 0000:00:10.0
PCI: Calling quirk c024e240 for 0000:00:10.0
PCI: Calling quirk c032f670 for 0000:00:10.0
PCI: Searching for i450NX host bridges on 0000:00:10.0
PCI: Scanning bus 0000:01
PCI: Found 0000:01:0c.0 [1011/0009] 000200 00
PCI: Calling quirk c024db60 for 0000:01:0c.0
PCI: Calling quirk c024e240 for 0000:01:0c.0
PCI: Fixups for bus 0000:01
PCI: Bus scan for 0000:01 returning with max=01
PCI: Found 0000:00:12.0 [8086/84cb] 000600 00
PCI: Calling quirk c024e050 for 0000:00:12.0
PCI: Calling quirk c024db60 for 0000:00:12.0
PCI: Calling quirk c024e240 for 0000:00:12.0
PCI: Found 0000:00:14.0 [8086/84cb] 000600 00
PCI: Calling quirk c024e050 for 0000:00:14.0
PCI: Calling quirk c024db60 for 0000:00:14.0
PCI: Calling quirk c024e240 for 0000:00:14.0
PCI: Fixups for bus 0000:00
PCI: Bus scan for 0000:00 returning with max=00PCI->APIC IRQ transform: 0000:01:0c.0[A] -> IRQ 40PCI->APIC IRQ transform: 0000:00:0a.0[A] -> IRQ 23PCI->APIC IRQ transform: 0000:00:0b.0[A] -> IRQ 19
got res [50000000:5001ffff] bus [50000000:5001ffff] flags 7202 for BAR 6 of 0000:00:0b.0 got res [50020000:5002ffff] bus [50020000:5002ffff] flags 7200 for BAR 6 of 0000:00:0a.0 got res [1000:101f] bus [1000:101f] flags 101 for BAR 4 of 0000:00:0e.2
PCI: moved device 0000:00:0e.2 resource 4 (101) to 1000
got res [50040000:5007ffff] bus [50040000:5007ffff] flags 7200 for BAR 6 of 0000:01:0c.0
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
highmem bounce pool size: 64 pages
Installing knfsd (copyright (C) 1996 [email protected]).
JFS: nTxBlock = 7986, nTxLock = 63889
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
PCI: Calling quirk c024da00 for 0000:00:0a.0
PCI: Calling quirk c031ce30 for 0000:00:0a.0
PCI: Calling quirk c024da00 for 0000:00:0b.0
PCI: Calling quirk c031ce30 for 0000:00:0b.0
PCI: Calling quirk c024da00 for 0000:00:0e.0
PCI: Calling quirk c031ce30 for 0000:00:0e.0
PCI: Calling quirk c024da00 for 0000:00:0e.1
PCI: Calling quirk c031ce30 for 0000:00:0e.1
PCI: Calling quirk c024da00 for 0000:00:0e.2
PCI: Calling quirk c031ce30 for 0000:00:0e.2
PCI: Calling quirk c024da00 for 0000:00:0e.3
PCI: Calling quirk c031ce30 for 0000:00:0e.3
PCI: Calling quirk c024da00 for 0000:00:10.0
PCI: Calling quirk c031ce30 for 0000:00:10.0
PCI: Calling quirk c024da00 for 0000:00:12.0
PCI: Calling quirk c05093b0 for 0000:00:12.0
PCI: Calling quirk c031ce30 for 0000:00:12.0
PCI: Calling quirk c024da00 for 0000:00:14.0
PCI: Calling quirk c05093b0 for 0000:00:14.0
PCI: Calling quirk c031ce30 for 0000:00:14.0
PCI: Calling quirk c024da00 for 0000:01:0c.0
PCI: Calling quirk c031ce30 for 0000:01:0c.0
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
loop: loaded (max 8 devices)
Intel(R) PRO/1000 Network Driver - version 7.2.9-k2
Copyright (c) 1999-2006 Intel Corporation.
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip: c024e92c *pde = 00529001 *pte = 00000000 Oops: 0000 [#1] SMP last sysfs file: Modules linked in: CPU: 0 EIP: 0060:[<c024e92c>] Not tainted VLI EFLAGS: 00010286 (2.6.18-mm3 #3)
EIP is at pci_call_probe+0x1c/0xe0
eax: 00000000 ebx: dfe63c00 ecx: c10fca90 edx: c048ad60
esi: dfe63c00 edi: 0000000f ebp: dfc3fd00 esp: c10ffe70
ds: 007b es: 007b ss: 0068
Process swapper (pid: 1, ti=c10fe000 task=c10fca90 task.ti=c10fe000)
Stack: dfe63c00 ffffffed ffffffed dfe63c00 c048b1c0 c024ea55 c048b1c0 dfe63c00
c048ad60 c048b1c0 dfe63c00 c048b1f4 c024ea9f c048b1c0 dfe63c00 dfe63c48
dfc3fd00 c02761f9 dfe63c48 00000286 dfff9060 000000d0 c048b1f4 dfc3fd00
Call Trace:
[<c024ea55>] __pci_device_probe+0x65/0x80
[<c024ea9f>] pci_device_probe+0x2f/0x50
[<c02761f9>] really_probe+0xf9/0x100
[<c02762e8>] driver_probe_device+0xc8/0xe0
[<c03ae92d>] klist_next+0x5d/0xa0
[<c02763a0>] __driver_attach+0x0/0xa0
[<c0276430>] __driver_attach+0x90/0xa0
[<c0275409>] bus_for_each_dev+0x69/0x80
[<c0276465>] driver_attach+0x25/0x30
[<c02763a0>] __driver_attach+0x0/0xa0
[<c0275ad3>] bus_add_driver+0x73/0x140
[<c024edc4>] __pci_register_driver+0x74/0x90
[<c050c6f9>] tulip_init+0x29/0x30
[<c04f2a62>] do_initcalls+0x42/0x140
[<c01433eb>] register_irq_proc+0xab/0xd0
[<c01003f0>] init+0x0/0x1a0
[<c0143479>] init_irq_proc+0x39/0x50
[<c01003f0>] init+0x0/0x1a0
[<c0100451>] init+0x61/0x1a0
[<c0103c0b>] kernel_thread_helper+0x7/0x1c
=======================
Code: 74 92 eb 8e 8d 74 26 00 8d bc 27 00 00 00 00 57 56 53 83 ec 08 8b 5c 24 1c 89 e0 25 00 e0 ff ff 8b 08 8b 43 10 8b 79 5c 8b 40 44 <8b> 50 04 85 d2 78 11 0f a3 15 c0 9f 4e c0 19 c0 85 c0 0f 85 8c
EIP: [<c024e92c>] pci_call_probe+0x1c/0xe0 SS:ESP 0068:c10ffe70
<0>Kernel panic - not syncing: Attempted to kill init!Quoted 41 lines · Martin Bligh
Quoted 40 lines · Badari Pulavarty
Quoted 16 lines · Martin J. Bligh
Not sure if you've seen this already ... catching up on test results.
This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this problem.
Full logs:
mm2 - http://test.kernel.org/abat/50727/debug/console.log
mm3 - http://test.kernel.org/abat/51442/debug/console.logconfig - http://test.kernel.org/abat/51442/build/dotconfig
I'm guessing from the 00000004 that the
pcibus_to_node(dev->bus)is failing becausebus->sysdatais NULL. The disassembly and structure offsets seem to line up for that.Quoted 8 lines
#define pcibus_to_node(bus) ( (struct pci_sysdata *)((bus)->sysdata))->nodestruct
pci_sysdata{int domain; /* PCI domain */ int node; /* NUMA node */};
Martin,
Jeff moved "node" to a proper field in sysdata, instead of overloading sysdata itself. I think this is causing the problem. I guess we could end up with sysdata = NULL in some cases ? Since you are the NUMA-Q expert, where does sysdata gets set for NUMA-Q ? :)
-mm2 changed:
#define
pcibus_to_node(bus)((long) (bus)->sysdata)
Quoted 8 lines
Quoted 4 lines
to
#definepcibus_to_node(bus)((structpci_sysdata*)((bus)->sysdata))-Quoted 1 line
node
Buggered if I know, that's some strange pci thing ;-)
But can we revert whatever patch that was until it gets fixed, please?
Unless I am going very very mad, this has came up once before some months ago. We went through lots of pain finding the cause of this for NUMA-Q and fixing it. Something about not having a sysdata and needing to initialise it.
Thought so, this was all discussed back in December 2005.
http://lkml.org/lkml/2005/12/20/226
I'll go see if I can forward port the patch and address the remaining issues with it.
-apw
Quoted 56 lines · Andy Whitcroft
Quoted 41 lines · Martin Bligh
Quoted 40 lines · Badari Pulavarty
Quoted 16 lines · Martin J. Bligh
Not sure if you've seen this already ... catching up on test results.
This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this problem.
Full logs:
mm2 - http://test.kernel.org/abat/50727/debug/console.log
mm3 - http://test.kernel.org/abat/51442/debug/console.logconfig - http://test.kernel.org/abat/51442/build/dotconfig
I'm guessing from the 00000004 that the
pcibus_to_node(dev->bus)is failing becausebus->sysdatais NULL. The disassembly and structure offsets seem to line up for that.Quoted 8 lines
#define pcibus_to_node(bus) ( (struct pci_sysdata *)((bus)->sysdata))->nodestruct
pci_sysdata{int domain; /* PCI domain */ int node; /* NUMA node */};
Martin,
Jeff moved "node" to a proper field in sysdata, instead of overloading sysdata itself. I think this is causing the problem. I guess we could end up with sysdata = NULL in some cases ? Since you are the NUMA-Q expert, where does sysdata gets set for NUMA-Q ? :)
-mm2 changed:
#define
pcibus_to_node(bus)((long) (bus)->sysdata)Quoted 8 lines
Quoted 4 lines
to
#definepcibus_to_node(bus)((structpci_sysdata*)((bus)->sysdata))-Quoted 1 line
node
Buggered if I know, that's some strange pci thing ;-)
But can we revert whatever patch that was until it gets fixed, please?
Unless I am going very very mad, this has came up once before some months ago. We went through lots of pain finding the cause of this for NUMA-Q and fixing it. Something about not having a sysdata and needing to initialise it.
Quoted 6 lines
Thought so, this was all discussed back in December 2005.
http://lkml.org/lkml/2005/12/20/226
I'll go see if I can forward port the patch and address the remaining issues with it.
Yes, and I explicitly asked if this issue had been addressed again in these patches. That is why I rejected them oh so long ago...
bleah.
greg k-h
PATCH 1/1 pci device ensure sysdata initialised v2
Ok, I've gone over the patches and retested them. Other than having to extend them to cover x86_64 they still seem to work as planned. I have updated the commentry to better explain the problem and the fix as encapsulated here.
When this was proposed last time there was push back to get the nodes right. Whist this is clearly a good thing, I think we need this as a first step if the underlying patches are going to stay in.
-apw
=== 8< ===
pci device ensure sysdata initialised v2
We have been seeing panic's on NUMA systems in pci_call_probe() in 2.6.19-rc1-mm1 and later. This is related to the changes introduced in the commit below:
[x86, PCI] Switch pci_bus::sysdata from NUMA node integer to a pointer 0a247a58fc3e2ecfc17654301033e8b8d08df2a2
In this change the sysdata has changed from directly representing a value (the node number in NUMA) to a pointer to a structure. However, it seems that we do not always initialise this sysdata before we probe the device.
Prior to the changes above the node was defaulted to 'NULL' allocating the devices to node 0 unconditionally. This patch adds a default sysdata entry (pci_default_sysdata), this is then used where 'NULL' was used previously. pci_default_sysdata defaults the node to unknown (-1). This is a more accurate assignment, mirroring the value returned where no topology support is provided and no locality information is available.
There are only two uses of this value in the affected architectures (x86, x86_64) and generic code:
1) in x86_64, dma_alloc_pages() looks up the node in order to allocate node local memory. Here if the node is invalid we will default to the first online node. Behaviour here should be unchanged.
2) in generic, pci_call_probe() looks up the node in order to restrict execution of the probe on the card local node, to favor node local allocation. Where this is unknown previously we would force execution (and thereby allocation) to node 0, this is arguably wrong and using -1 releases this restriction.
In an ideal world we should be supplying a sysdata for the appropriate node where it is known. Where it is not known defaulting to -1 seems a better course, and would help us where node 0 is short of memory.
---
arch/i386/pci/common.c +2 -0
@@ -27,6 +27,8 @@ unsigned long pirq_table_addr;struct pci_bus *pci_root_bus;struct pci_raw_ops *raw_pci_ops; +struct pci_sysdata pci_default_sysdata = { .node =-1 }; +static int pci_read (struct pci_bus *bus,unsigned int devfn,int where,int size,u32 *value) { return raw_pci_ops->read(pci_domain_nr(bus), bus->number,
arch/i386/pci/fixup.c +5 -3
@@ -25,9 +25,11 @@ static void __devinit pci_fixup_i450nx(spci_read_config_byte (d, reg++, &subb);DBG ("i450NX PXB %d: %02x/%02x/%02x\n" , pxb, busno, suba, subb);if (busno) -pci_scan_bus (busno, &pci_root_ops,NULL );/* Bus A */ +pci_scan_bus (busno, &pci_root_ops, + &pci_default_sysdata);/* Bus A */ if (suba < subb) -pci_scan_bus (suba+1 , &pci_root_ops,NULL );/* Bus B */ +pci_scan_bus (suba+1 , &pci_root_ops, + &pci_default_sysdata);/* Bus B */ } pcibios_last_bus =-1 ; }
@@ -42,7 +44,7 @@ static void __devinit pci_fixup_i450gx(su8 busno;pci_read_config_byte (d,0x4a , &busno);printk (KERN_INFO "PCI: i440KX/GX host bridge %s: secondary bus %02x\n" ,pci_name (d), busno); -pci_scan_bus (busno, &pci_root_ops,NULL ); +pci_scan_bus (busno, &pci_root_ops, &pci_default_sysdata); pcibios_last_bus =-1 ; }DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_INTEL ,PCI_DEVICE_ID_INTEL_82454GX , pci_fixup_i450gx);
arch/i386/pci/legacy.c +2 -1
@@ -26,7 +26,8 @@ static void __devinit pcibios_fixup_peer l != 0x0000 && l != 0xffff) {DBG ("Found device at %02x:%02x [%04x]\n" , n, devfn, l);printk (KERN_INFO "PCI: Discovered peer bus %02x\n" , n); -pci_scan_bus (n, &pci_root_ops,NULL ); +pci_scan_bus (n, &pci_root_ops, + &pci_default_sysdata);break ; } }
arch/i386/pci/numa.c +5 -3
@@ -97,9 +97,11 @@ static void __devinit pci_fixup_i450nx(spci_read_config_byte (d, reg++, &subb);DBG ("i450NX PXB %d: %02x/%02x/%02x\n" , pxb, busno, suba, subb);if (busno) -pci_scan_bus (QUADLOCAL2BUS (quad,busno), &pci_root_ops,NULL );/* Bus A */ +pci_scan_bus (QUADLOCAL2BUS (quad,busno), &pci_root_ops, + &pci_default_sysdata);/* Bus A */ if (suba < subb) -pci_scan_bus (QUADLOCAL2BUS (quad,suba+1 ), &pci_root_ops,NULL );/* Bus B */ +pci_scan_bus (QUADLOCAL2BUS (quad,suba+1 ), &pci_root_ops, + &pci_default_sysdata);/* Bus B */ } pcibios_last_bus =-1 ; }
@@ -124,7 +126,7 @@ static int __init pci_numa_init(void)printk ("Scanning PCI bus %d for quad %d\n" ,QUADLOCAL2BUS (quad,0 ), quad);pci_scan_bus (QUADLOCAL2BUS (quad,0 ), - &pci_root_ops,NULL ); + &pci_root_ops, &pci_default_sysdata); }return 0 ; }
arch/i386/pci/visws.c +2 -2
@@ -101,8 +101,8 @@ static int __init pcibios_init(void) "bridge B (PIIX4) bus: %u\n", pci_bus1, pci_bus0); raw_pci_ops = &pci_direct_conf1; -pci_scan_bus (pci_bus0, &pci_root_ops,NULL ); -pci_scan_bus (pci_bus1, &pci_root_ops,NULL ); +pci_scan_bus (pci_bus0, &pci_root_ops, &pci_default_sysdata); +pci_scan_bus (pci_bus1, &pci_root_ops, &pci_default_sysdata);pci_fixup_irqs (visws_swizzle, visws_map_irq);pcibios_resource_survey ();return 0 ;
include/asm-i386/pci.h +1 -0
@@ -8,6 +8,7 @@ struct pci_sysdata {int domain;/* PCI domain */ int node;/* NUMA node */ }; +extern struct pci_sysdata pci_default_sysdata;#ifdef CONFIG_PCI_DOMAINS static inline int pci_domain_nr (struct pci_bus *bus)
include/asm-x86_64/pci.h +1 -0
@@ -10,6 +10,7 @@ struct pci_sysdata {int node;/* NUMA node */ void * iommu;/* IOMMU private data */ }; +extern struct pci_sysdata pci_default_sysdata;#ifdef CONFIG_PCI_DOMAINS static inline int pci_domain_nr (struct pci_bus *bus)
Re: [PATCH] pci device ensure sysdata initialised v2
Quoted 9 lines · Andy Whitcroft
Ok, I've gone over the patches and retested them. Other than having to extend them to cover
x86_64they still seem to work as planned. I have updated the commentry to better explain the problem and the fix as encapsulated here.When this was proposed last time there was push back to get the nodes right. Whist this is clearly a good thing, I think we need this as a first step if the underlying patches are going to stay in.
We definitely need something like this. Having some sysdata static and some sysdata dynamic makes me nervous, and so I want to review the code paths in depth before applying this. My gut feeling is that there is a bug remaining in this area, that this patch might exacerbate.
Jeff