SLQB: Mark the allocator as broken PowerPC and S390
The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in Kconfig for those architectures until the issues are resolved.
--- init/Kconfig | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
init/Kconfig +1 -0
@@ -1033,6 +1033,7 @@ config SLUB config SLQB bool "SLQB (Queued allocator)" + depends on !PPC && !S390 help SLQB is a proposed new slab allocator.
Signature · 1 line
1.5.6.3
Quoted 20 lines · Pekka Enberg
The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in
Kconfigfor those architectures until the issues are resolved.Cc Andrew Morton <[email protected]>Cc Christoph Lameter <[email protected]>Cc Heiko Carstens <[email protected]>Cc Ingo Molnar <[email protected]>Cc Nick Piggin <[email protected]>Cc Sachin Sant <[email protected]>Signed-off-by Pekka Enberg <[email protected]>--- init/Kconfig | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)init/Kconfig +0
-0
init/Kconfig config SLUB · line 1039
Quoted 7 lines
init/Kconfig +1
-0@@ -1033,6 +1033,7 @@ config SLUB config SLQB bool "SLQB (Queued allocator)" + depends on !PPC && !S390 help SLQB is a proposed new slab allocator.
I think this should be (!PPC && !S390) || EXPERIMENTAL so that it can still be enabled for debugging and development.
Is this in preparation for slqb inclusion as a non-default slab allocator in 2.6.32? 2.6.33?
Hi David,
Quoted 27 lines · Pekka Enberg
Quoted 27 lines
The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in
Kconfigfor those architectures until the issues are resolved.Cc Andrew Morton <[email protected]>Cc Christoph Lameter <[email protected]>Cc Heiko Carstens <[email protected]>Cc Ingo Molnar <[email protected]>Cc Nick Piggin <[email protected]>Cc Sachin Sant <[email protected]>Signed-off-by Pekka Enberg <[email protected]>--- init/Kconfig | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)init/Kconfig +1
-0@@ -1033,6 +1033,7 @@ config SLUB config SLQB bool "SLQB (Queued allocator)" + depends on !PPC && !S390 help SLQB is a proposed new slab allocator.
Quoted 2 lines · David Rientjes
I think this should be (!PPC && !S390) || EXPERIMENTAL so that it can still be enabled for debugging and development.
Everybody enables EXPERIMENTAL so that seems pointless. Developers can hack Kconfig locally, no?
Quoted 2 lines · David Rientjes
Is this in preparation for slqb inclusion as a non-default slab allocator in 2.6.32? 2.6.33?
Non-default for 2.6.32.
Pekka
Quoted 4 lines · Pekka Enberg
Quoted 3 lines · Pekka Enberg
Quoted 3 lines
The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in
Kconfigfor those architectures until the issues are resolved.
Looks ok to me. I'll debug s390 when time permits.
Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
init/Kconfig | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)init/Kconfig +0
-0@@ -1033,6 +1033,7 @@ config SLUBQuoted 6 lines
init/Kconfig config SLUB · line 1039
Quoted 6 lines
init/Kconfig +1
-0config SLQB bool "SLQB (Queued allocator)" + depends on !PPC && !S390 help SLQB is a proposed new slab allocator.On Tue, 2009-09-15 at 23:55 -0700, David Rientjes wrote:
Quoted 5 lines
Quoted 2 lines
I think this should be (!PPC && !S390) || EXPERIMENTAL so that it can still be enabled for debugging and development.
Everybody enables EXPERIMENTAL so that seems pointless. Developers can hack
Kconfiglocally, no?
Exactly.
Quoted 4 lines · Pekka Enberg
The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in
Kconfigfor those architectures until the issues are resolved.
Can the issues be summarised?
The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/
Quoted 34 lines
Cc Andrew Morton <[email protected]>Cc Christoph Lameter <[email protected]>Cc Heiko Carstens <[email protected]>Cc Ingo Molnar <[email protected]>Cc Nick Piggin <[email protected]>Cc Sachin Sant <[email protected]>Signed-off-by Pekka Enberg <[email protected]>--- init/Kconfig | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)init/Kconfig +1
-0@@ -1033,6 +1033,7 @@ config SLUB config SLQB bool "SLQB (Queued allocator)" + depends on !PPC && !S390 help SLQB is a proposed new slab allocator.Signature · 9 lines
1.5.6.3 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Hi Mel,
Quoted 5 lines · Pekka Enberg
Quoted 3 lines
The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in
Kconfigfor those architectures until the issues are resolved.Can the issues be summarised?
It's a boot time crash during module load:
http://www.mail-archive.com/[email protected]/msg33092.html
AFAICT, it's related to a memoryless node 0. Nick suggested it could be a latent bug in the kernel that's triggered by SLQB.
Quoted 2 lines · Mel Gorman
The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/
Maybe, but that's not an excuse to push something that's known to break.
The other alternative is to skip this release cycle but I'm not sure what we'd gain with that. Nick already stated in private that he'll try to arrange for some time with ppc machines to debug the thing and we hope to be able to fix it by 2.6.32 final.
Btw, the code is in slqb/core branch of slab.git in case someone wants to take a stab at fixing the bug.
Pekka
Quoted 7 lines · Pekka Enberg
Hi Mel,
Quoted 4 lines · Pekka Enberg
Quoted 3 lines
The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in
Kconfigfor those architectures until the issues are resolved.
Quoted 9 lines
Quoted 1 line
Can the issues be summarised?
It's a boot time crash during module load:
http://www.mail-archive.com/[email protected]/msg33092.html
AFAICT, it's related to a memoryless node 0. Nick suggested it could be a latent bug in the kernel that's triggered by SLQB.
The danger is that this isn't a PPC or s390 bug then as such, but a bug where there are either memoryless nodes or when node 0 is memoryless. Hence, there is no guarantee that your Kconfig option will catch all instances where this bug triggers. Granted, the configuration is most likely a PPC machine :)
Quoted 6 lines · Mel Gorman
Quoted 2 lines
The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/
Maybe, but that's not an excuse to push something that's known to break.
Wow, this is from back in May! Lame.
I'm against silently disabling it. Memoryless nodes are extremely rare but bugs crop up there occasionally and take a long time to catch and squash. SLQB breaking there is not going to cause widespread damage but force a fix to be developed by the people with access to the affected machines.
Quoted 5 lines
The other alternative is to skip this release cycle but I'm not sure what we'd gain with that. Nick already stated in private that he'll try to arrange for some time with ppc machines to debug the thing and we hope to be able to fix it by 2.6.32 final.
I have access to a ppc machine but not necessarily one with a memoryless nodes that can reproduce this problem.
Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?
Total aside, does anybody know handily if fake NUMA support allows the creation of memoryless nodes help reproducing problems like this? If I can't get a real machine, that'll be the approach I'll be trying.
Quoted 3 lines
Btw, the code is in slqb/core branch of slab.git in case someone wants to take a stab at fixing the bug.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Hi Mel,
Quoted 4 lines · Mel Gorman
The danger is that this isn't a PPC or s390 bug then as such, but a bug where there are either memoryless nodes or when node 0 is memoryless. Hence, there is no guarantee that your
Kconfigoption will catch all instances where this bug triggers. Granted, the configuration is most likely a PPC machine :)
Yes, I suggested making SLQB depend on !NUMA to Nick but he didn't like that as it's known to be good on x86 NUMA configs.
Quoted 4 lines · Mel Gorman
Quoted 4 lines
Quoted 2 lines
The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/
Maybe, but that's not an excuse to push something that's known to break.
Quoted 1 line · Mel Gorman
Wow, this is from back in May! Lame.
Heh, my (lame) excuse is lack of relevant hardware.... ;-)
Quoted 4 lines · Mel Gorman
I'm against silently disabling it. Memoryless nodes are extremely rare but bugs crop up there occasionally and take a long time to catch and squash. SLQB breaking there is not going to cause widespread damage but force a fix to be developed by the people with access to the affected machines.
Hey, if someone sends me fix for the bug well before the merge window closes, that would be great! But there's no way we're adding new core kernel code that's _known_ to break peoples configs, at least not through slab.git. If disabling SLQB is not acceptable and we're unable to fix things, we'll just have to skip this release cycle.
Quoted 3 lines · Mel Gorman
Total aside, does anybody know handily if fake NUMA support allows the creation of memoryless nodes help reproducing problems like this? If I can't get a real machine, that'll be the approach I'll be trying.
That would be useful, yes.
Pekka
Quoted 11 lines · Pekka Enberg
Hi Mel,
Quoted 4 lines · Mel Gorman
The danger is that this isn't a PPC or s390 bug then as such, but a bug where there are either memoryless nodes or when node 0 is memoryless. Hence, there is no guarantee that your
Kconfigoption will catch all instances where this bug triggers. Granted, the configuration is most likely a PPC machine :)Yes, I suggested making SLQB depend on !NUMA to Nick but he didn't like that as it's known to be good on x86 NUMA configs.
Agreed.
Quoted 11 lines · Mel Gorman
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/
Maybe, but that's not an excuse to push something that's known to break.
Quoted 1 line · Mel Gorman
Wow, this is from back in May! Lame.
Heh, my (lame) excuse is lack of relevant hardware.... ;-)
I'm not blaming you. It's just ... unfortunate :/
Quoted 6 lines · Mel Gorman
Quoted 4 lines
I'm against silently disabling it. Memoryless nodes are extremely rare but bugs crop up there occasionally and take a long time to catch and squash. SLQB breaking there is not going to cause widespread damage but force a fix to be developed by the people with access to the affected machines.
Quoted 6 lines
Hey, if someone sends me fix for the bug well before the merge window closes, that would be great! But there's no way we're adding new core kernel code that's _known_ to break peoples configs, at least not through slab.git. If disabling SLQB is not acceptable and we're unable to fix things, we'll just have to skip this release cycle.
Please consider disabling it as an option in the rc3 stage or the like. With luck, I'll find a suitable machine in time and see what can be done. I just don't like the idea of x86 defaulting to one allocator and ppc defaulting to another.
Quoted 7 lines · Mel Gorman
Quoted 3 lines
Total aside, does anybody know handily if fake NUMA support allows the creation of memoryless nodes help reproducing problems like this? If I can't get a real machine, that'll be the approach I'll be trying.
That would be useful, yes.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Hi Mel,
Quoted 4 lines · Mel Gorman
Please consider disabling it as an option in the rc3 stage or the like. With luck, I'll find a suitable machine in time and see what can be done. I just don't like the idea of x86 defaulting to one allocator and ppc defaulting to another.
SLQB is _not_ the default allocator in slqb/core and won't be one for 2.6.32 even if we do manage to sneak the allocator to linus.git.
Pekka
Quoted 7 lines · Mel Gorman
Quoted 6 lines · Pekka Enberg
Quoted 4 lines · Mel Gorman
Quoted 4 lines
Quoted 2 lines
The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/
Maybe, but that's not an excuse to push something that's known to break.
Quoted 7 lines
Quoted 5 lines · Mel Gorman
Quoted 1 line
Wow, this is from back in May! Lame.
Heh, my (lame) excuse is lack of relevant hardware.... ;-)
I'm not blaming you. It's just ... unfortunate :/
Ahh... it's pretty lame of me. Sachin has been a willing tester :( I have spent quite a few hours looking at it but I never found many good leads. Much appreciated if you can make more progress on it.
Quoted 14 lines · Nick Piggin
Quoted 13 lines · Mel Gorman
Quoted 11 lines · Pekka Enberg
Quoted 4 lines · Mel Gorman
Quoted 4 lines
Quoted 2 lines
The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/
Maybe, but that's not an excuse to push something that's known to break.
Quoted 1 line · Mel Gorman
Wow, this is from back in May! Lame.
Heh, my (lame) excuse is lack of relevant hardware.... ;-)
Quoted 6 lines
Quoted 1 line
I'm not blaming you. It's just ... unfortunate :/
Ahh... it's pretty lame of me. Sachin has been a willing tester :( I have spent quite a few hours looking at it but I never found many good leads. Much appreciated if you can make more progress on it.
Nothing much so far. I've reproduced the problem based on 2.6.31 and slqb-core from Pekka's tree but not a whole pile else. I don't know SLQB at all so the investigation is fuzzy. It appears to initialise SLQB ok but crashes later when setting up SCSI. Not 100% sure what the triggering event is but it might be userspace starting up and other CPUs get involved, possibly corrupting lists.
This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to kmem_cache_create, I see in the console
MEL::Creating cache pgd_cache CPU 0 Node 0 MEL::Creating cache pmd_cache CPU 0 Node 0 MEL::Creating cache pid_namespace CPU 0 Node 0 MEL::Creating cache shmem_inode_cache CPU 0 Node 0 MEL::Creating cache scsi_data_buffer CPU 1 Node 0
It crashes at this point during creation before the struct kmem_cache has been allocated from kmem_cache_cache. Note it's kmem_cache_cache we are failing to allocate from, not scsi_data_buffer.
I have no theories yet but will stick with it. Any suggestions on where to investigate are welcome. Will pick this up again tomorrow.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 20 lines · Mel Gorman
Quoted 4 lines
Ahh... it's pretty lame of me. Sachin has been a willing tester :( I have spent quite a few hours looking at it but I never found many good leads. Much appreciated if you can make more progress on it.
Nothing much so far. I've reproduced the problem based on 2.6.31 and slqb-core from Pekka's tree but not a whole pile else. I don't know SLQB at all so the investigation is fuzzy. It appears to initialise SLQB ok but crashes later when setting up SCSI. Not 100% sure what the triggering event is but it might be userspace starting up and other CPUs get involved, possibly corrupting lists.
This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to
kmem_cache_create, I see in the console
MEL::Creatingcachepgd_cacheCPU 0 Node 0MEL::Creatingcachepmd_cacheCPU 0 Node 0MEL::Creatingcachepid_namespaceCPU 0 Node 0MEL::Creatingcacheshmem_inode_cacheCPU 0 Node 0MEL::Creatingcachescsi_data_bufferCPU 1 Node 0
Quoted 3 lines
It crashes at this point during creation before the struct
kmem_cachehas been allocated fromkmem_cache_cache. Note it'skmem_cache_cachewe are failing to allocate from, notscsi_data_buffer.
Yes, it's crashing in kmem_cache_create, when trying to allocate from kmem_cache_cache.
I didn't get much further. I had thought something must be NULL or not set up correctly in kmem_cache_cache, but I didn't work out what.
If you can identify the precondition which cases the crash (or even just have a static counter of the number of caches created, to trigger at the crashing cache create), then perhaps you can dump some more details of the kmem_cache_cache.
Thanks,
Nick
Quoted 8 lines · Nick Piggin
Quoted 8 lines
This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to
kmem_cache_create, I see in the console
MEL::Creatingcachepgd_cacheCPU 0 Node 0MEL::Creatingcachepmd_cacheCPU 0 Node 0MEL::Creatingcachepid_namespaceCPU 0 Node 0MEL::Creatingcacheshmem_inode_cacheCPU 0 Node 0MEL::Creatingcachescsi_data_bufferCPU 1 Node 0
So we have two nodes 2,3 nothing on node 0 and we are creating caches on the node that does not exist? SLQB assumes node 0 is present?
Quoted 5 lines · Christoph Lameter
Quoted 3 lines · Nick Piggin
Quoted 3 lines
This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to
kmem_cache_create, I see in the console
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
MEL::Creatingcachepgd_cacheCPU 0 Node 0MEL::Creatingcachepmd_cacheCPU 0 Node 0MEL::Creatingcachepid_namespaceCPU 0 Node 0MEL::Creatingcacheshmem_inode_cacheCPU 0 Node 0MEL::Creatingcachescsi_data_bufferCPU 1 Node 0So we have two nodes 2,3 nothing on node 0 and we are creating caches on the node that does not exist? SLQB assumes node 0 is present?
It might do somewhere, but it shouldn't. It will ask for node 0 by default presumably if CPU0 is on node 0, but it should be able to fall back...
Quoted 25 lines · Nick Piggin
Quoted 20 lines · Mel Gorman
Quoted 4 lines
Ahh... it's pretty lame of me. Sachin has been a willing tester :( I have spent quite a few hours looking at it but I never found many good leads. Much appreciated if you can make more progress on it.
Nothing much so far. I've reproduced the problem based on 2.6.31 and slqb-core from Pekka's tree but not a whole pile else. I don't know SLQB at all so the investigation is fuzzy. It appears to initialise SLQB ok but crashes later when setting up SCSI. Not 100% sure what the triggering event is but it might be userspace starting up and other CPUs get involved, possibly corrupting lists.
This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to
kmem_cache_create, I see in the console
MEL::Creatingcachepgd_cacheCPU 0 Node 0MEL::Creatingcachepmd_cacheCPU 0 Node 0MEL::Creatingcachepid_namespaceCPU 0 Node 0MEL::Creatingcacheshmem_inode_cacheCPU 0 Node 0MEL::Creatingcachescsi_data_bufferCPU 1 Node 0Quoted 3 lines
It crashes at this point during creation before the struct
kmem_cachehas been allocated fromkmem_cache_cache. Note it'skmem_cache_cachewe are failing to allocate from, notscsi_data_buffer.
Quoted 6 lines
Yes, it's crashing in
kmem_cache_create, when trying to allocate fromkmem_cache_cache.I didn't get much further. I had thought something must be NULL or not set up correctly in
kmem_cache_cache, but I didn't work out what.
Somehow it's getting scrambled but I couldn't see anything wrong with the locking. Weirdly, the following patch allows the kernel to boot much further. Is it possible the DEFINE_PER_CPU() trick for defining per-node data is being busted by recent per-cpu changes? Sorry that this is pretty crude, it's my first proper reading of SLQB so it's slow going.
Although booting gets further with the following patch, it quickly hits an OOM-storm until the machine dies with the vast majority of pages being allocated by the slab allocator. There must be some flaw in the node fallback logic that means that pages are allocated for every slab allocation.
mm/slqb.c +8 -8
@@ -1944,16 +1944,16 @@ static void init_kmem_cache_node(struct kmem_cache *s, static DEFINE_PER_CPU(struct kmem_cache_cpu, kmem_cache_cpus);#endif #ifdef CONFIG_NUMA -/* XXX: really need a DEFINE_PER_NODE for per-node data, but this is better than -* a static array */ -static DEFINE_PER_CPU(struct kmem_cache_node, kmem_cache_nodes); +/* XXX: really need a DEFINE_PER_NODE for per-node data because a static +* array is wasteful */ +static struct kmem_cache_node kmem_cache_nodes[MAX_NUMNODES ];#endif #ifdef CONFIG_SMP static struct kmem_cache kmem_cpu_cache; static DEFINE_PER_CPU(struct kmem_cache_cpu, kmem_cpu_cpus);#ifdef CONFIG_NUMA -static DEFINE_PER_CPU(struct kmem_cache_node, kmem_cpu_nodes); /* XXX per-nid */ +static struct kmem_cache_node kmem_cpu_nodes[MAX_NUMNODES ];/* XXX per-nid */ #endif #endif
@@ -1962,7 +1962,7 @@ static struct kmem_cache kmem_node_cache;#ifdef CONFIG_SMP static DEFINE_PER_CPU(struct kmem_cache_cpu, kmem_node_cpus);#endif -static DEFINE_PER_CPU(struct kmem_cache_node, kmem_node_nodes); /*XXX per-nid */ +static struct kmem_cache_node kmem_node_nodes[MAX_NUMNODES ];/*XXX per-nid */ #endif #ifdef CONFIG_SMP
@@ -2918,15 +2918,15 @@ void __init kmem_cache_init(void)for_each_node_state (i,N_NORMAL_MEMORY ) {struct kmem_cache_node *n; - n = &per_cpu (kmem_cache_nodes, i); + n = &kmem_cache_nodes[i];init_kmem_cache_node (&kmem_cache_cache, n); kmem_cache_cache.node_slab[i] = n;#ifdef CONFIG_SMP - n = &per_cpu (kmem_cpu_nodes, i); + n = &kmem_cpu_nodes[i];init_kmem_cache_node (&kmem_cpu_cache, n); kmem_cpu_cache.node_slab[i] = n;#endif - n = &per_cpu (kmem_node_nodes, i); + n = &kmem_node_nodes[i];init_kmem_cache_node (&kmem_node_cache, n); kmem_node_cache.node_slab[i] = n; }
Quoted 9 lines · Mel Gorman
I have access to a ppc machine but not necessarily one with a memoryless nodes that can reproduce this problem.
Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?
Mel,
Have sent you the access details for the machine. This bug showed up when SLQB was enabled as default in linux-next
Thanks
-Sachin
Quoted 11 lines
Total aside, does anybody know handily if fake NUMA support allows the creation of memoryless nodes help reproducing problems like this? If I can't get a real machine, that'll be the approach I'll be trying.
Quoted 4 lines
Btw, the code is in slqb/core branch of slab.git in case someone wants to take a stab at fixing the bug.
Signature · 6 lines
--------------------------------- Sachin Sant IBM Linux Technology Center India Systems and Technology Labs Bangalore, India ---------------------------------
Quoted 11 lines · Sachin Sant
Quoted 10 lines · Mel Gorman
I have access to a ppc machine but not necessarily one with a memoryless nodes
that can reproduce this problem.Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?
Quoted 4 lines
Mel,
Have sent you the access details for the machine. This bug showed up when SLQB was enabled as default in linux-next
Maybe it will be better to hold off merging until this is debugged then? If it is merged as a non-default option, then I can't see it being a huge issue to merge it a bit after the window? It doesn't touch anything else...
Hi Nick,
Quoted 9 lines · Mel Gorman
Quoted 9 lines
Quoted 9 lines
I have access to a ppc machine but not necessarily one with a memoryless nodes
that can reproduce this problem.Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?
Quoted 2 lines · Sachin Sant
Quoted 2 lines
Have sent you the access details for the machine. This bug showed up when SLQB was enabled as default in linux-next
Quoted 4 lines · Nick Piggin
Maybe it will be better to hold off merging until this is debugged then? If it is merged as a non-default option, then I can't see it being a huge issue to merge it a bit after the window? It doesn't touch anything else...
Me sending anything but bug fixes to Linus after the merge window is closed...? That's one scary thought!
But yeah, I can hold the pull request until the issue is resolved. If Linus doesn't want to merge SLQB for 2.6.32, we'll just try again for 2.6.33.
Pekka
Quoted 17 lines · Pekka Enberg
Hi Nick,
Quoted 9 lines · Mel Gorman
Quoted 9 lines
Quoted 9 lines
I have access to a ppc machine but not necessarily one with a memoryless nodes
that can reproduce this problem.Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?
Quoted 2 lines · Sachin Sant
Quoted 2 lines
Have sent you the access details for the machine. This bug showed up when SLQB was enabled as default in linux-next
Quoted 8 lines
Quoted 4 lines · Nick Piggin
Maybe it will be better to hold off merging until this is debugged then? If it is merged as a non-default option, then I can't see it being a huge issue to merge it a bit after the window? It doesn't touch anything else...
Me sending anything but bug fixes to Linus after the merge window is closed...? That's one scary thought!
Well, I don't know if it is much different than merging with known bugs and expecting to resolve them. Maybe it circumvents the _letter_ of the merge window law, but in spirit it is probably nicer to merge after fixing the bug :)
But I'll let you decide how to proceed.
Quoted 9 lines · Nick Piggin
Quoted 2 lines
Me sending anything but bug fixes to Linus after the merge window is closed...? That's one scary thought!
Well, I don't know if it is much different than merging with known bugs and expecting to resolve them. Maybe it circumvents the _letter_ of the merge window law, but in spirit it is probably nicer to merge after fixing the bug :)
But I'll let you decide how to proceed.
I'll just wait for a fix to appear. Saves me from sleepless nights while worrying for Linus' revenge.
Pekka
Quoted 12 lines · Mel Gorman
Quoted 11 lines · Pekka Enberg
Quoted 5 lines · Pekka Enberg
Quoted 3 lines
The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in
Kconfigfor those architectures until the issues are resolved.Can the issues be summarised?
It's a boot time crash during module load:
Quoted 7 lines
Quoted 2 lines
AFAICT, it's related to a memoryless node 0. Nick suggested it could be a latent bug in the kernel that's triggered by SLQB.
The danger is that this isn't a PPC or s390 bug then as such, but a bug where there are either memoryless nodes or when node 0 is memoryless. Hence, there is no guarantee that your
Kconfigoption will catch all instances where this bug triggers. Granted, the configuration is most likely a PPC machine :)
Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in
git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.git slqb/core
and tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...
Quoted 10 lines · Heiko Carstens
Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in
git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.gitslqb/coreand tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...
IIRC, this is the only bug fix that was merged after your report:
Pekka
Quoted 7 lines · Pekka Enberg
Quoted 6 lines · Heiko Carstens
Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in
git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.gitslqb/core
Quoted 8 lines
Quoted 4 lines
and tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...IIRC, this is the only bug fix that was merged after your report:
Still works even if I revert that patch. Probably something else interfered.
Quoted 13 lines · Heiko Carstens
Quoted 12 lines · Pekka Enberg
Quoted 10 lines · Heiko Carstens
Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in
git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.gitslqb/coreand tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...
Quoted 5 lines
Quoted 3 lines
IIRC, this is the only bug fix that was merged after your report:
Still works even if I revert that patch. Probably something else interfered.
Nasty. I don't suppose it would be too much work to try bisecting it? Unfortunately with this kind of thing, bisecting doesn't always point to anything meaningful anyway :(
Quoted 16 lines · Nick Piggin
Quoted 15 lines · Heiko Carstens
Quoted 14 lines · Pekka Enberg
Quoted 10 lines · Heiko Carstens
Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in
git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.gitslqb/coreand tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...IIRC, this is the only bug fix that was merged after your report:
Quoted 7 lines
Quoted 3 lines
Still works even if I revert that patch. Probably something else interfered.
Nasty. I don't suppose it would be too much work to try bisecting it? Unfortunately with this kind of thing, bisecting doesn't always point to anything meaningful anyway :(
Bisecting won't work here because I rebased the branch to fix merge conflicts and move the Kconfig and Makefile changes at the end of the series to make sure SLQB won't break "git bisect". Blah, I guess I shouldn't have do that. :-(
Pekka