Skip to contentSkip to conversation

SLQB: Mark the allocator as broken PowerPC and S390

RFC/PATCH Pekka Enberg 7 participants 26 messages Sep 16, 2009 – Sep 18, 2009 Memory management

lore
Share & tools
lore
#0 lore raw

The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in Kconfig for those architectures until the issues are resolved.

Cc Andrew Morton <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Nick Piggin <[email protected]>
Cc Sachin Sant <[email protected]>
---
 init/Kconfig |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
init/Kconfig +1 -0
index c0d8a47..aaeddeb 100644
@@ -1033,6 +1033,7 @@ config SLUB
 
 config SLQB
 	bool "SLQB (Queued allocator)"
+	depends on !PPC && !S390
 	help
 	  SLQB is a proposed new slab allocator.
 
Signature · 1 line
1.5.6.3
#1 lore raw
Quoted 20 lines · Pekka Enberg

The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in Kconfig for those architectures until the issues are resolved.

Cc Andrew Morton <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Nick Piggin <[email protected]>
Cc Sachin Sant <[email protected]>
---
 init/Kconfig |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
init/Kconfig +0 -0
index c0d8a47..aaeddeb 100644

init/Kconfig config SLUB · line 1039

Quoted 7 lines
init/Kconfig +1 -0
@@ -1033,6 +1033,7 @@ config SLUB
 
 config SLQB
 	bool "SLQB (Queued allocator)"
+	depends on !PPC && !S390
 	help
 	  SLQB is a proposed new slab allocator.
 

I think this should be (!PPC && !S390) || EXPERIMENTAL so that it can still be enabled for debugging and development.

Is this in preparation for slqb inclusion as a non-default slab allocator in 2.6.32? 2.6.33?

#2 lore raw

Hi David,

Quoted 27 lines · Pekka Enberg
Quoted 27 lines

The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in Kconfig for those architectures until the issues are resolved.

Cc Andrew Morton <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Nick Piggin <[email protected]>
Cc Sachin Sant <[email protected]>
---
 init/Kconfig |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
init/Kconfig +1 -0
index c0d8a47..aaeddeb 100644
@@ -1033,6 +1033,7 @@ config SLUB
 
 config SLQB
 	bool "SLQB (Queued allocator)"
+	depends on !PPC && !S390
 	help
 	  SLQB is a proposed new slab allocator.
 
Quoted 2 lines · David Rientjes

I think this should be (!PPC && !S390) || EXPERIMENTAL so that it can still be enabled for debugging and development.

Everybody enables EXPERIMENTAL so that seems pointless. Developers can hack Kconfig locally, no?

Quoted 2 lines · David Rientjes

Is this in preparation for slqb inclusion as a non-default slab allocator in 2.6.32? 2.6.33?

Non-default for 2.6.32.

Pekka

#3 lore raw
Quoted 4 lines · Pekka Enberg
Quoted 3 lines · Pekka Enberg
Quoted 3 lines

The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in Kconfig for those architectures until the issues are resolved.

Looks ok to me. I'll debug s390 when time permits.

Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
 init/Kconfig |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
init/Kconfig +0 -0
index c0d8a47..aaeddeb 100644
@@ -1033,6 +1033,7 @@ config SLUB
 
Quoted 6 lines

init/Kconfig config SLUB · line 1039

Quoted 6 lines
init/Kconfig +1 -0
 config SLQB
 	bool "SLQB (Queued allocator)"
+	depends on !PPC && !S390
 	help
 	  SLQB is a proposed new slab allocator.
 

On Tue, 2009-09-15 at 23:55 -0700, David Rientjes wrote:

Quoted 5 lines
Quoted 2 lines

I think this should be (!PPC && !S390) || EXPERIMENTAL so that it can still be enabled for debugging and development.

Everybody enables EXPERIMENTAL so that seems pointless. Developers can hack Kconfig locally, no?

Exactly.

#4 lore raw
In reply to Pekka Enberg
Quoted 4 lines · Pekka Enberg

The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in Kconfig for those architectures until the issues are resolved.

Can the issues be summarised?

The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/

Quoted 34 lines
Cc Andrew Morton <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Nick Piggin <[email protected]>
Cc Sachin Sant <[email protected]>
---
 init/Kconfig |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
init/Kconfig +1 -0
index c0d8a47..aaeddeb 100644
@@ -1033,6 +1033,7 @@ config SLUB
 
 config SLQB
 	bool "SLQB (Queued allocator)"
+	depends on !PPC && !S390
 	help
 	  SLQB is a proposed new slab allocator.
 
Signature · 9 lines
1.5.6.3



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#5 lore raw

Hi Mel,

Quoted 5 lines · Pekka Enberg
Quoted 3 lines

The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in Kconfig for those architectures until the issues are resolved.

Can the issues be summarised?

It's a boot time crash during module load:

http://www.mail-archive.com/[email protected]/msg33092.html

AFAICT, it's related to a memoryless node 0. Nick suggested it could be a latent bug in the kernel that's triggered by SLQB.

Quoted 2 lines · Mel Gorman

The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/

Maybe, but that's not an excuse to push something that's known to break.

The other alternative is to skip this release cycle but I'm not sure what we'd gain with that. Nick already stated in private that he'll try to arrange for some time with ppc machines to debug the thing and we hope to be able to fix it by 2.6.32 final.

Btw, the code is in slqb/core branch of slab.git in case someone wants to take a stab at fixing the bug.

Pekka

#6 lore raw
Quoted 7 lines · Pekka Enberg

Hi Mel,

Quoted 4 lines · Pekka Enberg
Quoted 3 lines

The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in Kconfig for those architectures until the issues are resolved.

Quoted 9 lines
Quoted 1 line

Can the issues be summarised?

It's a boot time crash during module load:

http://www.mail-archive.com/[email protected]/msg33092.html

AFAICT, it's related to a memoryless node 0. Nick suggested it could be a latent bug in the kernel that's triggered by SLQB.

The danger is that this isn't a PPC or s390 bug then as such, but a bug where there are either memoryless nodes or when node 0 is memoryless. Hence, there is no guarantee that your Kconfig option will catch all instances where this bug triggers. Granted, the configuration is most likely a PPC machine :)

Quoted 6 lines · Mel Gorman
Quoted 2 lines

The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/

Maybe, but that's not an excuse to push something that's known to break.

Wow, this is from back in May! Lame.

I'm against silently disabling it. Memoryless nodes are extremely rare but bugs crop up there occasionally and take a long time to catch and squash. SLQB breaking there is not going to cause widespread damage but force a fix to be developed by the people with access to the affected machines.

Quoted 5 lines

The other alternative is to skip this release cycle but I'm not sure what we'd gain with that. Nick already stated in private that he'll try to arrange for some time with ppc machines to debug the thing and we hope to be able to fix it by 2.6.32 final.

I have access to a ppc machine but not necessarily one with a memoryless nodes that can reproduce this problem.

Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?

Total aside, does anybody know handily if fake NUMA support allows the creation of memoryless nodes help reproducing problems like this? If I can't get a real machine, that'll be the approach I'll be trying.

Quoted 3 lines

Btw, the code is in slqb/core branch of slab.git in case someone wants to take a stab at fixing the bug.

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#7 lore raw

Hi Mel,

Quoted 4 lines · Mel Gorman

The danger is that this isn't a PPC or s390 bug then as such, but a bug where there are either memoryless nodes or when node 0 is memoryless. Hence, there is no guarantee that your Kconfig option will catch all instances where this bug triggers. Granted, the configuration is most likely a PPC machine :)

Yes, I suggested making SLQB depend on !NUMA to Nick but he didn't like that as it's known to be good on x86 NUMA configs.

Quoted 4 lines · Mel Gorman
Quoted 4 lines
Quoted 2 lines

The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/

Maybe, but that's not an excuse to push something that's known to break.

Quoted 1 line · Mel Gorman

Wow, this is from back in May! Lame.

Heh, my (lame) excuse is lack of relevant hardware.... ;-)

Quoted 4 lines · Mel Gorman

I'm against silently disabling it. Memoryless nodes are extremely rare but bugs crop up there occasionally and take a long time to catch and squash. SLQB breaking there is not going to cause widespread damage but force a fix to be developed by the people with access to the affected machines.

Hey, if someone sends me fix for the bug well before the merge window closes, that would be great! But there's no way we're adding new core kernel code that's _known_ to break peoples configs, at least not through slab.git. If disabling SLQB is not acceptable and we're unable to fix things, we'll just have to skip this release cycle.

Quoted 3 lines · Mel Gorman

Total aside, does anybody know handily if fake NUMA support allows the creation of memoryless nodes help reproducing problems like this? If I can't get a real machine, that'll be the approach I'll be trying.

That would be useful, yes.

Pekka

#8 lore raw
Quoted 11 lines · Pekka Enberg

Hi Mel,

Quoted 4 lines · Mel Gorman

The danger is that this isn't a PPC or s390 bug then as such, but a bug where there are either memoryless nodes or when node 0 is memoryless. Hence, there is no guarantee that your Kconfig option will catch all instances where this bug triggers. Granted, the configuration is most likely a PPC machine :)

Yes, I suggested making SLQB depend on !NUMA to Nick but he didn't like that as it's known to be good on x86 NUMA configs.

Agreed.

Quoted 11 lines · Mel Gorman
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/

Maybe, but that's not an excuse to push something that's known to break.

Quoted 1 line · Mel Gorman

Wow, this is from back in May! Lame.

Heh, my (lame) excuse is lack of relevant hardware.... ;-)

I'm not blaming you. It's just ... unfortunate :/

Quoted 6 lines · Mel Gorman
Quoted 4 lines

I'm against silently disabling it. Memoryless nodes are extremely rare but bugs crop up there occasionally and take a long time to catch and squash. SLQB breaking there is not going to cause widespread damage but force a fix to be developed by the people with access to the affected machines.

Quoted 6 lines

Hey, if someone sends me fix for the bug well before the merge window closes, that would be great! But there's no way we're adding new core kernel code that's _known_ to break peoples configs, at least not through slab.git. If disabling SLQB is not acceptable and we're unable to fix things, we'll just have to skip this release cycle.

Please consider disabling it as an option in the rc3 stage or the like. With luck, I'll find a suitable machine in time and see what can be done. I just don't like the idea of x86 defaulting to one allocator and ppc defaulting to another.

Quoted 7 lines · Mel Gorman
Quoted 3 lines

Total aside, does anybody know handily if fake NUMA support allows the creation of memoryless nodes help reproducing problems like this? If I can't get a real machine, that'll be the approach I'll be trying.

That would be useful, yes.

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#9 lore raw

Hi Mel,

Quoted 4 lines · Mel Gorman

Please consider disabling it as an option in the rc3 stage or the like. With luck, I'll find a suitable machine in time and see what can be done. I just don't like the idea of x86 defaulting to one allocator and ppc defaulting to another.

SLQB is _not_ the default allocator in slqb/core and won't be one for 2.6.32 even if we do manage to sneak the allocator to linus.git.

Pekka

#10 lore raw
In reply to Mel Gorman
Quoted 7 lines · Mel Gorman
Quoted 6 lines · Pekka Enberg
Quoted 4 lines · Mel Gorman
Quoted 4 lines
Quoted 2 lines

The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/

Maybe, but that's not an excuse to push something that's known to break.

Quoted 7 lines
Quoted 5 lines · Mel Gorman
Quoted 1 line

Wow, this is from back in May! Lame.

Heh, my (lame) excuse is lack of relevant hardware.... ;-)

I'm not blaming you. It's just ... unfortunate :/

Ahh... it's pretty lame of me. Sachin has been a willing tester :( I have spent quite a few hours looking at it but I never found many good leads. Much appreciated if you can make more progress on it.

#11 lore raw
Quoted 14 lines · Nick Piggin
Quoted 13 lines · Mel Gorman
Quoted 11 lines · Pekka Enberg
Quoted 4 lines · Mel Gorman
Quoted 4 lines
Quoted 2 lines

The danger is if SLQB is being silently disabled, it'll never be noticed or debugged :/

Maybe, but that's not an excuse to push something that's known to break.

Quoted 1 line · Mel Gorman

Wow, this is from back in May! Lame.

Heh, my (lame) excuse is lack of relevant hardware.... ;-)

Quoted 6 lines
Quoted 1 line

I'm not blaming you. It's just ... unfortunate :/

Ahh... it's pretty lame of me. Sachin has been a willing tester :( I have spent quite a few hours looking at it but I never found many good leads. Much appreciated if you can make more progress on it.

Nothing much so far. I've reproduced the problem based on 2.6.31 and slqb-core from Pekka's tree but not a whole pile else. I don't know SLQB at all so the investigation is fuzzy. It appears to initialise SLQB ok but crashes later when setting up SCSI. Not 100% sure what the triggering event is but it might be userspace starting up and other CPUs get involved, possibly corrupting lists.

This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to kmem_cache_create, I see in the console

MEL::Creating cache pgd_cache CPU 0 Node 0 MEL::Creating cache pmd_cache CPU 0 Node 0 MEL::Creating cache pid_namespace CPU 0 Node 0 MEL::Creating cache shmem_inode_cache CPU 0 Node 0 MEL::Creating cache scsi_data_buffer CPU 1 Node 0

It crashes at this point during creation before the struct kmem_cache has been allocated from kmem_cache_cache. Note it's kmem_cache_cache we are failing to allocate from, not scsi_data_buffer.

I have no theories yet but will stick with it. Any suggestions on where to investigate are welcome. Will pick this up again tomorrow.

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#12 lore raw
Quoted 20 lines · Mel Gorman
Quoted 4 lines

Ahh... it's pretty lame of me. Sachin has been a willing tester :( I have spent quite a few hours looking at it but I never found many good leads. Much appreciated if you can make more progress on it.

Nothing much so far. I've reproduced the problem based on 2.6.31 and slqb-core from Pekka's tree but not a whole pile else. I don't know SLQB at all so the investigation is fuzzy. It appears to initialise SLQB ok but crashes later when setting up SCSI. Not 100% sure what the triggering event is but it might be userspace starting up and other CPUs get involved, possibly corrupting lists.

This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to kmem_cache_create, I see in the console

MEL::Creating cache pgd_cache CPU 0 Node 0 MEL::Creating cache pmd_cache CPU 0 Node 0 MEL::Creating cache pid_namespace CPU 0 Node 0 MEL::Creating cache shmem_inode_cache CPU 0 Node 0 MEL::Creating cache scsi_data_buffer CPU 1 Node 0

Quoted 3 lines

It crashes at this point during creation before the struct kmem_cache has been allocated from kmem_cache_cache. Note it's kmem_cache_cache we are failing to allocate from, not scsi_data_buffer.

Yes, it's crashing in kmem_cache_create, when trying to allocate from kmem_cache_cache.

I didn't get much further. I had thought something must be NULL or not set up correctly in kmem_cache_cache, but I didn't work out what.

If you can identify the precondition which cases the crash (or even just have a static counter of the number of caches created, to trigger at the crashing cache create), then perhaps you can dump some more details of the kmem_cache_cache.

Thanks,
Nick

#13 lore raw
Quoted 8 lines · Nick Piggin
Quoted 8 lines

This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to kmem_cache_create, I see in the console

MEL::Creating cache pgd_cache CPU 0 Node 0 MEL::Creating cache pmd_cache CPU 0 Node 0 MEL::Creating cache pid_namespace CPU 0 Node 0 MEL::Creating cache shmem_inode_cache CPU 0 Node 0 MEL::Creating cache scsi_data_buffer CPU 1 Node 0

So we have two nodes 2,3 nothing on node 0 and we are creating caches on the node that does not exist? SLQB assumes node 0 is present?

#14 lore raw
Quoted 5 lines · Christoph Lameter
Quoted 3 lines · Nick Piggin
Quoted 3 lines

This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to kmem_cache_create, I see in the console

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

MEL::Creating cache pgd_cache CPU 0 Node 0 MEL::Creating cache pmd_cache CPU 0 Node 0 MEL::Creating cache pid_namespace CPU 0 Node 0 MEL::Creating cache shmem_inode_cache CPU 0 Node 0 MEL::Creating cache scsi_data_buffer CPU 1 Node 0

So we have two nodes 2,3 nothing on node 0 and we are creating caches on the node that does not exist? SLQB assumes node 0 is present?

It might do somewhere, but it shouldn't. It will ask for node 0 by default presumably if CPU0 is on node 0, but it should be able to fall back...

#15 lore raw
In reply to Nick Piggin
Quoted 25 lines · Nick Piggin
Quoted 20 lines · Mel Gorman
Quoted 4 lines

Ahh... it's pretty lame of me. Sachin has been a willing tester :( I have spent quite a few hours looking at it but I never found many good leads. Much appreciated if you can make more progress on it.

Nothing much so far. I've reproduced the problem based on 2.6.31 and slqb-core from Pekka's tree but not a whole pile else. I don't know SLQB at all so the investigation is fuzzy. It appears to initialise SLQB ok but crashes later when setting up SCSI. Not 100% sure what the triggering event is but it might be userspace starting up and other CPUs get involved, possibly corrupting lists.

This machine has two CPUs (0, 1) and two nodes with actual memory (2,3). After applying a patch to kmem_cache_create, I see in the console

MEL::Creating cache pgd_cache CPU 0 Node 0 MEL::Creating cache pmd_cache CPU 0 Node 0 MEL::Creating cache pid_namespace CPU 0 Node 0 MEL::Creating cache shmem_inode_cache CPU 0 Node 0 MEL::Creating cache scsi_data_buffer CPU 1 Node 0

Quoted 3 lines

It crashes at this point during creation before the struct kmem_cache has been allocated from kmem_cache_cache. Note it's kmem_cache_cache we are failing to allocate from, not scsi_data_buffer.

Quoted 6 lines

Yes, it's crashing in kmem_cache_create, when trying to allocate from kmem_cache_cache.

I didn't get much further. I had thought something must be NULL or not set up correctly in kmem_cache_cache, but I didn't work out what.

Somehow it's getting scrambled but I couldn't see anything wrong with the locking. Weirdly, the following patch allows the kernel to boot much further. Is it possible the DEFINE_PER_CPU() trick for defining per-node data is being busted by recent per-cpu changes? Sorry that this is pretty crude, it's my first proper reading of SLQB so it's slow going.

Although booting gets further with the following patch, it quickly hits an OOM-storm until the machine dies with the vast majority of pages being allocated by the slab allocator. There must be some flaw in the node fallback logic that means that pages are allocated for every slab allocation.

mm/slqb.c +8 -8
index 4ca85e2..4d72be2 100644
@@ -1944,16 +1944,16 @@ static void init_kmem_cache_node(struct kmem_cache *s,
 static DEFINE_PER_CPU(struct kmem_cache_cpu, kmem_cache_cpus);
 #endif
 #ifdef CONFIG_NUMA
-/* XXX: really need a DEFINE_PER_NODE for per-node data, but this is better than
- * a static array */
-static DEFINE_PER_CPU(struct kmem_cache_node, kmem_cache_nodes);
+/* XXX: really need a DEFINE_PER_NODE for per-node data because a static
+ *      array is wasteful */
+static struct kmem_cache_node kmem_cache_nodes[MAX_NUMNODES];
 #endif
 
 #ifdef CONFIG_SMP
 static struct kmem_cache kmem_cpu_cache;
 static DEFINE_PER_CPU(struct kmem_cache_cpu, kmem_cpu_cpus);
 #ifdef CONFIG_NUMA
-static DEFINE_PER_CPU(struct kmem_cache_node, kmem_cpu_nodes); /* XXX per-nid */
+static struct kmem_cache_node kmem_cpu_nodes[MAX_NUMNODES]; /* XXX per-nid */
 #endif
 #endif
 
@@ -1962,7 +1962,7 @@ static struct kmem_cache kmem_node_cache;
 #ifdef CONFIG_SMP
 static DEFINE_PER_CPU(struct kmem_cache_cpu, kmem_node_cpus);
 #endif
-static DEFINE_PER_CPU(struct kmem_cache_node, kmem_node_nodes); /*XXX per-nid */
+static struct kmem_cache_node kmem_node_nodes[MAX_NUMNODES]; /*XXX per-nid */
 #endif
 
 #ifdef CONFIG_SMP
@@ -2918,15 +2918,15 @@ void __init kmem_cache_init(void)
 	for_each_node_state(i, N_NORMAL_MEMORY) {
 		struct kmem_cache_node *n;
 
-		n = &per_cpu(kmem_cache_nodes, i);
+		n = &kmem_cache_nodes[i];
 		init_kmem_cache_node(&kmem_cache_cache, n);
 		kmem_cache_cache.node_slab[i] = n;
 #ifdef CONFIG_SMP
-		n = &per_cpu(kmem_cpu_nodes, i);
+		n = &kmem_cpu_nodes[i];
 		init_kmem_cache_node(&kmem_cpu_cache, n);
 		kmem_cpu_cache.node_slab[i] = n;
 #endif
-		n = &per_cpu(kmem_node_nodes, i);
+		n = &kmem_node_nodes[i];
 		init_kmem_cache_node(&kmem_node_cache, n);
 		kmem_node_cache.node_slab[i] = n;
 	}
#16 lore raw
In reply to Mel Gorman
Quoted 9 lines · Mel Gorman

I have access to a ppc machine but not necessarily one with a memoryless nodes that can reproduce this problem.

Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?

Mel,

Have sent you the access details for the machine. This bug showed up when SLQB was enabled as default in linux-next

Thanks
-Sachin

Quoted 11 lines

Total aside, does anybody know handily if fake NUMA support allows the creation of memoryless nodes help reproducing problems like this? If I can't get a real machine, that'll be the approach I'll be trying.

Quoted 4 lines

Btw, the code is in slqb/core branch of slab.git in case someone wants to take a stab at fixing the bug.

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
#17 lore raw
Quoted 11 lines · Sachin Sant
Quoted 10 lines · Mel Gorman

I have access to a ppc machine but not necessarily one with a memoryless nodes
that can reproduce this problem.

Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?

Quoted 4 lines

Mel,

Have sent you the access details for the machine. This bug showed up when SLQB was enabled as default in linux-next

Maybe it will be better to hold off merging until this is debugged then? If it is merged as a non-default option, then I can't see it being a huge issue to merge it a bit after the window? It doesn't touch anything else...

#18 lore raw

Hi Nick,

Quoted 9 lines · Mel Gorman
Quoted 9 lines
Quoted 9 lines

I have access to a ppc machine but not necessarily one with a memoryless nodes
that can reproduce this problem.

Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?

Quoted 2 lines · Sachin Sant
Quoted 2 lines

Have sent you the access details for the machine. This bug showed up when SLQB was enabled as default in linux-next

Quoted 4 lines · Nick Piggin

Maybe it will be better to hold off merging until this is debugged then? If it is merged as a non-default option, then I can't see it being a huge issue to merge it a bit after the window? It doesn't touch anything else...

Me sending anything but bug fixes to Linus after the merge window is closed...? That's one scary thought!

But yeah, I can hold the pull request until the issue is resolved. If Linus doesn't want to merge SLQB for 2.6.32, we'll just try again for 2.6.33.

Pekka

#19 lore raw
Quoted 17 lines · Pekka Enberg

Hi Nick,

Quoted 9 lines · Mel Gorman
Quoted 9 lines
Quoted 9 lines

I have access to a ppc machine but not necessarily one with a memoryless nodes
that can reproduce this problem.

Assuming Sachin is the reporter and we are in the same company, maybe I have access to the machine. Sachin, can you mail me privately what this machine is called and lets see can I get some time on that machine? By any chance, was this bisected or did it just show up when SLQB became the default?

Quoted 2 lines · Sachin Sant
Quoted 2 lines

Have sent you the access details for the machine. This bug showed up when SLQB was enabled as default in linux-next

Quoted 8 lines
Quoted 4 lines · Nick Piggin

Maybe it will be better to hold off merging until this is debugged then? If it is merged as a non-default option, then I can't see it being a huge issue to merge it a bit after the window? It doesn't touch anything else...

Me sending anything but bug fixes to Linus after the merge window is closed...? That's one scary thought!

Well, I don't know if it is much different than merging with known bugs and expecting to resolve them. Maybe it circumvents the _letter_ of the merge window law, but in spirit it is probably nicer to merge after fixing the bug :)

But I'll let you decide how to proceed.

#20 lore raw
Quoted 9 lines · Nick Piggin
Quoted 2 lines

Me sending anything but bug fixes to Linus after the merge window is closed...? That's one scary thought!

Well, I don't know if it is much different than merging with known bugs and expecting to resolve them. Maybe it circumvents the _letter_ of the merge window law, but in spirit it is probably nicer to merge after fixing the bug :)

But I'll let you decide how to proceed.

I'll just wait for a fix to appear. Saves me from sleepless nights while worrying for Linus' revenge.

Pekka

#21 lore raw
In reply to Mel Gorman
Quoted 12 lines · Mel Gorman
Quoted 11 lines · Pekka Enberg
Quoted 5 lines · Pekka Enberg
Quoted 3 lines

The SLQB allocator is known to be broken on certain PowerPC and S390 configurations. Disable the allocator in Kconfig for those architectures until the issues are resolved.

Can the issues be summarised?

It's a boot time crash during module load:

http://www.mail-archive.com/[email protected]/msg33092.html

Quoted 7 lines
Quoted 2 lines

AFAICT, it's related to a memoryless node 0. Nick suggested it could be a latent bug in the kernel that's triggered by SLQB.

The danger is that this isn't a PPC or s390 bug then as such, but a bug where there are either memoryless nodes or when node 0 is memoryless. Hence, there is no guarantee that your Kconfig option will catch all instances where this bug triggers. Granted, the configuration is most likely a PPC machine :)

Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in

git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.git slqb/core

and tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...

#22 lore raw
Quoted 10 lines · Heiko Carstens

Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in

git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.git slqb/core

and tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...

IIRC, this is the only bug fix that was merged after your report:

http://git.kernel.org/?p=linux/kernel/git/penberg/slab-2.6.git;a=commitdiff;h=ff61c4950125b09b5e5a83d48a6c81827e9d67ab

Pekka

#23 lore raw
Quoted 7 lines · Pekka Enberg
Quoted 6 lines · Heiko Carstens

Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in

git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.git slqb/core

Quoted 8 lines
Quoted 4 lines

and tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...

IIRC, this is the only bug fix that was merged after your report:

http://git.kernel.org/?p=linux/kernel/git/penberg/slab-2.6.git;a=commitdiff;h=ff61c4950125b09b5e5a83d48a6c81827e9d67ab

Still works even if I revert that patch. Probably something else interfered.

#24 lore raw
Quoted 13 lines · Heiko Carstens
Quoted 12 lines · Pekka Enberg
Quoted 10 lines · Heiko Carstens

Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in

git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.git slqb/core

and tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...

Quoted 5 lines
Quoted 3 lines

IIRC, this is the only bug fix that was merged after your report:

http://git.kernel.org/?p=linux/kernel/git/penberg/slab-2.6.git;a=commitdiff;h=ff61c4950125b09b5e5a83d48a6c81827e9d67ab

Still works even if I revert that patch. Probably something else interfered.

Nasty. I don't suppose it would be too much work to try bisecting it? Unfortunately with this kind of thing, bisecting doesn't always point to anything meaningful anyway :(

#25 lore raw
Quoted 16 lines · Nick Piggin
Quoted 15 lines · Heiko Carstens
Quoted 14 lines · Pekka Enberg
Quoted 10 lines · Heiko Carstens

Ok, I just wanted to debug this on s390. But... the bug seems to have disappeared.
I pulled in

git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.git slqb/core

and tried defconfig (+SLQB) as well as allyesconfig (+SLQB). Both started and didn't show the crash-before-console-is-active which went away when switching to a different allocator.
So the s390 restriction seems to be resolved. Don't know why...

IIRC, this is the only bug fix that was merged after your report:

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

http://git.kernel.org/?p=linux/kernel/git/penberg/slab-2.6.git;a=commitdiff;h=ff61c4950125b09b5e5a83d48a6c81827e9d67ab

Still works even if I revert that patch. Probably something else interfered.

Nasty. I don't suppose it would be too much work to try bisecting it? Unfortunately with this kind of thing, bisecting doesn't always point to anything meaningful anyway :(

Bisecting won't work here because I rebased the branch to fix merge conflicts and move the Kconfig and Makefile changes at the end of the series to make sure SLQB won't break "git bisect". Blah, I guess I shouldn't have do that. :-(

Pekka