Skip to contentSkip to conversation

"random: make /dev/urandom scalable for silly userspace programs" causes crash

BUG -next Heiko Carstens 4 participants 7 messages Jul 27, 2016 – Jul 28, 2016

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hi Ted,

it looks like your patch "random: make /dev/urandom scalable for silly userspace programs" within linux-next seems to be a bit broken:

It causes this allocation failure and subsequent crash on s390 with fake NUMA enabled:

[    0.533195] SLUB: Unable to allocate memory on node 1, gfp=0x24008c0(GFP_KERNEL|__GFP_NOFAIL)
[    0.533198]   cache: kmalloc-192, object size: 192, buffer size: 528, defaul order: 3, min order: 0
[    0.533202]   node 0: slabs: 2, objs: 124, free: 17
[    0.533208] Unable to handle kernel pointer dereference in virtual kernel address space
[    0.533211] Failing address: 0000000000000000 TEID: 0000000000000483
...
[    0.533276] Krnl PSW : 0704e00180000000 00000000001a853e (lockdep_init_map+0x1e/0x220)
[    0.533281]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
               Krnl GPRS: 0000000000a23400 00000000370c8008 0000000000000060 0000000000bedc90
[    0.533285]            0000000002070800 0000000000000000 0000000000000001 0000000000000000
[    0.533287]            000000003743d3f8 000000003743d408 0000000002070800 0000000000bedc90
[    0.533289]            0000000000000048 00000000009c2030 00000000370cfd00 00000000370cfcc0
[    0.533295] Krnl Code: 00000000001a852e: a7840001            brc     8,1a8530
           00000000001a8532: e3f0ffc0ff71       lay     %r15,-64(%r15)
          #00000000001a8538: e3e0f0980024       stg     %r14,152(%r15)
          >00000000001a853e: e54820080000       mvghi   8(%r2),0
           00000000001a8544: e54820100000       mvghi   16(%r2),0
           00000000001a854a: 58100370           l       %r1,880
           00000000001a854e: 50102020           st      %r1,32(%r2)
           00000000001a8552: b90400c2           lgr     %r12,%r2
[    0.533313] Call Trace:
[    0.533315] ([<0000000000000001>] 0x1)
[    0.533318] ([<00000000001b4220>] __raw_spin_lock_init+0x50/0x80)
[    0.533320] ([<0000000000759e7a>] rand_initialize+0xc2/0xf0)
[    0.533322] ([<00000000001002cc>] do_one_initcall+0xb4/0x140)
[    0.533325] ([<0000000000ef2cc0>] kernel_init_freeable+0x140/0x2d8)
[    0.533328] ([<00000000009b07ea>] kernel_init+0x2a/0x150)
[    0.533330] ([<00000000009bd782>] kernel_thread_starter+0x6/0xc)
[    0.533332] ([<00000000009bd77c>] kernel_thread_starter+0x0/0xc)

To me it looks rand_initialize is broken with CONFIG_NUMA:

static int rand_initialize(void)
{

#ifdef CONFIG_NUMA
	int i;
	int num_nodes = num_possible_nodes();
	struct crng_state *crng;
	struct crng_state **pool;

#endif

	init_std_data(&input_pool);
	init_std_data(&blocking_pool);
	crng_initialize(&primary_crng);
#ifdef CONFIG_NUMA
	pool = kmalloc(num_nodes * sizeof(void *),
		       GFP_KERNEL|__GFP_NOFAIL|__GFP_ZERO);
	for (i=0; i < num_nodes; i++) {
		crng = kmalloc_node(sizeof(struct crng_state),
				    GFP_KERNEL | __GFP_NOFAIL, i);
		spin_lock_init(&crng->lock);
		crng_initialize(crng);
		pool[i] = crng;

	}
	mb();
	crng_node_pool = pool;
#endif
	return 0;
}

early_initcall(rand_initialize);

First the for loop should use for_each_node() to skip not possible nodes, no?

However that wouldn't be enough, since in this case it crashed because node 1 is in the possible map, but it isn't online and doesn't have any memory, which explains why the allocation fails and the subsequent crash when calling spin_lock_init().

I think the proper fix would be to simply use for_each_online_node(); at least that fixes the crash on s390.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#1 lore raw
Quoted 5 lines · Heiko Carstens

it looks like your patch "random: make /dev/urandom scalable for silly userspace programs" within linux-next seems to be a bit broken:

It causes this allocation failure and subsequent crash on s390 with fake NUMA enabled

Thanks for reporting this. This patch fixes things for you, yes?

- Ted

commit 59b8d4f1f5d26e4ca92172ff6dcd1492cdb39613

Author: Theodore Ts'o <[email protected]>
Date:   Wed Jul 27 23:30:25 2016 -0400

random: use for_each_online_node() to iterate over NUMA nodes

This fixes a crash on s390 with fake NUMA enabled.

Fixes 1e7f583af67b ("random: make /dev/urandom scalable for silly userspace programs")
drivers/char/random.c +1 -2
index 8d0af74..7f06224 100644
@@ -1668,13 +1668,12 @@ static int rand_initialize(void)
 #ifdef CONFIG_NUMA
 	pool = kmalloc(num_nodes * sizeof(void *),
 		       GFP_KERNEL|__GFP_NOFAIL|__GFP_ZERO);
-	for (i=0; i < num_nodes; i++) {
+	for_each_online_node(i) {
 		crng = kmalloc_node(sizeof(struct crng_state),
 				    GFP_KERNEL | __GFP_NOFAIL, i);
 		spin_lock_init(&crng->lock);
 		crng_initialize(crng);
 		pool[i] = crng;
-
 	}
 	mb();
 	crng_node_pool = pool;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
Quoted 10 lines · Theodore Ts'o
Quoted 5 lines · Heiko Carstens

it looks like your patch "random: make /dev/urandom scalable for silly userspace programs" within linux-next seems to be a bit broken:

It causes this allocation failure and subsequent crash on s390 with fake NUMA enabled

Thanks for reporting this. This patch fixes things for you, yes?

- Ted

Yes, it does. It's actually the same what I did to fix this ;)

Quoted 32 lines

commit 59b8d4f1f5d26e4ca92172ff6dcd1492cdb39613

Author: Theodore Ts'o <[email protected]>
Date:   Wed Jul 27 23:30:25 2016 -0400

random: use for_each_online_node() to iterate over NUMA nodes

This fixes a crash on s390 with fake NUMA enabled.

Fixes 1e7f583af67b ("random: make /dev/urandom scalable for silly userspace programs")
drivers/char/random.c +1 -2
index 8d0af74..7f06224 100644
@@ -1668,13 +1668,12 @@ static int rand_initialize(void)
 #ifdef CONFIG_NUMA
 	pool = kmalloc(num_nodes * sizeof(void *),
 		       GFP_KERNEL|__GFP_NOFAIL|__GFP_ZERO);
-	for (i=0; i < num_nodes; i++) {
+	for_each_online_node(i) {
 		crng = kmalloc_node(sizeof(struct crng_state),
 				    GFP_KERNEL | __GFP_NOFAIL, i);
 		spin_lock_init(&crng->lock);
 		crng_initialize(crng);
 		pool[i] = crng;
-
 	}
 	mb();
 	crng_node_pool = pool;
#3 lore raw
Quoted 5 lines · Heiko Carstens
Quoted 4 lines · Theodore Ts'o
Quoted 3 lines · Heiko Carstens

it looks like your patch "random: make /dev/urandom scalable for silly userspace programs" within linux-next seems to be a bit broken:

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

It causes this allocation failure and subsequent crash on s390 with fake NUMA enabled

Thanks for reporting this. This patch fixes things for you, yes?

- Ted

Yes, it does. It's actually the same what I did to fix this ;)

Oh, I just realized that Linus pulled your changes. Actually I was hoping we could get this fixed before the broken code would be merged. Could you please make sure the bug fix gets included as soon as possible?

Right now booting a kernel with any defconfig on s390 will crash because of this.

I will also change the fake NUMA code on s390, since it doesn't make sense to have possible but not online nodes (in this case).

Quoted 33 lines
Quoted 32 lines

commit 59b8d4f1f5d26e4ca92172ff6dcd1492cdb39613

Author: Theodore Ts'o <[email protected]>
Date:   Wed Jul 27 23:30:25 2016 -0400

random: use for_each_online_node() to iterate over NUMA nodes

This fixes a crash on s390 with fake NUMA enabled.

Fixes 1e7f583af67b ("random: make /dev/urandom scalable for silly userspace programs")
drivers/char/random.c +1 -2
index 8d0af74..7f06224 100644
@@ -1668,13 +1668,12 @@ static int rand_initialize(void)
 #ifdef CONFIG_NUMA
 	pool = kmalloc(num_nodes * sizeof(void *),
 		       GFP_KERNEL|__GFP_NOFAIL|__GFP_ZERO);
-	for (i=0; i < num_nodes; i++) {
+	for_each_online_node(i) {
 		crng = kmalloc_node(sizeof(struct crng_state),
 				    GFP_KERNEL | __GFP_NOFAIL, i);
 		spin_lock_init(&crng->lock);
 		crng_initialize(crng);
 		pool[i] = crng;
-
 	}
 	mb();
 	crng_node_pool = pool;
maintainer
#4 lore raw
Quoted 4 lines · Heiko Carstens

Oh, I just realized that Linus pulled your changes. Actually I was hoping we could get this fixed before the broken code would be merged. Could you please make sure the bug fix gets included as soon as possible?

Yes, I'll send the pull request to ASAP.

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
In reply to Theodore Ts'o
Quoted 8 lines · Theodore Ts'o
Quoted 6 lines · Heiko Carstens

it looks like your patch "random: make /dev/urandom scalable for silly userspace programs" within linux-next seems to be a bit broken:

It causes this allocation failure and subsequent crash on s390 with fake NUMA enabled

Thanks for reporting this.  This patch fixes things for you, yes?

trivia:

drivers/char/random.c

Quoted 1 line
drivers/char/random.c +0 -0

[]

drivers/char/random.c static int rand_initialize(void) · line 1670

Quoted 4 lines
drivers/char/random.c +0 -0
@@ -1668,13 +1668,12 @@ static int rand_initialize(void)
  #ifdef CONFIG_NUMA
  	pool = kmalloc(num_nodes * sizeof(void *),
  		       GFP_KERNEL|__GFP_NOFAIL|__GFP_ZERO);

The __GFP_ZERO is unusual and this could use kcalloc instead.

drivers/char/random.c static int rand_initialize(void) · line 1679

Quoted 11 lines
drivers/char/random.c +1 -2
-	for (i=0; i < num_nodes; i++) {
+	for_each_online_node(i) {
  		crng = kmalloc_node(sizeof(struct crng_state),
  				    GFP_KERNEL | __GFP_NOFAIL, i);
  		spin_lock_init(&crng->lock);
  		crng_initialize(crng);
  		pool[i] = crng;
-
  	}
  	mb();
  	crng_node_pool = pool;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Tested-by
#5 lore raw
In reply to Theodore Ts'o
Quoted 7 lines · Theodore Ts'o
Quoted 4 lines · Heiko Carstens

Oh, I just realized that Linus pulled your changes. Actually I was hoping we could get this fixed before the broken code would be merged. Could you please make sure the bug fix gets included as soon as possible?

Yes, I'll send the pull request to ASAP.

Also broke ia64. Same fix works for me.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>