kernel BUG at include/net/netns/generic.h:41!

2.6.33-rc5 New Luca Tettamanti 4 participants 9 messages Jan 29, 2010 – Feb 4, 2010

Subsystems (3)

Networking Network drivers Networking IPsec

lore

Share & tools

lore

Luca Tettamanti

author

2010-01-29 09:48 UTC #0 lore raw

Hello,
with recent kernels I'm seeing this BUG - triggered by racoon - at boot:

NET: Registered protocol family 15

------------[ cut here ]------------
kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!

invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/kernel/uevent_seqnum
CPU 1
Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa /F3Sa

RIP: 0010:[<ffffffffa03035be>]  [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]
RSP: 0018:ffff88013ddebe98  EFLAGS: 00010246
RAX: ffff88013f894480 RBX: ffff88013f44de40 RCX: 0000000000000000
RDX: 0000000000000002 RSI: ffff88013f44de40 RDI: 0000000000000001
RBP: ffff88013ddebeb8 R08: ffff88013ddea000 R09: dead000000200200
R10: dead000000100100 R11: ffff88013ddebd80 R12: 0000000000000000
R13: ffffffff81771860 R14: 0000000000000002 R15: 0000000000000000
FS:  00007f02c3e1f710(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000794f38 CR3: 000000013d98d000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process racoon (pid: 1941, threadinfo ffff88013ddea000, task ffff88013dffd880)
Stack:
 000000000000000f ffff88013f44de40 0000000000000003 ffffffffa0304a80
<0> ffff88013ddebf28 ffffffff81220802 ffff88013ddebee8 00007f02c3a22e80
<0> ffffffff00000001 ffff88013ddebf60 ffffffff81771860 0000000200000001
Call Trace:
 [<ffffffff81220802>] __sock_create+0x242/0x3dd
 [<ffffffff812209e9>] sock_create+0x2b/0x2d
 [<ffffffff81220b91>] sys_socket+0x26/0x57
 [<ffffffff8129fe0f>] ? page_fault+0x1f/0x30
 [<ffffffff81002a2b>] system_call_fastpath+0x16/0x1b
Code: 49 89 fd 41 54 bf 01 00 00 00 53 44 8b 25 27 19 00 00 48 89 f3 e8 4f f1 f9 e0 49 8b 85 48 08 00 00 45 85 e4 74 05 44 3b 20 76 04 <0f> 0b eb fe 4d 63 e4 4e 8b 64 e0 10 bf 01 00 00 00 e8 aa f0 f9 
RIP  [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]
 RSP <ffff88013ddebe98>
---[ end trace 78cabe73779ec9df ]---

note: racoon[1941] exited with preempt_count 1

It looks like a bug I reported a while ago[1] and which was fixed by Eric. The fix is still in place, but the bug has resurfaced recently, probably in .33. It's quiet elusive, i.e. it happens maybe once every 5 boots; restarting racoon doesn't seem to trigger it...

Luca
[1] http://bugzilla.kernel.org/show_bug.cgi?id=13838

Alexey Dobriyan

2010-01-29 10:17 UTC #1 lore raw

Quoted 10 lines · Luca Tettamanti

with recent kernels I'm seeing this BUG - triggered by racoon - at boot:
NET: Registered protocol family 15
------------[ cut here ]------------
kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/kernel/uevent_seqnum
CPU 1
Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa /F3Sa
RIP: 0010:[<ffffffffa03035be>]  [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]

Does it triggers after succesfull boot if you do

rmmod af_key; modprobe af_key

a couple of times?

Post .config, just in case.

Eric Dumazet

2010-01-29 15:22 UTC #2 lore raw

Quoted 12 lines · Alexey Dobriyan

Quoted 10 lines · Luca Tettamanti
with recent kernels I'm seeing this BUG - triggered by racoon - at boot:
NET: Registered protocol family 15
------------[ cut here ]------------
kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/kernel/uevent_seqnum
CPU 1
Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa /F3Sa
RIP: 0010:[<ffffffffa03035be>]  [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]

Quoted 7 lines

Does it triggers after succesfull boot if you do
rmmod af_key; modprobe af_key
a couple of times?
Post .config, just in case.

I am looking at ipsec_pfkey_init()

We call sock_register(&pfkey_family_ops) before pfkey_net_id being initialized (by the call to register_pernet_subsys(&pfkey_net_ops);

As soon as sock_register(&pfkey_family_ops) is done, another thread can open a socket and call pfkey_create() -> crash

We should change order of initializations somehow

Eric Dumazet

2010-01-29 16:33 UTC #3 lore raw

PATCH 1/1 xfrm: Change initializations order in ipsec_pfkey_init()

Quoted 23 lines · Eric Dumazet

Quoted 12 lines · Alexey Dobriyan
Quoted 10 lines · Luca Tettamanti
with recent kernels I'm seeing this BUG - triggered by racoon - at boot:
NET: Registered protocol family 15
------------[ cut here ]------------
kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/kernel/uevent_seqnum
CPU 1
Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa /F3Sa
RIP: 0010:[<ffffffffa03035be>]  [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]
Quoted 7 lines
Does it triggers after succesfull boot if you do
rmmod af_key; modprobe af_key
a couple of times?
Post .config, just in case.
I am looking at ipsec_pfkey_init()

Quoted 8 lines

We call sock_register(&pfkey_family_ops) before pfkey_net_id being initialized (by the call to register_pernet_subsys(&pfkey_net_ops);
As soon as sock_register(&pfkey_family_ops) is done, another thread can open a socket and call pfkey_create() -> crash
We should change order of initializations somehow

Something like this (compiled but not tested) patch ?

Should probably be sent to stable team...

[PATCH] xfrm: Change initializations order in ipsec_pfkey_init()

Before allowing other threads to create PF_KEY sockets, we must make sure pfkey_net_id is properly initialized.

That means calling register_pernet_subsys(&pfkey_net_ops) before sock_register(&pfkey_family_ops)

Reported-by Luca Tettamanti <[email protected]>

Signed-off-by Eric Dumazet <[email protected]>

---
 net/key/af_key.c |   15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

net/key/af_key.c +9 -6

index 76fa6fe..e399ddf 100644
@@ -3807,21 +3807,24 @@ static int __init ipsec_pfkey_init(void)
 	if (err != 0)
 		goto out;
 
-	err = sock_register(&pfkey_family_ops);
-	if (err != 0)
-		goto out_unregister_key_proto;
 	err = xfrm_register_km(&pfkeyv2_mgr);
 	if (err != 0)
-		goto out_sock_unregister;
+		goto out_unregister_key_proto;
+
 	err = register_pernet_subsys(&pfkey_net_ops);
 	if (err != 0)
 		goto out_xfrm_unregister_km;
+
+	err = sock_register(&pfkey_family_ops);
+	if (err != 0)
+		goto out_unregister_pernet;
 out:
 	return err;
+
+out_unregister_pernet:
+	unregister_pernet_subsys(&pfkey_net_ops);
 out_xfrm_unregister_km:
 	xfrm_unregister_km(&pfkeyv2_mgr);
-out_sock_unregister:
-	sock_unregister(PF_KEY);
 out_unregister_key_proto:
 	proto_unregister(&key_proto);
 	goto out;

Alexey Dobriyan

2010-01-30 12:53 UTC #4 lore raw

PATCH 1/1 af_key: fix netns ops ordering on module load/unload

Quoted 21 lines · Eric Dumazet

(quoted diff) +6 -4

@@ -3807,21 +3807,24 @@ static int __init ipsec_pfkey_init(void)
 	if (err != 0)
 		goto out;
 
-	err = sock_register(&pfkey_family_ops);
-	if (err != 0)
-		goto out_unregister_key_proto;
 	err = xfrm_register_km(&pfkeyv2_mgr);
 	if (err != 0)
-		goto out_sock_unregister;
+		goto out_unregister_key_proto;
+
 	err = register_pernet_subsys(&pfkey_net_ops);
 	if (err != 0)
 		goto out_xfrm_unregister_km;
+
+	err = sock_register(&pfkey_family_ops);
+	if (err != 0)
+		goto out_unregister_pernet;
 out:
 	return err;

Review static int __init ipsec_pfkey_init(void) · line 3830

Quoted 10 lines

(quoted diff) +3 -2

+
+out_unregister_pernet:
+	unregister_pernet_subsys(&pfkey_net_ops);
 out_xfrm_unregister_km:
 	xfrm_unregister_km(&pfkeyv2_mgr);
-out_sock_unregister:
-	sock_unregister(PF_KEY);
 out_unregister_key_proto:
 	proto_unregister(&key_proto);
 	goto out;

ACK analysis, except this is not enough.

Here is patch which survived netns start/stop/modprobe/rmmod cycles.

Alexey, who still doesn't get why bug reproduces so easily for bug reporter.

Luca, please confirm.

Audit for the rest of the modules pending.

[PATCH] af_key: fix netns ops ordering on module load/unload

1. After sock_register() returns, it's possible to create sockets, even if module still not initialized fully (blame generic module code for that!)
2. Consequently, pfkey_create() can be called with pfkey_net_id still not initialized which will BUG_ON in net_generic():
kernel BUG at include/net/netns/generic.h:43!
3. During netns shutdown, netns ops should be unregistered after key manager unregistered because key manager calls can be triggered from xfrm_user module:

   	general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
	pfkey_broadcast+0x111/0x210 [af_key]
	pfkey_send_notify+0x16a/0x300 [af_key]
	km_state_notify+0x41/0x70
	xfrm_flush_sa+0x75/0x90 [xfrm_user]

4. Unregister netns ops after socket ops just in case and for symmetry.

Reported by Luca Tettamanti.

Signed-off-by Alexey Dobriyan <[email protected]>

---

 net/key/af_key.c |   15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

net/key/af_key.c +8 -7

@@ -3794,9 +3794,9 @@ static struct pernet_operations pfkey_net_ops = {
 
 static void __exit ipsec_pfkey_exit(void)
 {
-	unregister_pernet_subsys(&pfkey_net_ops);
 	xfrm_unregister_km(&pfkeyv2_mgr);
 	sock_unregister(PF_KEY);
+	unregister_pernet_subsys(&pfkey_net_ops);
 	proto_unregister(&key_proto);
 }

@@ -3807,21 +3807,22 @@ static int __init ipsec_pfkey_init(void)
 	if (err != 0)
 		goto out;
 
-	err = sock_register(&pfkey_family_ops);
+	err = register_pernet_subsys(&pfkey_net_ops);
 	if (err != 0)
 		goto out_unregister_key_proto;
+	err = sock_register(&pfkey_family_ops);
+	if (err != 0)
+		goto out_unregister_pernet;
 	err = xfrm_register_km(&pfkeyv2_mgr);
 	if (err != 0)
 		goto out_sock_unregister;
-	err = register_pernet_subsys(&pfkey_net_ops);
-	if (err != 0)
-		goto out_xfrm_unregister_km;
 out:
 	return err;
-out_xfrm_unregister_km:
-	xfrm_unregister_km(&pfkeyv2_mgr);
+
 out_sock_unregister:
 	sock_unregister(PF_KEY);
+out_unregister_pernet:
+	unregister_pernet_subsys(&pfkey_net_ops);
 out_unregister_key_proto:
 	proto_unregister(&key_proto);
 	goto out;

Luca Tettamanti

author Tested-by on v1 1/1

2010-02-01 13:50 UTC #5 lore raw

Re: [PATCH] af_key: fix netns ops ordering on module load/unload

Quoted 35 lines · Alexey Dobriyan

Quoted 24 lines · Eric Dumazet

(quoted diff) +6 -4

@@ -3807,21 +3807,24 @@ static int __init ipsec_pfkey_init(void)
       if (err != 0)
               goto out;
 
-     err = sock_register(&pfkey_family_ops);
-     if (err != 0)
-             goto out_unregister_key_proto;
       err = xfrm_register_km(&pfkeyv2_mgr);
       if (err != 0)
-             goto out_sock_unregister;
+             goto out_unregister_key_proto;
+
       err = register_pernet_subsys(&pfkey_net_ops);
       if (err != 0)
               goto out_xfrm_unregister_km;
+
+     err = sock_register(&pfkey_family_ops);
+     if (err != 0)
+             goto out_unregister_pernet;

out:
return err;

net/key/af_key.c +3 -0

+
+out_unregister_pernet:
+     unregister_pernet_subsys(&pfkey_net_ops);

Quoted 7 lines

out_xfrm_unregister_km:
xfrm_unregister_km(&pfkeyv2_mgr);
-out_sock_unregister:
- sock_unregister(PF_KEY);
out_unregister_key_proto:
proto_unregister(&key_proto);
goto out;

ACK analysis, except this is not enough.

Quoted 5 lines

Here is patch which survived netns start/stop/modprobe/rmmod cycles.
Alexey, who still doesn't get why bug reproduces so easily for bug reporter.
Luca, please confirm.

Seems to work fine.

Quoted 17 lines

[PATCH] af_key: fix netns ops ordering on module load/unload
1. After sock_register() returns, it's possible to create sockets, even if module still not initialized fully (blame generic module code for that!)
2. Consequently, pfkey_create() can be called with pfkey_net_id still not initialized which will BUG_ON in net_generic():
kernel BUG at include/net/netns/generic.h:43!
3. During netns shutdown, netns ops should be unregistered after key manager unregistered because key manager calls can be triggered from xfrm_user module:
       general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
       pfkey_broadcast+0x111/0x210 [af_key]
       pfkey_send_notify+0x16a/0x300 [af_key]
       km_state_notify+0x41/0x70
       xfrm_flush_sa+0x75/0x90 [xfrm_user]

Quoted 5 lines

4. Unregister netns ops after socket ops just in case and for symmetry.
Reported by Luca Tettamanti.
Signed-off-by Alexey Dobriyan <[email protected]>

Tested-by Luca Tettamanti <[email protected]>

thanks,
Luca

Eric Dumazet

2010-02-01 13:56 UTC #6 lore raw

Re: [PATCH] af_key: fix netns ops ordering on module load/unload

Quoted 21 lines · Luca Tettamanti

Quoted 19 lines · Alexey Dobriyan
[PATCH] af_key: fix netns ops ordering on module load/unload
1. After sock_register() returns, it's possible to create sockets, even if module still not initialized fully (blame generic module code for that!)
2. Consequently, pfkey_create() can be called with pfkey_net_id still not initialized which will BUG_ON in net_generic():
kernel BUG at include/net/netns/generic.h:43!
3. During netns shutdown, netns ops should be unregistered after key manager unregistered because key manager calls can be triggered from xfrm_user module:
       general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
       pfkey_broadcast+0x111/0x210 [af_key]
       pfkey_send_notify+0x16a/0x300 [af_key]
       km_state_notify+0x41/0x70
       xfrm_flush_sa+0x75/0x90 [xfrm_user]
4. Unregister netns ops after socket ops just in case and for symmetry.

Quoted 5 lines

Quoted 3 lines
Reported by Luca Tettamanti.
Signed-off-by Alexey Dobriyan <[email protected]>
Tested-by Luca Tettamanti <[email protected]>

Signed-off-by Eric Dumazet <[email protected]>

David Miller

maintainer Applied

2010-02-04 02:11 UTC #7 lore raw

Re: [PATCH] af_key: fix netns ops ordering on module load/unload

From: Eric Dumazet <[email protected]>
Date: Mon, 01 Feb 2010 14:56:30 +0100

Quoted 22 lines · Luca Tettamanti

Quoted 21 lines · Alexey Dobriyan
Quoted 20 lines
[PATCH] af_key: fix netns ops ordering on module load/unload
1. After sock_register() returns, it's possible to create sockets, even if module still not initialized fully (blame generic module code for that!)
2. Consequently, pfkey_create() can be called with pfkey_net_id still not initialized which will BUG_ON in net_generic():
kernel BUG at include/net/netns/generic.h:43!
3. During netns shutdown, netns ops should be unregistered after key manager unregistered because key manager calls can be triggered from xfrm_user module:
       general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
       pfkey_broadcast+0x111/0x210 [af_key]
       pfkey_send_notify+0x16a/0x300 [af_key]
       km_state_notify+0x41/0x70
       xfrm_flush_sa+0x75/0x90 [xfrm_user]
4. Unregister netns ops after socket ops just in case and for symmetry.

Quoted 7 lines

Quoted 5 lines
Quoted 3 lines
Reported by Luca Tettamanti.
Signed-off-by Alexey Dobriyan <[email protected]>
Tested-by Luca Tettamanti <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Applied, thanks everyone!

Luca Tettamanti

author

2010-01-30 13:23 UTC #8 lore raw

In reply to Alexey Dobriyan

Quoted 12 lines · Alexey Dobriyan

Quoted 10 lines · Luca Tettamanti
with recent kernels I'm seeing this BUG - triggered by racoon - at boot:
NET: Registered protocol family 15
------------[ cut here ]------------
kernel BUG at /home/kronos/src/linux-2.6.git/include/net/netns/generic.h:43!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/kernel/uevent_seqnum
CPU 1
Pid: 1941, comm: racoon Not tainted 2.6.33-rc5-00271-gbe8cde8-dirty #238 F3Sa /F3Sa
RIP: 0010:[<ffffffffa03035be>]  [<ffffffffa03035be>] pfkey_create+0x36/0x18b [af_key]

Quoted 5 lines

Does it triggers after succesfull boot if you do
rmmod af_key; modprobe af_key
a couple of times?

No. Neither does:
racoon stop; rmmod af_key; racoon start

It might be connected to the fact that I'm using Debian startpar to run multiple init scripts in parallel.
I'm now testing Eric's patch and I haven't seen the BUG after a few reboots.

Quoted 1 line

Post .config, just in case.

Attached.

Luca