crypto_remove_spawns: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018

DISCUSSION Alexey Dobriyan 2 participants 10 messages Feb 14, 2010 – Feb 18, 2010 Crypto API

Share & tools

Alexey Dobriyan

author

2010-02-14 17:45 UTC #0 lore raw

BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
IP: [<ffffffff81145bf4>] crypto_remove_spawns+0xd4/0x340
PGD bdc48067 PUD bc954067 PMD 0 
Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/uevent
CPU 0 
Pid: 16500, comm: rmmod Not tainted 2.6.33-rc7-next-20100212+ #9 P5E/P5E
RIP: 0010:[<ffffffff81145bf4>]  [<ffffffff81145bf4>] crypto_remove_spawns+0xd4/0x340
RSP: 0018:ffff8800bc9dfde8  EFLAGS: 00010282
RAX: ffff8800bc901498 RBX: 0000000000000000 RCX: ffff8800ba859610
RDX: ffff8800bc900380 RSI: ffff8800bc9dfe18 RDI: ffff8800bc9015c0
RBP: ffff8800bc9dfe68 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800bc901488
R13: ffff8800bc9dfe18 R14: ffffffffa05817e0 R15: 0000000000000000
FS:  00007fdd2ec1c6f0(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000018 CR3: 00000000bca34000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process rmmod (pid: 16500, threadinfo ffff8800bc9de000, task ffff8800bd53ad90)
Stack:
 ffff8800bc9dfe08 ffff8800bc9dfe28 ffff8800bc9dfe98 0000042181636020
<0> ffff8800bc9dfe08 ffff8800bc9dfe08 ffff8800bc9015c0 ffff8800bc900380
<0> ffff8800ba859808 ffff8800ba859610 ffff8800bc9dfe98 ffffffffa05817e0
Call Trace:
 [<ffffffff81145eb1>] crypto_remove_alg+0x51/0x60
 [<ffffffff81145ef3>] crypto_unregister_alg+0x33/0x90
 [<ffffffffa058175c>] aes_fini+0x10/0x12 [aes_x86_64]
 [<ffffffff8107266c>] sys_delete_module+0x19c/0x250
 [<ffffffff8100256b>] system_call_fastpath+0x16/0x1b
Code: 02 00 eb c3 0f 1f 00 48 8b 47 08 48 8d 75 c0 4c 89 28 49 89 45 08 48 8b 55 c0 e8 a8 fa 02 00 48 8d 45 a0 48 8b 18 48 39 d8 74 44 <4c> 8b 63 18 4d 39 f4 0f 84 4e 02 00 00 48 8b 13 48 8b 43 08 4c 
RIP  [<ffffffff81145bf4>] crypto_remove_spawns+0xd4/0x340
 RSP <ffff8800bc9dfde8>

CR2: 0000000000000018

crypto_remove_spawns:

	spawn = list_first_entry(spawns, struct crypto_spawn, list);
	inst = spawn->inst;

spawn is NULL here.

Herbert Xu

maintainer

2010-02-15 05:27 UTC #1 lore raw

Quoted 36 lines · Alexey Dobriyan

BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
IP: [<ffffffff81145bf4>] crypto_remove_spawns+0xd4/0x340
PGD bdc48067 PUD bc954067 PMD 0 
Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/uevent
CPU 0 
Pid: 16500, comm: rmmod Not tainted 2.6.33-rc7-next-20100212+ #9 P5E/P5E
RIP: 0010:[<ffffffff81145bf4>]  [<ffffffff81145bf4>] crypto_remove_spawns+0xd4/0x340
RSP: 0018:ffff8800bc9dfde8  EFLAGS: 00010282
RAX: ffff8800bc901498 RBX: 0000000000000000 RCX: ffff8800ba859610
RDX: ffff8800bc900380 RSI: ffff8800bc9dfe18 RDI: ffff8800bc9015c0
RBP: ffff8800bc9dfe68 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800bc901488
R13: ffff8800bc9dfe18 R14: ffffffffa05817e0 R15: 0000000000000000
FS:  00007fdd2ec1c6f0(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000018 CR3: 00000000bca34000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process rmmod (pid: 16500, threadinfo ffff8800bc9de000, task ffff8800bd53ad90)
Stack:
 ffff8800bc9dfe08 ffff8800bc9dfe28 ffff8800bc9dfe98 0000042181636020
<0> ffff8800bc9dfe08 ffff8800bc9dfe08 ffff8800bc9015c0 ffff8800bc900380
<0> ffff8800ba859808 ffff8800ba859610 ffff8800bc9dfe98 ffffffffa05817e0
Call Trace:
 [<ffffffff81145eb1>] crypto_remove_alg+0x51/0x60
 [<ffffffff81145ef3>] crypto_unregister_alg+0x33/0x90
 [<ffffffffa058175c>] aes_fini+0x10/0x12 [aes_x86_64]
 [<ffffffff8107266c>] sys_delete_module+0x19c/0x250
 [<ffffffff8100256b>] system_call_fastpath+0x16/0x1b
Code: 02 00 eb c3 0f 1f 00 48 8b 47 08 48 8d 75 c0 4c 89 28 49 89 45 08 48 8b 55 c0 e8 a8 fa 02 00 48 8d 45 a0 48 8b 18 48 39 d8 74 44 <4c> 8b 63 18 4d 39 f4 0f 84 4e 02 00 00 48 8b 13 48 8b 43 08 4c 
RIP  [<ffffffff81145bf4>] crypto_remove_spawns+0xd4/0x340
 RSP <ffff8800bc9dfde8>

CR2: 0000000000000018

Quoted 6 lines

crypto_remove_spawns:
	spawn = list_first_entry(spawns, struct crypto_spawn, list);
	inst = spawn->inst;
spawn is NULL here.

Is this reproducible every time you unload aes_x86_64 after boot? Please attach your config file?

Thanks,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Alexey Dobriyan

author

2010-02-15 07:47 UTC #2 lore raw

Quoted 1 line · Herbert Xu

Is this reproducible every time you unload aes_x86_64 after boot?

No, what I do is

1. setup ipcomp in tunnel mode _in fresh netns_ and immediately exit
2. modprobe/rmmod all modules (not much)

~1 hour of this workload and it hits sometimes with aes_x86_64, sometimes with aes_generic.

Quoted 1 line

Please attach your config file?

Full config later, for now it's ipv4 only, XFRM stuff as modules, crypto modules as modules, almost all debugging on.

Herbert Xu

maintainer

2010-02-15 08:11 UTC #3 lore raw

Quoted 10 lines · Alexey Dobriyan

Quoted 1 line · Herbert Xu
Is this reproducible every time you unload aes_x86_64 after boot?
No, what I do is
1. setup ipcomp in tunnel mode _in fresh netns_ and immediately exit
2. modprobe/rmmod all modules (not much)
~1 hour of this workload and it hits sometimes with aes_x86_64, sometimes with aes_generic.

Was this with that IPCOMP bug fixed?

Cheers,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Alexey Dobriyan

author

2010-02-15 08:14 UTC #4 lore raw

Quoted 6 lines · Herbert Xu

Quoted 5 lines · Alexey Dobriyan
Quoted 1 line · Herbert Xu
Is this reproducible every time you unload aes_x86_64 after boot?
No, what I do is

Quoted 7 lines

Quoted 5 lines
1. setup ipcomp in tunnel mode _in fresh netns_ and immediately exit
2. modprobe/rmmod all modules (not much)
~1 hour of this workload and it hits sometimes with aes_x86_64, sometimes with aes_generic.
Was this with that IPCOMP bug fixed?

Yes, ipcomp bug triggers almost immediately.
Anyway, this is just description of what I do.

Herbert Xu

maintainer

2010-02-16 12:02 UTC #5 lore raw

Quoted 3 lines · Alexey Dobriyan

Yes, ipcomp bug triggers almost immediately.
Anyway, this is just description of what I do.

Can you see if this patch makes it go away?

This can happen when you're unloading aes just as an algorithm that uses aes (such as cbc(aes)) is being created.

crypto/algapi.c +2 -0

index f149b1c..88c5f6c 100644
@@ -165,6 +165,8 @@ static void crypto_remove_spawns(struct crypto_alg *alg,
 
 			spawn->alg = NULL;
 			spawns = &inst->alg.cra_users;
+			if (!spawns->next)
+				break;
 		}
 	} while ((spawns = crypto_more_spawns(alg, &stack, &top,
 					      &secondary_spawns)));

Thanks,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Alexey Dobriyan

author

2010-02-16 19:31 UTC #6 lore raw

Quoted 9 lines · Herbert Xu

Quoted 3 lines · Alexey Dobriyan
Yes, ipcomp bug triggers almost immediately.
Anyway, this is just description of what I do.
Can you see if this patch makes it go away?
This can happen when you're unloading aes just as an algorithm that uses aes (such as cbc(aes)) is being created.

Which codepath exactly?
I'd say try_module_get() should fail somewhere.

BTW, CBC or AES aren't used, just loaded.

Here is setkey script:

	#!/usr/sbin/setkey -f
	flush;
	spdflush;
	
	add A B ipcomp 44 -m tunnel -C deflate;
	add B A ipcomp 45 -m tunnel -C deflate;
	
	spdadd A B any -P in ipsec
	        ipcomp/tunnel/192.168.1.2-192.168.1.3/use;
	
	spdadd B A any -P out ipsec
	        ipcomp/tunnel/192.168.1.3-192.168.1.2/use;

Quoted 11 lines

crypto/algapi.c +2 -0

@@ -165,6 +165,8 @@ static void crypto_remove_spawns(struct crypto_alg *alg,
 
 			spawn->alg = NULL;
 			spawns = &inst->alg.cra_users;
+			if (!spawns->next)
+				break;
 		}
 	} while ((spawns = crypto_more_spawns(alg, &stack, &top,
 					      &secondary_spawns)));

Herbert Xu

maintainer

2010-02-17 00:37 UTC #7 lore raw

Quoted 2 lines · Alexey Dobriyan

Which codepath exactly?

When a spawn is created the instance associated with it will have a zero-initialised cra_users entry.

Quoted 1 line

BTW, CBC or AES aren't used, just loaded.

Can you boot without aes/cbc loaded and see if it gets automatically loaded? PF_KEY users tend to load all possible algorithms when they start up.

Cheers,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Alexey Dobriyan

author

2010-02-17 19:31 UTC #8 lore raw

Quoted 4 lines · Herbert Xu

Quoted 2 lines · Alexey Dobriyan
Which codepath exactly?

Quoted 8 lines

When a spawn is created the instance associated with it will have a zero-initialised cra_users entry.
Quoted 1 line
BTW, CBC or AES aren't used, just loaded.
Can you boot without aes/cbc loaded and see if it gets automatically loaded? PF_KEY users tend to load all possible algorithms when they start up.

AES and CBC are indeed loaded even if not used.

xfrm4_tunnel            1701  0
tunnel4                 2319  1 xfrm4_tunnel
ipcomp                  2063  0
xfrm_ipcomp             5154  1 ipcomp
xfrm4_mode_tunnel       1861  0
deflate                 2033  0
zlib_deflate           21231  1 deflate
zlib_inflate           18334  1 deflate
aes_generic            26652  0
des_generic            16263  0
cbc                     2945  0
sha1_generic            2223  0
md5                     4161  0
hmac                    3041  0
cryptomgr             105876  0
aead                    6474  1 cryptomgr
pcompress               1767  1 cryptomgr
crypto_null             2910  0
crypto_blkcipher       12080  3 cbc,cryptomgr,crypto_null
crypto_hash            15857  5

sha1_generic,md5,hmac,cryptomgr,crypto_null
crypto_algapi 16753 11
deflate,aes_generic,des_generic,cbc,hmac,cryptomgr,aead,pcompress,crypto_null,crypto_blkcipher,crypto_hash
af_key 30024 0

Race started to become even less reproducible, so I can't confirm your patch quickly. :-(

Herbert Xu

maintainer

2010-02-18 00:26 UTC #9 lore raw

Quoted 2 lines · Alexey Dobriyan

Race started to become even less reproducible, so I can't confirm your patch quickly. :-(

See if it's setkey or racoon that's loading the algorithms. Once you know which it is, just repeatedly restart that while running rmmod on aes.

Cheers,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt