Oops on 2.4.22 when mounting from broken NFS server
Attached is the output of ksymoops from an Oops when mounting from a broken NFS server. I was experimenting with a new security policy for the NFS server and didn't grant the daemons all the access they needed. Afterwards I noticed that kernel had Oops'd on a mount command (I should have suspected when the mount SEGV'd).
I can probably reproduce this if requested. It's 2.4.22 client and server.
Signature · 4 lines
http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
ksymoops 2.4.5 on i586 2.4.19-cobalt. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.19-cobalt/ (default) -m /boot/System.map-2.4.19-cobalt (default)
Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.
Unable to handle kernel NULL pointer dereference at virtual address 00000021 c0109d85 *pde = 00000000 Oops: 0000 CPU: 0 EIP: 0010:[<c0109d85>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010216
eax: 00000240 ebx: 00000001 ecx: 0000000c edx: c025a910
esi: 00000400 edi: c025b304 ebp: 00000005 esp: c7cade78
ds: 0018 es: 0018 ss: 0018
Process ifconfig (pid: 74, stackpage=c7cad000)
Stack: c7705000 c7705000 c7705160 c8820000 00000013 00000003 c8819872 0000000a
c881a464 04000000 c7705000 c7705000 c881a43c c881be8a c7705000 c7705000
00000000 00001043 00000000 c01a5a1f c7705000 c7705000 00001002 c01a6863
Call Trace: [<c8819872>] [<c881a464>] [<c881a43c>] [<c881be8a>] [<c01a5a1f>]
[<c01a6863>] [<c01d0241>] [<c01d20f7>] [<c01db602>] [<c019fd46>] [<c0139986>]
[<c01085a3>]
Code: 39 5d 1c 75 06 8b 34 39 09 34 10 8b 6d 18 85 ed 75 ee 8b 54 Quoted 5 lines
Quoted 5 lines
EIP;
c0109d85<request_irq_class+1e1/208> <=====edx;
c025a910<irq_desc+10/200>
edi;c025b304<irq_classes+4/180>
esp;c7cade78<_end+7a2a38c/8595514>
Trace; c8819872 <[natsemi]netdev_open+32/100> Trace; c881a464 <[natsemi]intr_handler+0/e8> Trace; c881a43c <[natsemi]intr_mask+0/28>
Trace; c881be8a <[natsemi]__module_pci_device_size+2b6/b00> Trace; c01a5a1f <dev_open+47/a0>
Trace; c01a6863 <dev_change_flags+4f/fc>
Trace; c01d0241 <devinet_ioctl+315/664>
Trace; c01d20f7 <inet_ioctl+133/17c>
Trace; c01db602 <bw_sock_ioctl+1a/38>
Trace; c019fd46 <sock_ioctl+1e/24>
Trace; c0139986 <sys_ioctl+16a/184>
Trace; c01085a3 <system_call+33/40>
Code; c0109d85 <request_irq_class+1e1/208>
00000000 <_EIP>:
Code; c0109d85 <request_irq_class+1e1/208> <=====
0: 39 5d 1c cmp %ebx,0x1c(%ebp) <=====
Code; c0109d88 <request_irq_class+1e4/208>
3: 75 06 jne b <_EIP+0xb>
Code; c0109d8a <request_irq_class+1e6/208>
5: 8b 34 39 mov (%ecx,%edi,1),%esi
Code; c0109d8d <request_irq_class+1e9/208>
8: 09 34 10 or %esi,(%eax,%edx,1)
Code; c0109d90 <request_irq_class+1ec/208>
b: 8b 6d 18 mov 0x18(%ebp),%ebp
Code; c0109d93 <request_irq_class+1ef/208>
e: 85 ed test %ebp,%ebp
Code; c0109d95 <request_irq_class+1f1/208>
10: 75 ee jne 0 <_EIP>
Code; c0109d97 <request_irq_class+1f3/208>
12: 8b 54 00 00 mov 0x0(%eax,%eax,1),%edx
1 warning issued. Results may not be reliable.
Quoted 7 lines · Russell Coker
Attached is the output of ksymoops from an Oops when mounting from a broken NFS server. I was experimenting with a new security policy for the NFS server and didn't grant the daemons all the access they needed. Afterwards I noticed that kernel had Oops'd on a mount command (I should have suspected when the mount SEGV'd).
I can probably reproduce this if requested. It's 2.4.22 client and server.
This is embarassing. I attached the wrong file to the last message. Attached is the correct one.
Signature · 4 lines
http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
ksymoops 2.4.8 on i586 2.4.22-cobalt. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.22-cobalt/ (default) -m /boot/System.map-2.4.22-cobalt (default)
Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.
Unable to handle kernel NULL pointer dereference at virtual address 00000000 d88891ee *pde = 00000000 Oops: 0000 CPU: 0 EIP: 0010:[<d88891ee>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 00000000 ebx: 00000000 ecx: c1425348 edx: c1425350
esi: c69958a0 edi: d88b5940 ebp: d88b59c0 esp: c9787da4
ds: 0018 es: 0018 ss: 0018
Process mount (pid: 8396, stackpage=c9787000)
Stack: c7f9a4f0 d88a98ee 00000000 c15b9e60 00000286 c9787ed0 00000000 c0129fe7
c14255d0 00000286 d7981ec0 c01df4ab c7f9a410 00000002 00000001 c9787e84
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Call Trace: [<d88a98ee>] [<c0129fe7>] [<c01df4ab>] [<c013730a>] [<d88b7c50>]
[<d88b7c50>] [<d88b7c50>] [<c01375f7>] [<d88b7c50>] [<c01377e0>] [<d88b7c50>]
[<c0148cf8>] [<c0148fd2>] [<c0148e36>] [<c0149394>] [<c0107203>]
Code: 8b 03 85 c0 74 29 f6 05 2c 89 89 d8 02 75 29 80 63 20 3f 53 Quoted 8 lines
Quoted 8 lines
EIP;
d88891ee<[sunrpc]rpc_shutdown_client+e/80> <=====ecx;
c1425348<_end+11492f8/1853f010>
edx;c1425350<_end+1149300/1853f010>
esi;c69958a0<_end+66b9850/1853f010>
edi;d88b5940<[nfs].rodata.end+659/2599>
ebp;d88b59c0<[nfs].rodata.end+6d9/2599>
esp;c9787da4<_end+94abd54/1853f010>
Trace; d88a98ee <[nfs]nfs_read_super+42e/960>
Trace; c0129fe7 <kfree+27/40>
Trace; c01df4ab <kfree_skbmem+b/60>
Trace; c013730a <get_anon_super+8a/e0>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; c01375f7 <get_sb_nodev+37/80>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; c01377e0 <do_kern_mount+100/140>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; c0148cf8 <do_add_mount+58/140>
Trace; c0148fd2 <do_mount+132/180>
Trace; c0148e36 <copy_mount_options+56/c0>
Trace; c0149394 <sys_mount+74/c0>
Trace; c0107203 <system_call+33/40>
Code; d88891ee <[sunrpc]rpc_shutdown_client+e/80> 00000000 <_EIP>:
Code; d88891ee <[sunrpc]rpc_shutdown_client+e/80> <=====
0: 8b 03 mov (%ebx),%eax <=====
Code; d88891f0 <[sunrpc]rpc_shutdown_client+10/80>
2: 85 c0 test %eax,%eax
Code; d88891f2 <[sunrpc]rpc_shutdown_client+12/80>
4: 74 29 je 2f <_EIP+0x2f>
Code; d88891f4 <[sunrpc]rpc_shutdown_client+14/80>
6: f6 05 2c 89 89 d8 02 testb $0x2,0xd889892c
Code; d88891fb <[sunrpc]rpc_shutdown_client+1b/80>
d: 75 29 jne 38 <_EIP+0x38>
Code; d88891fd <[sunrpc]rpc_shutdown_client+1d/80>
f: 80 63 20 3f andb $0x3f,0x20(%ebx)
Code; d8889201 <[sunrpc]rpc_shutdown_client+21/80>
13: 53 push %ebx
1 warning issued. Results may not be reliable.
Without a tcpdump, there's no way to know why this dumped. In what way was the NFS server broken, and exactly how do you expect the Linux NFS client to protect you against it?
Cheers,
Trond
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Trond Myklebust <[email protected]> writes:
> Without a tcpdump, there's no way to know why this dumped. In > what way was the NFS server broken, and exactly how do you > expect the Linux NFS client to protect you against it?
BTW: that Oops you posted looked very much like a memory corruption problem. Were you running vanilla 2.4.22 on the client, or was it too patched?
Cheers,
Trond
Quoted 3 lines · Trond Myklebust
Without a tcpdump, there's no way to know why this dumped. In what way was the NFS server broken, and exactly how do you expect the Linux NFS client to protect you against it?
The NFS server was unable to read the directories it was exporting, so it would allow the mount command but then fail to do anything.
I expect the NFS client to behave sensibly in the face of all possible server errors, including the possibility of a hostile NFS server.
Quoted 3 lines
BTW: that Oops you posted looked very much like a memory corruption problem. Were you running vanilla 2.4.22 on the client, or was it too patched?
It was also patched. I will try and reproduce the error with an unpatched kernel and a tcpdump running.
Signature · 4 lines
http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Russell Coker <[email protected]> writes:
> I expect the NFS client to behave sensibly in the face of all > possible server errors, including the possibility of a hostile > NFS server.
No can do... There are 100s of scenarios where the server can screw the client by giving it bogus information. You might possibly be able to protect against a few of them, but at a heavy price in the form of code bloat.
Neither NFSv2 nor NFSv3 are protocols that were designed to operate safely in a hostile environment. They were designed for LANs where servers and clients trust one another.
I agree that we shouldn't Oops though.
Quoted 3 lines
Quoted 3 lines
BTW: that Oops you posted looked very much like a memory corruption problem. Were you running vanilla 2.4.22 on the client, or was it too patched?
> It was also patched. I will try and reproduce the error with > an unpatched kernel and a tcpdump running.
Please do...
Cheers,
Trond