Skip to contentSkip to conversation

Oops on 2.4.22 when mounting from broken NFS server

DISCUSSION Russell Coker 2 participants 6 messages Sep 13, 2003 – Sep 15, 2003

lore
Share & tools
lore
#0 lore raw

Attached is the output of ksymoops from an Oops when mounting from a broken NFS server. I was experimenting with a new security policy for the NFS server and didn't grant the daemons all the access they needed. Afterwards I noticed that kernel had Oops'd on a mount command (I should have suspected when the mount SEGV'd).

I can probably reproduce this if requested. It's 2.4.22 client and server.

Signature · 4 lines
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

ksymoops 2.4.5 on i586 2.4.19-cobalt. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.19-cobalt/ (default) -m /boot/System.map-2.4.19-cobalt (default)

Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.

Unable to handle kernel NULL pointer dereference at virtual address 00000021
c0109d85
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c0109d85>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010216
eax: 00000240 ebx: 00000001 ecx: 0000000c edx: c025a910
esi: 00000400 edi: c025b304 ebp: 00000005 esp: c7cade78
ds: 0018 es: 0018 ss: 0018
Process ifconfig (pid: 74, stackpage=c7cad000)

Stack: c7705000 c7705000 c7705160 c8820000 00000013 00000003 c8819872 0000000a 
       c881a464 04000000 c7705000 c7705000 c881a43c c881be8a c7705000 c7705000 
       00000000 00001043 00000000 c01a5a1f c7705000 c7705000 00001002 c01a6863 
Call Trace:    [<c8819872>] [<c881a464>] [<c881a43c>] [<c881be8a>] [<c01a5a1f>]
  [<c01a6863>] [<c01d0241>] [<c01d20f7>] [<c01db602>] [<c019fd46>] [<c0139986>]
  [<c01085a3>]
Code: 39 5d 1c 75 06 8b 34 39 09 34 10 8b 6d 18 85 ed 75 ee 8b 54 
Quoted 5 lines
Quoted 5 lines

EIP; c0109d85 <request_irq_class+1e1/208> <=====

edx; c025a910 <irq_desc+10/200>
edi; c025b304 <irq_classes+4/180>
esp; c7cade78 <_end+7a2a38c/8595514>

Trace; c8819872 <[natsemi]netdev_open+32/100> Trace; c881a464 <[natsemi]intr_handler+0/e8> Trace; c881a43c <[natsemi]intr_mask+0/28>
Trace; c881be8a <[natsemi]__module_pci_device_size+2b6/b00> Trace; c01a5a1f <dev_open+47/a0>
Trace; c01a6863 <dev_change_flags+4f/fc>
Trace; c01d0241 <devinet_ioctl+315/664>
Trace; c01d20f7 <inet_ioctl+133/17c>
Trace; c01db602 <bw_sock_ioctl+1a/38>
Trace; c019fd46 <sock_ioctl+1e/24>
Trace; c0139986 <sys_ioctl+16a/184>
Trace; c01085a3 <system_call+33/40>

Code; c0109d85 <request_irq_class+1e1/208>
00000000 <_EIP>:
Code; c0109d85 <request_irq_class+1e1/208> <=====
0: 39 5d 1c cmp %ebx,0x1c(%ebp) <=====
Code; c0109d88 <request_irq_class+1e4/208>
3: 75 06 jne b <_EIP+0xb>
Code; c0109d8a <request_irq_class+1e6/208>
5: 8b 34 39 mov (%ecx,%edi,1),%esi
Code; c0109d8d <request_irq_class+1e9/208>
8: 09 34 10 or %esi,(%eax,%edx,1)
Code; c0109d90 <request_irq_class+1ec/208>
b: 8b 6d 18 mov 0x18(%ebp),%ebp
Code; c0109d93 <request_irq_class+1ef/208>
e: 85 ed test %ebp,%ebp
Code; c0109d95 <request_irq_class+1f1/208>
10: 75 ee jne 0 <_EIP>
Code; c0109d97 <request_irq_class+1f3/208>
12: 8b 54 00 00 mov 0x0(%eax,%eax,1),%edx

1 warning issued. Results may not be reliable.

#1 lore raw
Quoted 7 lines · Russell Coker

Attached is the output of ksymoops from an Oops when mounting from a broken NFS server. I was experimenting with a new security policy for the NFS server and didn't grant the daemons all the access they needed. Afterwards I noticed that kernel had Oops'd on a mount command (I should have suspected when the mount SEGV'd).

I can probably reproduce this if requested. It's 2.4.22 client and server.

This is embarassing. I attached the wrong file to the last message. Attached is the correct one.

Signature · 4 lines
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

ksymoops 2.4.8 on i586 2.4.22-cobalt. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.22-cobalt/ (default) -m /boot/System.map-2.4.22-cobalt (default)

Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.

Unable to handle kernel NULL pointer dereference at virtual address 00000000
d88891ee
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<d88891ee>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 00000000 ebx: 00000000 ecx: c1425348 edx: c1425350
esi: c69958a0 edi: d88b5940 ebp: d88b59c0 esp: c9787da4
ds: 0018 es: 0018 ss: 0018
Process mount (pid: 8396, stackpage=c9787000)

Stack: c7f9a4f0 d88a98ee 00000000 c15b9e60 00000286 c9787ed0 00000000 c0129fe7 
       c14255d0 00000286 d7981ec0 c01df4ab c7f9a410 00000002 00000001 c9787e84 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:    [<d88a98ee>] [<c0129fe7>] [<c01df4ab>] [<c013730a>] [<d88b7c50>]
  [<d88b7c50>] [<d88b7c50>] [<c01375f7>] [<d88b7c50>] [<c01377e0>] [<d88b7c50>]
  [<c0148cf8>] [<c0148fd2>] [<c0148e36>] [<c0149394>] [<c0107203>]
Code: 8b 03 85 c0 74 29 f6 05 2c 89 89 d8 02 75 29 80 63 20 3f 53 
Quoted 8 lines
Quoted 8 lines

EIP; d88891ee <[sunrpc]rpc_shutdown_client+e/80> <=====

ecx; c1425348 <_end+11492f8/1853f010>
edx; c1425350 <_end+1149300/1853f010>
esi; c69958a0 <_end+66b9850/1853f010>
edi; d88b5940 <[nfs].rodata.end+659/2599>
ebp; d88b59c0 <[nfs].rodata.end+6d9/2599>
esp; c9787da4 <_end+94abd54/1853f010>

Trace; d88a98ee <[nfs]nfs_read_super+42e/960>
Trace; c0129fe7 <kfree+27/40>
Trace; c01df4ab <kfree_skbmem+b/60>
Trace; c013730a <get_anon_super+8a/e0>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; c01375f7 <get_sb_nodev+37/80>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; c01377e0 <do_kern_mount+100/140>
Trace; d88b7c50 <[nfs]nfs_fs_type+0/30>
Trace; c0148cf8 <do_add_mount+58/140>
Trace; c0148fd2 <do_mount+132/180>
Trace; c0148e36 <copy_mount_options+56/c0>
Trace; c0149394 <sys_mount+74/c0>
Trace; c0107203 <system_call+33/40>

Code; d88891ee <[sunrpc]rpc_shutdown_client+e/80> 00000000 <_EIP>:
Code; d88891ee <[sunrpc]rpc_shutdown_client+e/80> <=====
0: 8b 03 mov (%ebx),%eax <=====
Code; d88891f0 <[sunrpc]rpc_shutdown_client+10/80>
2: 85 c0 test %eax,%eax
Code; d88891f2 <[sunrpc]rpc_shutdown_client+12/80>
4: 74 29 je 2f <_EIP+0x2f>
Code; d88891f4 <[sunrpc]rpc_shutdown_client+14/80>
6: f6 05 2c 89 89 d8 02 testb $0x2,0xd889892c
Code; d88891fb <[sunrpc]rpc_shutdown_client+1b/80>
d: 75 29 jne 38 <_EIP+0x38>
Code; d88891fd <[sunrpc]rpc_shutdown_client+1d/80>
f: 80 63 20 3f andb $0x3f,0x20(%ebx)
Code; d8889201 <[sunrpc]rpc_shutdown_client+21/80>
13: 53 push %ebx

1 warning issued. Results may not be reliable.

#2 lore raw

Without a tcpdump, there's no way to know why this dumped. In what way was the NFS server broken, and exactly how do you expect the Linux NFS client to protect you against it?

Cheers,
Trond

#3 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Trond Myklebust <[email protected]> writes:

> Without a tcpdump, there's no way to know why this dumped. In > what way was the NFS server broken, and exactly how do you > expect the Linux NFS client to protect you against it?

BTW: that Oops you posted looked very much like a memory corruption problem. Were you running vanilla 2.4.22 on the client, or was it too patched?

Cheers,
Trond

#4 lore raw
In reply to Trond Myklebust
Quoted 3 lines · Trond Myklebust

Without a tcpdump, there's no way to know why this dumped. In what way was the NFS server broken, and exactly how do you expect the Linux NFS client to protect you against it?

The NFS server was unable to read the directories it was exporting, so it would allow the mount command but then fail to do anything.

I expect the NFS client to behave sensibly in the face of all possible server errors, including the possibility of a hostile NFS server.

Quoted 3 lines

BTW: that Oops you posted looked very much like a memory corruption problem. Were you running vanilla 2.4.22 on the client, or was it too patched?

It was also patched. I will try and reproduce the error with an unpatched kernel and a tcpdump running.

Signature · 4 lines
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page
#5 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Russell Coker <[email protected]> writes:

> I expect the NFS client to behave sensibly in the face of all > possible server errors, including the possibility of a hostile > NFS server.

No can do... There are 100s of scenarios where the server can screw the client by giving it bogus information. You might possibly be able to protect against a few of them, but at a heavy price in the form of code bloat.
Neither NFSv2 nor NFSv3 are protocols that were designed to operate safely in a hostile environment. They were designed for LANs where servers and clients trust one another.

I agree that we shouldn't Oops though.

Quoted 3 lines
Quoted 3 lines

BTW: that Oops you posted looked very much like a memory corruption problem. Were you running vanilla 2.4.22 on the client, or was it too patched?

> It was also patched. I will try and reproduce the error with > an unpatched kernel and a tcpdump running.

Please do...

Cheers,
Trond