Issues with a rather unusual configured NFS server

PATCH Toralf Förster 7 participants 23 messages Aug 11, 2013 – Oct 2, 2013

Subsystems (3)

NFS client NFS server User-mode Linux (UML)

lore

Share & tools

lore

Toralf Förster

author

2013-08-11 09:48 UTC #0 lore raw

so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.

It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used

I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727

Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400

nfsd4: shut down more of delegation earlier

to be the one after which the user mode linux server crashes with a back trace like this:

$ cat /mnt/ramdisk/bt.v3.11-rc4-172-g8ae3f1d
[New LWP 14025]
Core was generated by `/home/tfoerste/devel/linux/linux earlyprintk ubda=/home/tfoerste/virtual/uml/tr'.
Program terminated with signal 6, Aborted.
#0  0xb77ef424 in __kernel_vsyscall ()
#0  0xb77ef424 in __kernel_vsyscall ()
#1  0x083a33c5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85a1518 <panic_exit_notifier>, unused1=0, unused2=0x85d6ce0 <buf.15904>) at arch/um/kernel/um_arch.c:240
#5  0x0809a3b8 in notifier_call_chain (nl=0x0, val=0, v=0x85d6ce0 <buf.15904>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a503 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85d6cc4 <panic_notifier_list>, val=0, v=0x85d6ce0 <buf.15904>) at kernel/notifier.c:191
#8  0x08400ba8 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x0818edf4 in ext4_put_super (sb=0x4a042690) at fs/ext4/super.c:818
#10 0x081010d2 in generic_shutdown_super (sb=0x4a042690) at fs/super.c:418
#11 0x0810209a in kill_block_super (sb=0x0) at fs/super.c:1028
#12 0x08100f6a in deactivate_locked_super (s=0x4a042690) at fs/super.c:299
#13 0x08101001 in deactivate_super (s=0x4a042690) at fs/super.c:324
#14 0x08118e0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#15 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#16 0x0811a2f5 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#17 SyS_umount (name=134541632, flags=0) at fs/namespace.c:1305
#18 0x0811a369 in SYSC_oldumount (name=<optimized out>) at fs/namespace.c:1347
#19 SyS_oldumount (name=134541632) at fs/namespace.c:1345
#20 0x080618e2 in handle_syscall (r=0x49e919d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198
#22 userspace (regs=0x49e919d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()

A real system however would not crash bug would give a kernel BUG as reported here:
http://article.gmane.org/gmane.comp.file-systems.ext4/38915
Furthermore the server won't be able any longer to reboot - it would hang infinitely in the reboot phase. Just the magic sysrq keys still works then.

Steps to reproduce at two 32 bit Gentoo Linux user mode linux images:

1. prepare the server :
<mount a tempfs onto /mnt/ramdisk>
mkdir /mnt/ramdisk/victims
dd if=/dev/zero of=/mnt/ramdisk/disk1 bs=1M count=257 2>/dev/null yes | mkfs.ext4 -q /mnt/ramdisk/disk1 1>/dev/null mount -o loop /mnt/ramdisk/disk1 /mnt/ramdisk/victims chmod 777 /mnt/ramdisk/victims
/etc/init.d/nfs restart

2. prepare the client
mount the NFS share onto the local mount point /mnt/ramdisk/victims/ with NFSv4

3. run the fuzzy tool trinity at the client:

	while [[ : ]]; do
		<(re-)create and fill /mnt/ramdisk/victims/v1/v2 with 100 empty files and 100 empty directories>
		trinity -V /mnt/ramdisk/victims/v1/v2 -C 1 -N 10000 -q
		sleep 3
	done

4. after 15 min kill the user mode linux client with -9

5. now run at the server

	umount /mnt/ramdisk/victims || /etc/init.d/nfs restart && umount /mnt/ramdisk/victims && echo ' no issue so far'

You might need this patch too from Oleg Nesterov <[email protected]> (not in mainline currently) .

kernel/exit.c +1 -1

@@ -783,8 +783,8 @@ void do_exit(long code)
        exit_shm(tsk);
        exit_files(tsk);
        exit_fs(tsk);
-       exit_task_namespaces(tsk);
        exit_task_work(tsk);
+       exit_task_namespaces(tsk);
        check_stack_usage();
        exit_thread();

Signature · 7 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jan Kara

2013-08-12 14:36 UTC #1 lore raw

Quoted 10 lines · Toralf Förster

so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used

Quoted 6 lines

I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier

Added Bruce to CC.

Quoted 36 lines

to be the one after which the user mode linux server crashes with a back trace like this:

$ cat /mnt/ramdisk/bt.v3.11-rc4-172-g8ae3f1d
[New LWP 14025]
Core was generated by `/home/tfoerste/devel/linux/linux earlyprintk ubda=/home/tfoerste/virtual/uml/tr'.
Program terminated with signal 6, Aborted.
#0  0xb77ef424 in __kernel_vsyscall ()
#0  0xb77ef424 in __kernel_vsyscall ()
#1  0x083a33c5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85a1518 <panic_exit_notifier>, unused1=0, unused2=0x85d6ce0 <buf.15904>) at arch/um/kernel/um_arch.c:240
#5  0x0809a3b8 in notifier_call_chain (nl=0x0, val=0, v=0x85d6ce0 <buf.15904>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a503 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85d6cc4 <panic_notifier_list>, val=0, v=0x85d6ce0 <buf.15904>) at kernel/notifier.c:191
#8  0x08400ba8 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x0818edf4 in ext4_put_super (sb=0x4a042690) at fs/ext4/super.c:818
#10 0x081010d2 in generic_shutdown_super (sb=0x4a042690) at fs/super.c:418
#11 0x0810209a in kill_block_super (sb=0x0) at fs/super.c:1028
#12 0x08100f6a in deactivate_locked_super (s=0x4a042690) at fs/super.c:299
#13 0x08101001 in deactivate_super (s=0x4a042690) at fs/super.c:324
#14 0x08118e0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#15 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#16 0x0811a2f5 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#17 SyS_umount (name=134541632, flags=0) at fs/namespace.c:1305
#18 0x0811a369 in SYSC_oldumount (name=<optimized out>) at fs/namespace.c:1347
#19 SyS_oldumount (name=134541632) at fs/namespace.c:1345
#20 0x080618e2 in handle_syscall (r=0x49e919d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198
#22 userspace (regs=0x49e919d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()

Quoted 2 lines

A real system however would not crash bug would give a kernel BUG as reported here:
http://article.gmane.org/gmane.comp.file-systems.ext4/38915

We have deleted inodes (regular files) in the orphan list during
ext4_put_super(). My guess is that NFS is still holding some inode references to these inodes and thus inodes don't get deleted. So ext3/4 would be just a victim here.

Quoted 3 lines

Furthermore the server won't be able any longer to reboot - it would hang infinitely in the reboot phase. Just the magic sysrq keys still works then.

Well, this is likely because the filesystem cannot be shut down.

Honza

Signature · 6 lines

Jan Kara <[email protected]>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

J. Bruce Fields

2013-08-13 21:53 UTC #2 lore raw

Quoted 11 lines · Jan Kara

Quoted 10 lines · Toralf Förster
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used

Quoted 6 lines

Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier

Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().

Let me think about how to fix that....

--b.

Quoted 58 lines

Added Bruce to CC.

Quoted 36 lines

to be the one after which the user mode linux server crashes with a back trace like this:

$ cat /mnt/ramdisk/bt.v3.11-rc4-172-g8ae3f1d
[New LWP 14025]
Core was generated by `/home/tfoerste/devel/linux/linux earlyprintk ubda=/home/tfoerste/virtual/uml/tr'.
Program terminated with signal 6, Aborted.
#0  0xb77ef424 in __kernel_vsyscall ()
#0  0xb77ef424 in __kernel_vsyscall ()
#1  0x083a33c5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85a1518 <panic_exit_notifier>, unused1=0, unused2=0x85d6ce0 <buf.15904>) at arch/um/kernel/um_arch.c:240
#5  0x0809a3b8 in notifier_call_chain (nl=0x0, val=0, v=0x85d6ce0 <buf.15904>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a503 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85d6cc4 <panic_notifier_list>, val=0, v=0x85d6ce0 <buf.15904>) at kernel/notifier.c:191
#8  0x08400ba8 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x0818edf4 in ext4_put_super (sb=0x4a042690) at fs/ext4/super.c:818
#10 0x081010d2 in generic_shutdown_super (sb=0x4a042690) at fs/super.c:418
#11 0x0810209a in kill_block_super (sb=0x0) at fs/super.c:1028
#12 0x08100f6a in deactivate_locked_super (s=0x4a042690) at fs/super.c:299
#13 0x08101001 in deactivate_super (s=0x4a042690) at fs/super.c:324
#14 0x08118e0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#15 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#16 0x0811a2f5 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#17 SyS_umount (name=134541632, flags=0) at fs/namespace.c:1305
#18 0x0811a369 in SYSC_oldumount (name=<optimized out>) at fs/namespace.c:1347
#19 SyS_oldumount (name=134541632) at fs/namespace.c:1345
#20 0x080618e2 in handle_syscall (r=0x49e919d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198
#22 userspace (regs=0x49e919d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()

Quoted 2 lines

A real system however would not crash bug would give a kernel BUG as reported here:
http://article.gmane.org/gmane.comp.file-systems.ext4/38915

Quoted 3 lines

Furthermore the server won't be able any longer to reboot - it would hang infinitely in the reboot phase. Just the magic sysrq keys still works then.

Well, this is likely because the filesystem cannot be shut down.

Honza

Signature · 6 lines

Jan Kara <[email protected]>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Toralf Förster

author

2013-08-14 16:44 UTC #3 lore raw

Quoted 19 lines · J. Bruce Fields

Quoted 11 lines · Jan Kara
Quoted 10 lines · Toralf Förster
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier

Quoted 53 lines

Let me think about how to fix that....

--b.

Quoted 44 lines

Added Bruce to CC.

Quoted 36 lines

to be the one after which the user mode linux server crashes with a back trace like this:

$ cat /mnt/ramdisk/bt.v3.11-rc4-172-g8ae3f1d
[New LWP 14025]
Core was generated by `/home/tfoerste/devel/linux/linux earlyprintk ubda=/home/tfoerste/virtual/uml/tr'.
Program terminated with signal 6, Aborted.
#0  0xb77ef424 in __kernel_vsyscall ()
#0  0xb77ef424 in __kernel_vsyscall ()
#1  0x083a33c5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85a1518 <panic_exit_notifier>, unused1=0, unused2=0x85d6ce0 <buf.15904>) at arch/um/kernel/um_arch.c:240
#5  0x0809a3b8 in notifier_call_chain (nl=0x0, val=0, v=0x85d6ce0 <buf.15904>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a503 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85d6cc4 <panic_notifier_list>, val=0, v=0x85d6ce0 <buf.15904>) at kernel/notifier.c:191
#8  0x08400ba8 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x0818edf4 in ext4_put_super (sb=0x4a042690) at fs/ext4/super.c:818
#10 0x081010d2 in generic_shutdown_super (sb=0x4a042690) at fs/super.c:418
#11 0x0810209a in kill_block_super (sb=0x0) at fs/super.c:1028
#12 0x08100f6a in deactivate_locked_super (s=0x4a042690) at fs/super.c:299
#13 0x08101001 in deactivate_super (s=0x4a042690) at fs/super.c:324
#14 0x08118e0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#15 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#16 0x0811a2f5 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#17 SyS_umount (name=134541632, flags=0) at fs/namespace.c:1305
#18 0x0811a369 in SYSC_oldumount (name=<optimized out>) at fs/namespace.c:1347
#19 SyS_oldumount (name=134541632) at fs/namespace.c:1345
#20 0x080618e2 in handle_syscall (r=0x49e919d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198
#22 userspace (regs=0x49e919d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()

Quoted 2 lines

A real system however would not crash bug would give a kernel BUG as reported here:
http://article.gmane.org/gmane.comp.file-systems.ext4/38915

Just FWIW, EXT2 is not affected.

Quoted 14 lines

Quoted 13 lines
Quoted 3 lines
Furthermore the server won't be able any longer to reboot - it would hang infinitely in the reboot phase. Just the magic sysrq keys still works then.
Well, this is likely because the filesystem cannot be shut down.
Honza
Signature · 6 lines
Jan Kara <[email protected]>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

J. Bruce Fields

2013-08-27 18:06 UTC #4 lore raw

In reply to J. Bruce Fields

Quoted 17 lines · bfields

Quoted 16 lines · Jan Kara
Quoted 15 lines · Toralf Förster
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400

Quoted 8 lines

Quoted 1 line
Quoted 1 line
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Let me think about how to fix that....

Sorry for the slow response--can you check whether this fixes the problem?

--b.

commit 624a0ee0375940ce4aa36330b0b5a70af6d2b6f5

Author: J. Bruce Fields <[email protected]>
Date:   Thu Aug 15 16:55:26 2013 -0400

nfsd4: fix leak of inode reference on delegation failure

This fixes a regression from 68a3396178e6688ad7367202cdf0af8ed03c8727 "nfsd4: shut down more of delegation earlier".

After that commit, nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().

This can result in an oops on later unmounting the exported filesystem.

Note also delaying the fi_had_conflict check we're able to return a better error (hence give 4.1 clients a better idea why the delegation failed; though note CONFLICT isn't an exact match here, as that's supposed to indicate a current conflict, but all we know here is that there was one recently).

Reported-by Toralf Förster <[email protected]>

Signed-off-by J. Bruce Fields <[email protected]>

fs/nfsd/nfs4state.c +20 ~~-11~~

index eb9cf81..0874998 100644
@@ -368,11 +368,8 @@ static struct nfs4_delegation *
 alloc_init_deleg(struct nfs4_client *clp, struct nfs4_ol_stateid *stp, struct svc_fh *current_fh)
 {
 	struct nfs4_delegation *dp;
-	struct nfs4_file *fp = stp->st_file;
 
 	dprintk("NFSD alloc_init_deleg\n");
-	if (fp->fi_had_conflict)
-		return NULL;
 	if (num_delegations > max_delegations)
 		return NULL;
 	dp = delegstateid(nfs4_alloc_stid(clp, deleg_slab));

@@ -389,8 +386,7 @@ alloc_init_deleg(struct nfs4_client *clp, struct nfs4_ol_stateid *stp, struct sv
 	INIT_LIST_HEAD(&dp->dl_perfile);
 	INIT_LIST_HEAD(&dp->dl_perclnt);
 	INIT_LIST_HEAD(&dp->dl_recall_lru);
-	get_nfs4_file(fp);
-	dp->dl_file = fp;
+	dp->dl_file = NULL;
 	dp->dl_type = NFS4_OPEN_DELEGATE_READ;
 	fh_copy_shallow(&dp->dl_fh, &current_fh->fh_handle);
 	dp->dl_time = 0;

@@ -3044,22 +3040,35 @@ static int nfs4_setlease(struct nfs4_delegation *dp)
 	return 0;
 }
 
-static int nfs4_set_delegation(struct nfs4_delegation *dp)
+static int nfs4_set_delegation(struct nfs4_delegation *dp, struct nfs4_file *fp)
 {
-	struct nfs4_file *fp = dp->dl_file;
+	int status;
 
-	if (!fp->fi_lease)
-		return nfs4_setlease(dp);
+	if (fp->fi_had_conflict)
+		return -EAGAIN;
+	get_nfs4_file(fp);
+	dp->dl_file = fp;
+	if (!fp->fi_lease) {
+		status = nfs4_setlease(dp);
+		if (status)
+			goto out_free;
+		return 0;
+	}
 	spin_lock(&recall_lock);
 	if (fp->fi_had_conflict) {
 		spin_unlock(&recall_lock);
-		return -EAGAIN;
+		status = -EAGAIN;
+		goto out_free;
 	}
 	atomic_inc(&fp->fi_delegees);
 	list_add(&dp->dl_perfile, &fp->fi_delegations);
 	spin_unlock(&recall_lock);
 	list_add(&dp->dl_perclnt, &dp->dl_stid.sc_client->cl_delegations);
 	return 0;
+out_free:
+	put_nfs4_file(fp);
+	dp->dl_file = fp;
+	return status;
 }
 
 static void nfsd4_open_deleg_none_ext(struct nfsd4_open *open, int status)

@@ -3134,7 +3143,7 @@ nfs4_open_delegation(struct net *net, struct svc_fh *fh,
 	dp = alloc_init_deleg(oo->oo_owner.so_client, stp, fh);
 	if (dp == NULL)
 		goto out_no_deleg;
-	status = nfs4_set_delegation(dp);
+	status = nfs4_set_delegation(dp, stp->st_file);
 	if (status)
 		goto out_free;

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Toralf Förster

author

2013-08-28 17:21 UTC #5 lore raw

Quoted 25 lines · J. Bruce Fields

Quoted 24 lines · bfields
Quoted 11 lines · Jan Kara
Quoted 10 lines · Toralf Förster
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().

Quoted 5 lines

Quoted 1 line
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?

Yes.

With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.

Quoted 103 lines

--b.

commit 624a0ee0375940ce4aa36330b0b5a70af6d2b6f5

Author: J. Bruce Fields <[email protected]>
Date:   Thu Aug 15 16:55:26 2013 -0400

nfsd4: fix leak of inode reference on delegation failure

This fixes a regression from 68a3396178e6688ad7367202cdf0af8ed03c8727 "nfsd4: shut down more of delegation earlier".

This can result in an oops on later unmounting the exported filesystem.

Reported-by Toralf Förster <[email protected]>

Signed-off-by J. Bruce Fields <[email protected]>

fs/nfsd/nfs4state.c +20 ~~-11~~

index eb9cf81..0874998 100644
@@ -368,11 +368,8 @@ static struct nfs4_delegation *
 alloc_init_deleg(struct nfs4_client *clp, struct nfs4_ol_stateid *stp, struct svc_fh *current_fh)
 {
 	struct nfs4_delegation *dp;
-	struct nfs4_file *fp = stp->st_file;
 
 	dprintk("NFSD alloc_init_deleg\n");
-	if (fp->fi_had_conflict)
-		return NULL;
 	if (num_delegations > max_delegations)
 		return NULL;
 	dp = delegstateid(nfs4_alloc_stid(clp, deleg_slab));

@@ -389,8 +386,7 @@ alloc_init_deleg(struct nfs4_client *clp, struct nfs4_ol_stateid *stp, struct sv
 	INIT_LIST_HEAD(&dp->dl_perfile);
 	INIT_LIST_HEAD(&dp->dl_perclnt);
 	INIT_LIST_HEAD(&dp->dl_recall_lru);
-	get_nfs4_file(fp);
-	dp->dl_file = fp;
+	dp->dl_file = NULL;
 	dp->dl_type = NFS4_OPEN_DELEGATE_READ;
 	fh_copy_shallow(&dp->dl_fh, &current_fh->fh_handle);
 	dp->dl_time = 0;

@@ -3044,22 +3040,35 @@ static int nfs4_setlease(struct nfs4_delegation *dp)
 	return 0;
 }
 
-static int nfs4_set_delegation(struct nfs4_delegation *dp)
+static int nfs4_set_delegation(struct nfs4_delegation *dp, struct nfs4_file *fp)
 {
-	struct nfs4_file *fp = dp->dl_file;
+	int status;
 
-	if (!fp->fi_lease)
-		return nfs4_setlease(dp);
+	if (fp->fi_had_conflict)
+		return -EAGAIN;
+	get_nfs4_file(fp);
+	dp->dl_file = fp;
+	if (!fp->fi_lease) {
+		status = nfs4_setlease(dp);
+		if (status)
+			goto out_free;
+		return 0;
+	}
 	spin_lock(&recall_lock);
 	if (fp->fi_had_conflict) {
 		spin_unlock(&recall_lock);
-		return -EAGAIN;
+		status = -EAGAIN;
+		goto out_free;
 	}
 	atomic_inc(&fp->fi_delegees);
 	list_add(&dp->dl_perfile, &fp->fi_delegations);
 	spin_unlock(&recall_lock);
 	list_add(&dp->dl_perclnt, &dp->dl_stid.sc_client->cl_delegations);
 	return 0;
+out_free:
+	put_nfs4_file(fp);
+	dp->dl_file = fp;
+	return status;
 }
 
 static void nfsd4_open_deleg_none_ext(struct nfsd4_open *open, int status)

@@ -3134,7 +3143,7 @@ nfs4_open_delegation(struct net *net, struct svc_fh *fh,
 	dp = alloc_init_deleg(oo->oo_owner.so_client, stp, fh);
 	if (dp == NULL)
 		goto out_no_deleg;
-	status = nfs4_set_delegation(dp);
+	status = nfs4_set_delegation(dp, stp->st_file);
 	if (status)
 		goto out_free;

Signature · 3 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

richard -rw- weinberger

2013-08-29 09:57 UTC #6 lore raw

Quoted 28 lines · Toralf Förster

Quoted 27 lines · J. Bruce Fields
Quoted 25 lines · bfields
Quoted 11 lines · Jan Kara
Quoted 10 lines · Toralf Förster
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Let me think about how to fix that....

Quoted 7 lines

Quoted 3 lines
Sorry for the slow response--can you check whether this fixes the problem?
Yes.
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.

BTW: Is nobody else fuzz testing NFS?
Or are these bugs just more likely to hit on UML?
This is not the first NFS issue found by Toralf using UML and Trinity.

Signature · 2 lines

Thanks,
//richard

J. Bruce Fields

2013-08-29 13:30 UTC #7 lore raw

Quoted 32 lines · richard -rw- weinberger

Quoted 31 lines · Toralf Förster
Quoted 30 lines · J. Bruce Fields
Quoted 17 lines · bfields
Quoted 16 lines · Jan Kara
Quoted 15 lines · Toralf Förster
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
Quoted 8 lines
Quoted 1 line
Quoted 1 line
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?

Quoted 6 lines

Quoted 4 lines
Yes.
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.
BTW: Is nobody else fuzz testing NFS?

I don't know. Toralf's reports are the only ones I recall off the top of my head, but I may have forgotten others.

Quoted 1 line

Or are these bugs just more likely to hit on UML?

That's also possible.

Quoted 1 line

This is not the first NFS issue found by Toralf using UML and Trinity.

Yep. The testing is definitely appreciated.

--b.

Toralf Förster

author

2013-08-30 14:10 UTC #8 lore raw

Quoted 35 lines · J. Bruce Fields

Quoted 34 lines · richard -rw- weinberger
Quoted 33 lines · Toralf Förster
Quoted 25 lines · J. Bruce Fields
Quoted 24 lines · bfields
Quoted 11 lines · Jan Kara
Quoted 10 lines · Toralf Förster
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Quoted 5 lines
Quoted 1 line
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?
Yes.

Quoted 8 lines

Quoted 4 lines
Quoted 2 lines
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.
BTW: Is nobody else fuzz testing NFS?
I don't know. Toralf's reports are the only ones I recall off the top of my head, but I may have forgotten others.

well, 7255e71 and 3c50ba8 I'd say.

Quoted 1 line

Quoted 1 line
Or are these bugs just more likely to hit on UML?

This definitely not. I observed at a real system EXT4 corruptions/ issues but reported them to the EXT4 mailing list.
It just took me a longer time to figure out a reliable configuration with 2 UML machiens to automatic bisect it.

Quoted 5 lines

That's also possible.
Quoted 1 line
This is not the first NFS issue found by Toralf using UML and Trinity.
Yep. The testing is definitely appreciated.

Thx - in the mean while although my UML bisect scripts are working fine and trinity is stable enough even in UML environments to be trust worth.

Quoted 3 lines

--b.

Signature · 7 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

J. Bruce Fields

2013-08-30 14:25 UTC #9 lore raw

Quoted 39 lines · Toralf Förster

Quoted 38 lines · J. Bruce Fields
Quoted 37 lines · richard -rw- weinberger
Quoted 35 lines · Toralf Förster
Quoted 25 lines · J. Bruce Fields
Quoted 24 lines · bfields
Quoted 11 lines · Jan Kara
Quoted 10 lines · Toralf Förster
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Quoted 5 lines
Quoted 1 line
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?
Yes.
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.

Quoted 7 lines

Quoted 5 lines
Quoted 1 line
BTW: Is nobody else fuzz testing NFS?
I don't know. Toralf's reports are the only ones I recall off the top of my head, but I may have forgotten others.
well, 7255e71 and 3c50ba8 I'd say.

I don't know any 3c50ba8. 0c7c3e67 "nfsd4: don't close read-write opens too soon" and 64a817cf "nfsd4: reject "negative" acl lengths" are two other serious bugs found by your testing. I don't recall fuzz testing results from anyone else, but as I say I may have forgotten.

--b.

Richard Weinberger

maintainer

2013-08-30 14:36 UTC #10 lore raw

In reply to Toralf Förster

Am 30.08.2013 16:10, schrieb Toralf Förster:

Quoted 55 lines · J. Bruce Fields

Quoted 35 lines · richard -rw- weinberger
Quoted 34 lines · Toralf Förster
Quoted 33 lines · J. Bruce Fields
Quoted 25 lines · bfields
Quoted 24 lines · Jan Kara
Quoted 11 lines · Toralf Förster
Quoted 10 lines
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Quoted 5 lines
Quoted 1 line
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?
Yes.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.
BTW: Is nobody else fuzz testing NFS?
I don't know. Toralf's reports are the only ones I recall off the top of my head, but I may have forgotten others.
well, 7255e71 and 3c50ba8 I'd say.
Quoted 1 line
Quoted 1 line
Or are these bugs just more likely to hit on UML?
This definitely not. I observed at a real system EXT4 corruptions/ issues but reported them to the EXT4 mailing list.
It just took me a longer time to figure out a reliable configuration with 2 UML machiens to automatic bisect it.

Quoted 8 lines

Quoted 5 lines
That's also possible.
Quoted 1 line
This is not the first NFS issue found by Toralf using UML and Trinity.
Yep. The testing is definitely appreciated.
Thx - in the mean while although my UML bisect scripts are working fine and trinity is stable enough even in UML environments to be trust worth.

That's good to know.
Thanks you and trinity we got rid of some nasty UML bugs.

Thanks,
//richard

Toralf Förster

author

2013-09-01 16:09 UTC #11 lore raw

Quoted 60 lines · Richard Weinberger

Am 30.08.2013 16:10, schrieb Toralf Förster:
Quoted 59 lines · J. Bruce Fields
Quoted 35 lines · richard -rw- weinberger
Quoted 34 lines · Toralf Förster
Quoted 33 lines · J. Bruce Fields
Quoted 25 lines · bfields
Quoted 24 lines · Jan Kara
Quoted 11 lines · Toralf Förster
Quoted 10 lines
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Quoted 5 lines
Quoted 1 line
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?
Yes.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.
BTW: Is nobody else fuzz testing NFS?
I don't know. Toralf's reports are the only ones I recall off the top of my head, but I may have forgotten others.
well, 7255e71 and 3c50ba8 I'd say.
Quoted 1 line
Quoted 1 line
Or are these bugs just more likely to hit on UML?
This definitely not. I observed at a real system EXT4 corruptions/ issues but reported them to the EXT4 mailing list.
It just took me a longer time to figure out a reliable configuration with 2 UML machiens to automatic bisect it.
Quoted 4 lines
That's also possible.
Quoted 1 line
This is not the first NFS issue found by Toralf using UML and Trinity.

Quoted 7 lines

Quoted 4 lines
Quoted 1 line
Yep. The testing is definitely appreciated.
Thx - in the mean while although my UML bisect scripts are working fine and trinity is stable enough even in UML environments to be trust worth.
That's good to know.
Thanks you and trinity we got rid of some nasty UML bugs.

yeah, but there's (among others) one of them still in the wild:
If I start an UML and resize the terminal (KDE Konsole) during the boot phase then it just crashes sometimes.

Quoted 4 lines

Thanks,
//richard

Signature · 14 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Richard Weinberger

maintainer

2013-09-01 21:15 UTC #12 lore raw

Am 01.09.2013 18:09, schrieb Toralf Förster:

Quoted 66 lines · Richard Weinberger

Quoted 65 lines
Am 30.08.2013 16:10, schrieb Toralf Förster:
Quoted 63 lines · J. Bruce Fields
Quoted 35 lines · richard -rw- weinberger
Quoted 34 lines · Toralf Förster
Quoted 33 lines · J. Bruce Fields
Quoted 25 lines · bfields
Quoted 24 lines · Jan Kara
Quoted 11 lines · Toralf Förster
Quoted 10 lines
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Quoted 5 lines
Quoted 1 line
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?
Yes.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.
BTW: Is nobody else fuzz testing NFS?
I don't know. Toralf's reports are the only ones I recall off the top of my head, but I may have forgotten others.
well, 7255e71 and 3c50ba8 I'd say.
Quoted 1 line
Quoted 1 line
Or are these bugs just more likely to hit on UML?
This definitely not. I observed at a real system EXT4 corruptions/ issues but reported them to the EXT4 mailing list.
It just took me a longer time to figure out a reliable configuration with 2 UML machiens to automatic bisect it.
Quoted 5 lines
That's also possible.
Quoted 1 line
This is not the first NFS issue found by Toralf using UML and Trinity.
Yep. The testing is definitely appreciated.
Thx - in the mean while although my UML bisect scripts are working fine and trinity is stable enough even in UML environments to be trust worth.

Quoted 6 lines

Quoted 2 lines
That's good to know.
Thanks you and trinity we got rid of some nasty UML bugs.
yeah, but there's (among others) one of them still in the wild:
If I start an UML and resize the terminal (KDE Konsole) during the boot phase then it just crashes sometimes.

Does the attached patch fix the problem?

Thanks,
//richard

arch/um/kernel/irq.c +1 -0

index 36e12f0..e7d1983 100644
@@ -337,6 +337,7 @@ static struct irq_chip normal_irq_type = {
 	.irq_disable = dummy,
 	.irq_enable = dummy,
 	.irq_ack = dummy,
+	.irq_mask = dummy,
 };
 
 static struct irq_chip SIGVTALRM_irq_type = {

------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk

_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Toralf Förster

author

2013-09-02 16:53 UTC #13 lore raw

Quoted 67 lines · Richard Weinberger

Am 01.09.2013 18:09, schrieb Toralf Förster:
Quoted 66 lines · Richard Weinberger
Quoted 65 lines
Am 30.08.2013 16:10, schrieb Toralf Förster:
Quoted 63 lines · J. Bruce Fields
Quoted 35 lines · richard -rw- weinberger
Quoted 34 lines · Toralf Förster
Quoted 33 lines · J. Bruce Fields
Quoted 25 lines · bfields
Quoted 24 lines · Jan Kara
Quoted 11 lines · Toralf Förster
Quoted 10 lines
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Quoted 5 lines
Quoted 1 line
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?
Yes.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.
BTW: Is nobody else fuzz testing NFS?
I don't know. Toralf's reports are the only ones I recall off the top of my head, but I may have forgotten others.
well, 7255e71 and 3c50ba8 I'd say.
Quoted 1 line
Quoted 1 line
Or are these bugs just more likely to hit on UML?
This definitely not. I observed at a real system EXT4 corruptions/ issues but reported them to the EXT4 mailing list.
It just took me a longer time to figure out a reliable configuration with 2 UML machiens to automatic bisect it.
Quoted 5 lines
That's also possible.
Quoted 1 line
This is not the first NFS issue found by Toralf using UML and Trinity.
Yep. The testing is definitely appreciated.
Thx - in the mean while although my UML bisect scripts are working fine and trinity is stable enough even in UML environments to be trust worth.

Quoted 9 lines

Quoted 6 lines
Quoted 2 lines
That's good to know.
Thanks you and trinity we got rid of some nasty UML bugs.
yeah, but there's (among others) one of them still in the wild:
If I start an UML and resize the terminal (KDE Konsole) during the boot phase then it just crashes sometimes.
Does the attached patch fix the problem?

oh yes
And I do have the feeling that the UML start phase till the Gentoo openrc line is seen is speed up too :-)

Quoted 3 lines

Thanks,
//richard

Signature · 14 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Richard Weinberger

maintainer

2013-09-02 16:57 UTC #14 lore raw

Am 02.09.2013 18:53, schrieb Toralf Förster:

Quoted 75 lines · Richard Weinberger

Quoted 74 lines
Am 01.09.2013 18:09, schrieb Toralf Förster:
Quoted 72 lines · Richard Weinberger
Quoted 60 lines
Am 30.08.2013 16:10, schrieb Toralf Förster:
Quoted 59 lines · J. Bruce Fields
Quoted 35 lines · richard -rw- weinberger
Quoted 34 lines · Toralf Förster
Quoted 33 lines · J. Bruce Fields
Quoted 25 lines · bfields
Quoted 24 lines · Jan Kara
Quoted 11 lines · Toralf Förster
Quoted 10 lines
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Quoted 5 lines
Quoted 1 line
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?
Yes.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.
BTW: Is nobody else fuzz testing NFS?
I don't know. Toralf's reports are the only ones I recall off the top of my head, but I may have forgotten others.
well, 7255e71 and 3c50ba8 I'd say.
Quoted 1 line
Quoted 1 line
Or are these bugs just more likely to hit on UML?
This definitely not. I observed at a real system EXT4 corruptions/ issues but reported them to the EXT4 mailing list.
It just took me a longer time to figure out a reliable configuration with 2 UML machiens to automatic bisect it.
Quoted 4 lines
That's also possible.
Quoted 1 line
This is not the first NFS issue found by Toralf using UML and Trinity.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Yep. The testing is definitely appreciated.
Thx - in the mean while although my UML bisect scripts are working fine and trinity is stable enough even in UML environments to be trust worth.
That's good to know.
Thanks you and trinity we got rid of some nasty UML bugs.
yeah, but there's (among others) one of them still in the wild:
If I start an UML and resize the terminal (KDE Konsole) during the boot phase then it just crashes sometimes.

Quoted 5 lines

Quoted 2 lines
Does the attached patch fix the problem?
oh yes
And I do have the feeling that the UML start phase till the Gentoo openrc line is seen is speed up too :-)

Thanks for testing!
Now I'm sure what the problem is, patch is on the way. :)

Thanks,
//richard

Michael Richardson

2013-08-30 18:27 UTC #15 lore raw

In reply to richard -rw- weinberger

Quoted 10 lines · richard -rw- weinberger

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?

Quoted 8 lines

Quoted 4 lines
Yes.
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.
BTW: Is nobody else fuzz testing NFS? Or are these bugs just more likely to hit on UML? This is not the first NFS issue found by Toralf using UML and Trinity.

Kernel thread scheduling is likely very different on UML than other architectures.

My guess is that there could well be be gaps where no kernel thread is scheduled (because another process is running), followed by resumption of a thread other than the one which would have been resumed on other virtualization.

--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works | network architect [
] mcr-SWp7JaYWvAQV+D8aMU/[email protected] http://www.sandelman.ca/ | ruby on rails [

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Toralf Förster

author

2013-09-07 20:44 UTC #16 lore raw

In reply to Toralf Förster

Quoted 128 lines · Toralf Förster

Quoted 25 lines · J. Bruce Fields
Quoted 24 lines · bfields
Quoted 11 lines · Jan Kara
Quoted 10 lines · Toralf Förster
so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken
- if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id.
It can re reproduced, if
- the NFS share is an EXT3 or EXT4 directory
- and it is created at file located at tempfs and mounted via loop device
- and the NFS server is forced to umount the NFS share
- and the server forced to restart the NSF service afterwards
- and trinity is used
Quoted 6 lines
Quoted 6 lines
I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727
Author: J. Bruce Fields <[email protected]>
Date:   Thu Mar 21 11:21:50 2013 -0400
nfsd4: shut down more of delegation earlier
Thanks for the report. I think I see the problem--after this commit nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
Quoted 5 lines
Quoted 1 line
Let me think about how to fix that....
Sorry for the slow response--can you check whether this fixes the problem?
Yes.
With the attached patch the problem can't be reproduced any longer with the prepared test case and current git kernels.
Quoted 92 lines
--b.
commit 624a0ee0375940ce4aa36330b0b5a70af6d2b6f5
Author: J. Bruce Fields <[email protected]>
Date:   Thu Aug 15 16:55:26 2013 -0400
nfsd4: fix leak of inode reference on delegation failure
This fixes a regression from 68a3396178e6688ad7367202cdf0af8ed03c8727 "nfsd4: shut down more of delegation earlier".
After that commit, nfs4_set_delegation() failures result in nfs4_put_delegation being called, but nfs4_put_delegation doesn't free the nfs4_file that has already been set by alloc_init_deleg().
This can result in an oops on later unmounting the exported filesystem.
Note also delaying the fi_had_conflict check we're able to return a better error (hence give 4.1 clients a better idea why the delegation failed; though note CONFLICT isn't an exact match here, as that's supposed to indicate a current conflict, but all we know here is that there was one recently).
Reported-by Toralf Förster <[email protected]>
Signed-off-by J. Bruce Fields <[email protected]>
fs/nfsd/nfs4state.c +19 ~~-10~~
index eb9cf81..0874998 100644
@@ -368,11 +368,8 @@ static struct nfs4_delegation *
 alloc_init_deleg(struct nfs4_client *clp, struct nfs4_ol_stateid *stp, struct svc_fh *current_fh)
 {
 	struct nfs4_delegation *dp;
-	struct nfs4_file *fp = stp->st_file;
 
 	dprintk("NFSD alloc_init_deleg\n");
-	if (fp->fi_had_conflict)
-		return NULL;
 	if (num_delegations > max_delegations)
 		return NULL;
 	dp = delegstateid(nfs4_alloc_stid(clp, deleg_slab));
@@ -389,8 +386,7 @@ alloc_init_deleg(struct nfs4_client *clp, struct nfs4_ol_stateid *stp, struct sv
 	INIT_LIST_HEAD(&dp->dl_perfile);
 	INIT_LIST_HEAD(&dp->dl_perclnt);
 	INIT_LIST_HEAD(&dp->dl_recall_lru);
-	get_nfs4_file(fp);
-	dp->dl_file = fp;
+	dp->dl_file = NULL;
 	dp->dl_type = NFS4_OPEN_DELEGATE_READ;
 	fh_copy_shallow(&dp->dl_fh, &current_fh->fh_handle);
 	dp->dl_time = 0;
@@ -3044,22 +3040,35 @@ static int nfs4_setlease(struct nfs4_delegation *dp)
 	return 0;
 }
 
-static int nfs4_set_delegation(struct nfs4_delegation *dp)
+static int nfs4_set_delegation(struct nfs4_delegation *dp, struct nfs4_file *fp)
 {
-	struct nfs4_file *fp = dp->dl_file;
+	int status;
 
-	if (!fp->fi_lease)
-		return nfs4_setlease(dp);
+	if (fp->fi_had_conflict)
+		return -EAGAIN;
+	get_nfs4_file(fp);
+	dp->dl_file = fp;
+	if (!fp->fi_lease) {
+		status = nfs4_setlease(dp);
+		if (status)
+			goto out_free;
+		return 0;
+	}
 	spin_lock(&recall_lock);
 	if (fp->fi_had_conflict) {
 		spin_unlock(&recall_lock);
-		return -EAGAIN;
+		status = -EAGAIN;
+		goto out_free;
 	}
 	atomic_inc(&fp->fi_delegees);
 	list_add(&dp->dl_perfile, &fp->fi_delegations);
 	spin_unlock(&recall_lock);
 	list_add(&dp->dl_perclnt, &dp->dl_stid.sc_client->cl_delegations);
 	return 0;
+out_free:
+	put_nfs4_file(fp);
+	dp->dl_file = fp;
+	return status;
 }
 

Quoted 13 lines

fs/nfsd/nfs4state.c nfs4_open_delegation(struct net *net, struct svc_fh *fh, · line 3149

Quoted 11 lines

fs/nfsd/nfs4state.c +1 -1

 static void nfsd4_open_deleg_none_ext(struct nfsd4_open *open, int status)

@@ -3134,7 +3143,7 @@ nfs4_open_delegation(struct net *net, struct svc_fh *fh,
 	dp = alloc_init_deleg(oo->oo_owner.so_client, stp, fh);
 	if (dp == NULL)
 		goto out_no_deleg;
-	status = nfs4_set_delegation(dp);
+	status = nfs4_set_delegation(dp, stp->st_file);
 	if (status)
 		goto out_free;

Today I run latest git tree with a patched UML (this patch + one for xterm issues) and got 2 times a core dump when I fuzzy test an UML machine with a nearly identical scenario as already described but just shutdowned both UML images instead of shooting one of it in the head.

I'll probably need time to figure out a test case, but just as a pre-info here's the back trace:

tfoerste@n22 ~ $ gdb --core=/mnt/ramdisk/core /usr/local/bin/linux-v3.11-7550-g768c9d3 -n -batch -ex bt

warning: core file may not match specified executable file.
[New LWP 7470]
[New LWP 7479]
[New LWP 7477]
[New LWP 7478]
Core was generated by `/usr/local/bin/linux-v3.11-7550-g768c9d3 earlyprintk ubda=/home/tfoerste/virtua'. Program terminated with signal 6, Aborted.

#0  0xb77be424 in __kernel_vsyscall ()
#0  0xb77be424 in __kernel_vsyscall ()
#1  0x083aada5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85b1518 <panic_exit_notifier>, unused1=0, unused2=0x85e76e0 <buf.15920>) at arch/um/kernel/um_arch.c:240
#5  0x0809a398 in notifier_call_chain (nl=0x0, val=0, v=0x85e76e0 <buf.15920>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a4e3 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85e76c4 <panic_notifier_list>, val=0, v=0x85e76e0 <buf.15920>) at kernel/notifier.c:191
#8  0x08408628 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x081131c9 in shrink_dcache_for_umount_subtree (dentry=0x428028f0) at fs/dcache.c:941

#10 0x08113948 in shrink_dcache_for_umount (sb=0x463b8000) at fs/dcache.c:1002
#11 0x08101677 in generic_shutdown_super (sb=0x463b8000) at fs/super.c:404
#12 0x08102395 in kill_anon_super (sb=0x0) at fs/super.c:875
#13 0x081d3ff8 in nfs_kill_super (s=0x0) at fs/nfs/super.c:2598
#14 0x0810153a in deactivate_locked_super (s=0x463b8000) at fs/super.c:294
#15 0x081015d1 in deactivate_super (s=0x463b8000) at fs/super.c:319
#16 0x08119c0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#17 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#18 0x0811b195 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#19 SyS_umount (name=134633856, flags=2) at fs/namespace.c:1305
#20 0x080618e2 in handle_syscall (r=0x498be5d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198 #22 userspace (regs=0x498be5d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()

Signature · 8 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

richard -rw- weinberger

2013-09-07 20:51 UTC #17 lore raw

Quoted 15 lines · Toralf Förster

Today I run latest git tree with a patched UML (this patch + one for xterm issues) and got 2 times a core dump when I fuzzy test an UML machine with a nearly identical scenario as already described but just shutdowned both UML images instead of shooting one of it in the head.
I'll probably need time to figure out a test case, but just as a pre-info here's the back trace:
tfoerste@n22 ~ $ gdb --core=/mnt/ramdisk/core /usr/local/bin/linux-v3.11-7550-g768c9d3 -n -batch -ex bt
warning: core file may not match specified executable file.
[New LWP 7470]
[New LWP 7479]
[New LWP 7477]
[New LWP 7478]
Core was generated by `/usr/local/bin/linux-v3.11-7550-g768c9d3 earlyprintk ubda=/home/tfoerste/virtua'. Program terminated with signal 6, Aborted.

Quoted 11 lines

#0  0xb77be424 in __kernel_vsyscall ()
#0  0xb77be424 in __kernel_vsyscall ()
#1  0x083aada5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85b1518 <panic_exit_notifier>, unused1=0, unused2=0x85e76e0 <buf.15920>) at arch/um/kernel/um_arch.c:240
#5  0x0809a398 in notifier_call_chain (nl=0x0, val=0, v=0x85e76e0 <buf.15920>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a4e3 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85e76c4 <panic_notifier_list>, val=0, v=0x85e76e0 <buf.15920>) at kernel/notifier.c:191
#8  0x08408628 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x081131c9 in shrink_dcache_for_umount_subtree (dentry=0x428028f0) at fs/dcache.c:941

That is:

                        if (dentry->d_lockref.count != 0) {
                                printk(KERN_ERR
                                       "BUG: Dentry %p{i=%lx,n=%s}"
                                       " still in use (%d)"
                                       " [unmount of %s %s]\n",
                                       dentry,
                                       dentry->d_inode ?
                                       dentry->d_inode->i_ino : 0UL,
                                       dentry->d_name.name,
                                       dentry->d_lockref.count,
                                       dentry->d_sb->s_type->name,
                                       dentry->d_sb->s_id);
                                BUG();
                        }

Do you also have the console output?

Signature · 6 lines

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

J. Bruce Fields

2013-09-10 14:09 UTC #18 lore raw

In reply to Toralf Förster

Quoted 3 lines · Toralf Förster

Today I run latest git tree with a patched UML (this patch + one for xterm issues) and got 2 times a core dump when I fuzzy test an UML machine with a nearly identical scenario as already described but just shutdowned both UML images instead of shooting one of it in the head.

This is a slightly different test case, so for now it sounds like this could be a preexisting problem, not a regression?

--b.

Quoted 23 lines

I'll probably need time to figure out a test case, but just as a pre-info here's the back trace:
tfoerste@n22 ~ $ gdb --core=/mnt/ramdisk/core /usr/local/bin/linux-v3.11-7550-g768c9d3 -n -batch -ex bt
warning: core file may not match specified executable file.
[New LWP 7470]
[New LWP 7479]
[New LWP 7477]
[New LWP 7478]
Core was generated by `/usr/local/bin/linux-v3.11-7550-g768c9d3 earlyprintk ubda=/home/tfoerste/virtua'. Program terminated with signal 6, Aborted.
#0  0xb77be424 in __kernel_vsyscall ()
#0  0xb77be424 in __kernel_vsyscall ()
#1  0x083aada5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85b1518 <panic_exit_notifier>, unused1=0, unused2=0x85e76e0 <buf.15920>) at arch/um/kernel/um_arch.c:240
#5  0x0809a398 in notifier_call_chain (nl=0x0, val=0, v=0x85e76e0 <buf.15920>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a4e3 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85e76c4 <panic_notifier_list>, val=0, v=0x85e76e0 <buf.15920>) at kernel/notifier.c:191
#8  0x08408628 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x081131c9 in shrink_dcache_for_umount_subtree (dentry=0x428028f0) at fs/dcache.c:941

Quoted 22 lines

#10 0x08113948 in shrink_dcache_for_umount (sb=0x463b8000) at fs/dcache.c:1002
#11 0x08101677 in generic_shutdown_super (sb=0x463b8000) at fs/super.c:404
#12 0x08102395 in kill_anon_super (sb=0x0) at fs/super.c:875
#13 0x081d3ff8 in nfs_kill_super (s=0x0) at fs/nfs/super.c:2598
#14 0x0810153a in deactivate_locked_super (s=0x463b8000) at fs/super.c:294
#15 0x081015d1 in deactivate_super (s=0x463b8000) at fs/super.c:319
#16 0x08119c0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#17 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#18 0x0811b195 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#19 SyS_umount (name=134633856, flags=2) at fs/namespace.c:1305
#20 0x080618e2 in handle_syscall (r=0x498be5d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198 #22 userspace (regs=0x498be5d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()
Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Toralf Förster

author

2013-09-10 15:51 UTC #19 lore raw

Quoted 9 lines · J. Bruce Fields

Quoted 3 lines · Toralf Förster
Today I run latest git tree with a patched UML (this patch + one for xterm issues) and got 2 times a core dump when I fuzzy test an UML machine with a nearly identical scenario as already described but just shutdowned both UML images instead of shooting one of it in the head.
This is a slightly different test case, so for now it sounds like this could be a preexisting problem, not a regression?
--b.

If I found a test case I'll bisect it - but currently it does no longer happen during normal operations w/ current git.#

(
Well, if I kill the NFS client during shutdown, then I get often a core dump with the mentioned back trace but I think this doesn't count :-D )

Quoted 46 lines

Quoted 45 lines
I'll probably need time to figure out a test case, but just as a pre-info here's the back trace:
tfoerste@n22 ~ $ gdb --core=/mnt/ramdisk/core /usr/local/bin/linux-v3.11-7550-g768c9d3 -n -batch -ex bt
warning: core file may not match specified executable file.
[New LWP 7470]
[New LWP 7479]
[New LWP 7477]
[New LWP 7478]
Core was generated by `/usr/local/bin/linux-v3.11-7550-g768c9d3 earlyprintk ubda=/home/tfoerste/virtua'. Program terminated with signal 6, Aborted.
#0  0xb77be424 in __kernel_vsyscall ()
#0  0xb77be424 in __kernel_vsyscall ()
#1  0x083aada5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85b1518 <panic_exit_notifier>, unused1=0, unused2=0x85e76e0 <buf.15920>) at arch/um/kernel/um_arch.c:240
#5  0x0809a398 in notifier_call_chain (nl=0x0, val=0, v=0x85e76e0 <buf.15920>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a4e3 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85e76c4 <panic_notifier_list>, val=0, v=0x85e76e0 <buf.15920>) at kernel/notifier.c:191
#8  0x08408628 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x081131c9 in shrink_dcache_for_umount_subtree (dentry=0x428028f0) at fs/dcache.c:941
#10 0x08113948 in shrink_dcache_for_umount (sb=0x463b8000) at fs/dcache.c:1002
#11 0x08101677 in generic_shutdown_super (sb=0x463b8000) at fs/super.c:404
#12 0x08102395 in kill_anon_super (sb=0x0) at fs/super.c:875
#13 0x081d3ff8 in nfs_kill_super (s=0x0) at fs/nfs/super.c:2598
#14 0x0810153a in deactivate_locked_super (s=0x463b8000) at fs/super.c:294
#15 0x081015d1 in deactivate_super (s=0x463b8000) at fs/super.c:319
#16 0x08119c0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#17 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#18 0x0811b195 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#19 SyS_umount (name=134633856, flags=2) at fs/namespace.c:1305
#20 0x080618e2 in handle_syscall (r=0x498be5d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198 #22 userspace (regs=0x498be5d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()
Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

Signature · 7 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Toralf Förster

author

2013-09-22 16:58 UTC #20 lore raw

In reply to J. Bruce Fields

Quoted 8 lines · J. Bruce Fields

Quoted 3 lines · Toralf Förster
Today I run latest git tree with a patched UML (this patch + one for xterm issues) and got 2 times a core dump when I fuzzy test an UML machine with a nearly identical scenario as already described but just shutdowned both UML images instead of shooting one of it in the head.
This is a slightly different test case, so for now it sounds like this could be a preexisting problem, not a regression?

Yes, pre-existing. I'm trying to bisect it, the issue is already in 7d3107d.

But every attempt to get the first bad commit before that commit id failed till now. I started many attempts between v3.10 and 7d3107d. I could not reproduce that issue 100% in that interval. In that interval either all tested commits doesn't show the issue or the NFS client hangs during the stop of NFS service infinitely - and counting that as a "bad" commit doesn't work.

FWIW a single test case currently takes 2 1/2 hour and even then I do not fully trust the result (except for a bad commit, that's clear.

Quoted 48 lines

--b.
Quoted 45 lines
I'll probably need time to figure out a test case, but just as a pre-info here's the back trace:
tfoerste@n22 ~ $ gdb --core=/mnt/ramdisk/core /usr/local/bin/linux-v3.11-7550-g768c9d3 -n -batch -ex bt
warning: core file may not match specified executable file.
[New LWP 7470]
[New LWP 7479]
[New LWP 7477]
[New LWP 7478]
Core was generated by `/usr/local/bin/linux-v3.11-7550-g768c9d3 earlyprintk ubda=/home/tfoerste/virtua'. Program terminated with signal 6, Aborted.
#0  0xb77be424 in __kernel_vsyscall ()
#0  0xb77be424 in __kernel_vsyscall ()
#1  0x083aada5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85b1518 <panic_exit_notifier>, unused1=0, unused2=0x85e76e0 <buf.15920>) at arch/um/kernel/um_arch.c:240
#5  0x0809a398 in notifier_call_chain (nl=0x0, val=0, v=0x85e76e0 <buf.15920>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a4e3 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85e76c4 <panic_notifier_list>, val=0, v=0x85e76e0 <buf.15920>) at kernel/notifier.c:191
#8  0x08408628 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x081131c9 in shrink_dcache_for_umount_subtree (dentry=0x428028f0) at fs/dcache.c:941
#10 0x08113948 in shrink_dcache_for_umount (sb=0x463b8000) at fs/dcache.c:1002
#11 0x08101677 in generic_shutdown_super (sb=0x463b8000) at fs/super.c:404
#12 0x08102395 in kill_anon_super (sb=0x0) at fs/super.c:875
#13 0x081d3ff8 in nfs_kill_super (s=0x0) at fs/nfs/super.c:2598
#14 0x0810153a in deactivate_locked_super (s=0x463b8000) at fs/super.c:294
#15 0x081015d1 in deactivate_super (s=0x463b8000) at fs/super.c:319
#16 0x08119c0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#17 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#18 0x0811b195 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#19 SyS_umount (name=134633856, flags=2) at fs/namespace.c:1305
#20 0x080618e2 in handle_syscall (r=0x498be5d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198 #22 userspace (regs=0x498be5d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()
Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

Signature · 7 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

J. Bruce Fields

2013-09-23 17:41 UTC #21 lore raw

Quoted 18 lines · Toralf Förster

Quoted 8 lines · J. Bruce Fields
Quoted 3 lines · Toralf Förster
Today I run latest git tree with a patched UML (this patch + one for xterm issues) and got 2 times a core dump when I fuzzy test an UML machine with a nearly identical scenario as already described but just shutdowned both UML images instead of shooting one of it in the head.
This is a slightly different test case, so for now it sounds like this could be a preexisting problem, not a regression?
Yes, pre-existing. I'm trying to bisect it, the issue is already in 7d3107d.
But every attempt to get the first bad commit before that commit id failed till now. I started many attempts between v3.10 and 7d3107d. I could not reproduce that issue 100% in that interval. In that interval either all tested commits doesn't show the issue or the NFS client hangs during the stop of NFS service infinitely - and counting that as a "bad" commit doesn't work.

Quoted 2 lines

FWIW a single test case currently takes 2 1/2 hour and even then I do not fully trust the result (except for a bad commit, that's clear.

Well, we should figure out some other way to narrow down the problem....

I was trying to at least work out where exactly this crash was, but I don't have a commit 768c9d3.

--b.

Quoted 45 lines

Quoted 45 lines
--b.
Quoted 41 lines
I'll probably need time to figure out a test case, but just as a pre-info here's the back trace:
tfoerste@n22 ~ $ gdb --core=/mnt/ramdisk/core /usr/local/bin/linux-v3.11-7550-g768c9d3 -n -batch -ex bt
warning: core file may not match specified executable file.
[New LWP 7470]
[New LWP 7479]
[New LWP 7477]
[New LWP 7478]
Core was generated by `/usr/local/bin/linux-v3.11-7550-g768c9d3 earlyprintk ubda=/home/tfoerste/virtua'. Program terminated with signal 6, Aborted.
#0  0xb77be424 in __kernel_vsyscall ()
#0  0xb77be424 in __kernel_vsyscall ()
#1  0x083aada5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85b1518 <panic_exit_notifier>, unused1=0, unused2=0x85e76e0 <buf.15920>) at arch/um/kernel/um_arch.c:240
#5  0x0809a398 in notifier_call_chain (nl=0x0, val=0, v=0x85e76e0 <buf.15920>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a4e3 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85e76c4 <panic_notifier_list>, val=0, v=0x85e76e0 <buf.15920>) at kernel/notifier.c:191
#8  0x08408628 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x081131c9 in shrink_dcache_for_umount_subtree (dentry=0x428028f0) at fs/dcache.c:941
#10 0x08113948 in shrink_dcache_for_umount (sb=0x463b8000) at fs/dcache.c:1002
#11 0x08101677 in generic_shutdown_super (sb=0x463b8000) at fs/super.c:404
#12 0x08102395 in kill_anon_super (sb=0x0) at fs/super.c:875
#13 0x081d3ff8 in nfs_kill_super (s=0x0) at fs/nfs/super.c:2598
#14 0x0810153a in deactivate_locked_super (s=0x463b8000) at fs/super.c:294
#15 0x081015d1 in deactivate_super (s=0x463b8000) at fs/super.c:319
#16 0x08119c0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#17 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#18 0x0811b195 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#19 SyS_umount (name=134633856, flags=2) at fs/namespace.c:1305
#20 0x080618e2 in handle_syscall (r=0x498be5d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198 #22 userspace (regs=0x498be5d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()

Quoted 15 lines

Quoted 5 lines

Quoted 4 lines

Signature · 3 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

Signature · 7 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Toralf Förster

author

2013-10-02 20:29 UTC #22 lore raw

Quoted 22 lines · J. Bruce Fields

Quoted 18 lines · Toralf Förster
Quoted 8 lines · J. Bruce Fields
Quoted 3 lines · Toralf Förster
Today I run latest git tree with a patched UML (this patch + one for xterm issues) and got 2 times a core dump when I fuzzy test an UML machine with a nearly identical scenario as already described but just shutdowned both UML images instead of shooting one of it in the head.
This is a slightly different test case, so for now it sounds like this could be a preexisting problem, not a regression?
Yes, pre-existing. I'm trying to bisect it, the issue is already in 7d3107d.
But every attempt to get the first bad commit before that commit id failed till now. I started many attempts between v3.10 and 7d3107d. I could not reproduce that issue 100% in that interval. In that interval either all tested commits doesn't show the issue or the NFS client hangs during the stop of NFS service infinitely - and counting that as a "bad" commit doesn't work.
Quoted 2 lines
FWIW a single test case currently takes 2 1/2 hour and even then I do not fully trust the result (except for a bad commit, that's clear.

Quoted 7 lines

Well, we should figure out some other way to narrow down the problem....
I was trying to at least work out where exactly this crash was, but I don't have a commit 768c9d3.
--b.

just for completeness of this thread, that issue could be bisected in the mean while :

http://thread.gmane.org/gmane.linux.kernel/1569818

Quoted 61 lines

Quoted 60 lines
Quoted 50 lines
--b.
Quoted 45 lines
I'll probably need time to figure out a test case, but just as a pre-info here's the back trace:
tfoerste@n22 ~ $ gdb --core=/mnt/ramdisk/core /usr/local/bin/linux-v3.11-7550-g768c9d3 -n -batch -ex bt
warning: core file may not match specified executable file.
[New LWP 7470]
[New LWP 7479]
[New LWP 7477]
[New LWP 7478]
Core was generated by `/usr/local/bin/linux-v3.11-7550-g768c9d3 earlyprintk ubda=/home/tfoerste/virtua'. Program terminated with signal 6, Aborted.
#0  0xb77be424 in __kernel_vsyscall ()
#0  0xb77be424 in __kernel_vsyscall ()
#1  0x083aada5 in kill ()
#2  0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138
#4  0x080613a7 in panic_exit (self=0x85b1518 <panic_exit_notifier>, unused1=0, unused2=0x85e76e0 <buf.15920>) at arch/um/kernel/um_arch.c:240
#5  0x0809a398 in notifier_call_chain (nl=0x0, val=0, v=0x85e76e0 <buf.15920>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a4e3 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182
#7  atomic_notifier_call_chain (nh=0x85e76c4 <panic_notifier_list>, val=0, v=0x85e76e0 <buf.15920>) at kernel/notifier.c:191
#8  0x08408628 in panic (fmt=0x0) at kernel/panic.c:128
#9  0x081131c9 in shrink_dcache_for_umount_subtree (dentry=0x428028f0) at fs/dcache.c:941
#10 0x08113948 in shrink_dcache_for_umount (sb=0x463b8000) at fs/dcache.c:1002
#11 0x08101677 in generic_shutdown_super (sb=0x463b8000) at fs/super.c:404
#12 0x08102395 in kill_anon_super (sb=0x0) at fs/super.c:875
#13 0x081d3ff8 in nfs_kill_super (s=0x0) at fs/nfs/super.c:2598
#14 0x0810153a in deactivate_locked_super (s=0x463b8000) at fs/super.c:294
#15 0x081015d1 in deactivate_super (s=0x463b8000) at fs/super.c:319
#16 0x08119c0c in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#17 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#18 0x0811b195 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#19 SyS_umount (name=134633856, flags=2) at fs/namespace.c:1305
#20 0x080618e2 in handle_syscall (r=0x498be5d4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198 #22 userspace (regs=0x498be5d4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x00000000 in ?? ()
Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
Signature · 7 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 7 lines

MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html