add procfs tunable to enable immediate panic when there are busy inodes after umount
Subsystems (2)
PATCH 1/1 add procfs tunable to enable immediate panic when there are busy inodes after umount
After spending quite a bit of time tracking down a "VFS: busy inodes after unmount" problem, it occurs to me that it would be nice to be able to force a panic when that occurs. While an oops message alone is not generally helpful for tracking down this sort of problem, collecting and analyzing a coredump when this occurs can be.
The following patch adds a procfs tunable that allows you to force a core when a "busy inodes after umount" problem occurs. It also changes the classic error message to be something a bit less cryptic to users.
fs/block_dev.c +0 -0
fs/inode.c +1 -1
@@ -327,7 +327,7 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose) count++;continue ; } - busy =1 ; + ++busy; }/* only unused inodes may be cached with i_count zero */ inodes_stat.nr_unused -= count;
fs/super.c +11 -4
@@ -287,6 +287,8 @@ int fsync_super(struct super_block *sb)void generic_shutdown_super (struct super_block *sb) {const struct super_operations *sop = sb->s_op; +extern int umount_debug; +int busy;if (sb->s_root) {shrink_dcache_for_umount (sb);
@@ -303,10 +305,15 @@ void generic_shutdown_super(struct super_block *sb) sop->put_super (sb);/* Forget any remaining inodes */ -if (invalidate_inodes (sb)) { -printk ("VFS: Busy inodes after unmount of %s. " -"Self-destruct in 5 seconds. Have a nice day...\n" , - sb->s_id); +if (busy =invalidate_inodes (sb)) { +printk ("VFS: %d busy inodes after unmount of %s. " , + busy, sb->s_id); +if (umount_debug !=0 ) { +printk ("Crashing host on request.\n" ); +BUG (); + }else { +printk ("This machine will likely crash eventually. Consider a reboot.\n" ); + } }unlock_kernel ();
include/linux/fs.h +0 -0
include/linux/sysctl.h +1 -0
@@ -818,6 +818,7 @@ enumFS_AIO_NR =18 ,/* current system-wide number of aio requests */ FS_AIO_MAX_NR =19 ,/* system-wide maximum number of aio requests */ FS_INOTIFY =20 ,/* inotify submenu */ +FS_UMOUNT_DEBUG =21 ,/* busy inodes on umount debug switch */ FS_OCFS2=988, /* ocfs2 */ };
kernel/sysctl.c +9 -0
@@ -156,6 +156,7 @@ extern ctl_table pty_table[];extern ctl_table inotify_table[];#endif +int umount_debug;#ifdef HAVE_ARCH_PICK_MMAP_LAYOUT int sysctl_legacy_va_layout;#endif
@@ -962,6 +963,14 @@ static ctl_table fs_table[] = { .mode = 0644, .proc_handler = &proc_dointvec, }, + { + .ctl_name = FS_UMOUNT_DEBUG, + .procname = "umount_debug", + .data = &umount_debug, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, #ifdef CONFIG_DNOTIFY { .ctl_name = FS_DIR_NOTIFY,
Quoted 17 lines · Jeff Layton
After spending quite a bit of time tracking down a "VFS: busy inodes after unmount" problem, it occurs to me that it would be nice to be able to force a panic when that occurs. While an oops message alone is not generally helpful for tracking down this sort of problem, collecting and analyzing a coredump when this occurs can be.
The following patch adds a procfs tunable that allows you to force a core when a "busy inodes after umount" problem occurs. It also changes the classic error message to be something a bit less cryptic to users.
fs/super.c +0
-3@@ -303,10 +305,15 @@ void generic_shutdown_super(struct super_block *sb) sop->put_super (sb);/* Forget any remaining inodes */ -if (invalidate_inodes (sb)) { - printk("VFS: Busy inodes after unmount of %s. " - "Self-destruct in 5 seconds. Have a nice day...\n",
fs/super.c void generic_shutdown_super(struct super_block *sb) · line 316
Quoted 10 lines
fs/super.c +9
-1- sb->s_id); +if (busy =invalidate_inodes (sb)) { +printk ("VFS: %d busy inodes after unmount of %s. " , + busy, sb->s_id); +if (umount_debug !=0 ) { +printk ("Crashing host on request.\n" ); +BUG (); + }else { +printk ("This machine will likely crash eventually. Consider a reboot.\n" ); + }
You can add just BUG_ON here and do
echo 1 >/proc/sys/kernel/panic_on_oops
Quoted 31 lines · On Tue, 29 May 2007 23:38:13 +0400 Alexey Dobriyan <[email protected]> wrote:
Quoted 17 lines · Jeff Layton
After spending quite a bit of time tracking down a "VFS: busy inodes after unmount" problem, it occurs to me that it would be nice to be able to force a panic when that occurs. While an oops message alone is not generally helpful for tracking down this sort of problem, collecting and analyzing a coredump when this occurs can be.
The following patch adds a procfs tunable that allows you to force a core when a "busy inodes after umount" problem occurs. It also changes the classic error message to be something a bit less cryptic to users.
fs/super.c +0
-3@@ -303,10 +305,15 @@ void generic_shutdown_super(struct super_block *sb) sop->put_super (sb);/* Forget any remaining inodes */ -if (invalidate_inodes (sb)) { - printk("VFS: Busy inodes after unmount of %s. " - "Self-destruct in 5 seconds. Have a nice day...\n",fs/super.c void generic_shutdown_super(struct super_block *sb) · line 316
Quoted 10 lines
fs/super.c +9
-1- sb->s_id); +if (busy =invalidate_inodes (sb)) { +printk ("VFS: %d busy inodes after unmount of %s. " , + busy, sb->s_id); +if (umount_debug !=0 ) { +printk ("Crashing host on request.\n" ); +BUG (); + }else { +printk ("This machine will likely crash eventually. Consider a reboot.\n" ); + }You can add just
BUG_ONhere and do
Quoted 2 lines
echo 1 >/proc/sys/kernel/panic_on_oops
I certainly could, but the problem is that there's little point in panicing immediately here if you can't collect a coredump. Oops messages aren't very helpful for tracking this sort of thing down, so I'd think we want the BUG() conditional on something more granular than panic_on_oops.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 5 lines · Jeff Layton
After spending quite a bit of time tracking down a "VFS: busy inodes after unmount" problem, it occurs to me that it would be nice to be able to force a panic when that occurs. While an oops message alone is not generally helpful for tracking down this sort of problem, collecting and analyzing a coredump when this occurs can be.
Agreed - we've found that we've had roughly 50% success in finding the cause of these problems from crash dumps triggered immediately like this vs ~0% from a crash that occurred some time later.
Given that this problem will always result in a crash of the kernel at some random time in the future, why don't we just make this error an unconditional panic on get the crash over and done with?
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 7 lines · On Wed, 30 May 2007 10:28:57 +1000 David Chinner <[email protected]> wrote:
Quoted 5 lines · Jeff Layton
After spending quite a bit of time tracking down a "VFS: busy inodes after unmount" problem, it occurs to me that it would be nice to be able to force a panic when that occurs. While an oops message alone is not generally helpful for tracking down this sort of problem, collecting and analyzing a coredump when this occurs can be.
Quoted 8 lines
Agreed - we've found that we've had roughly 50% success in finding the cause of these problems from crash dumps triggered immediately like this vs ~0% from a crash that occurred some time later.
Given that this problem will always result in a crash of the kernel at some random time in the future, why don't we just make this error an unconditional panic on get the crash over and done with?
Perhaps that's the best course of action. Then again, there can be a long time between the problem and crash (weeks even). For someone who can't collect a coredump, it might be preferable to not immediately crash the box and allow them to try to reboot it at a convenient time. That was my reasoning for adding the procfs tunable.
Either way, if the machine doesn't crash immediately, I'd like to see a different error message here. The current one is confusing to users. They see it and figure "my box didn't crash in 5 mins, so everything must be OK!"
Signature · 1 line
Jeff Layton <[email protected]>