fs: include device name in error messages about freezing
PATCH 1/2 fs: include device name in error messages about freezing
While here use pr_err instead of printk(KERN_ERR...)
--- fs/super.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
fs/super.c +2 -4
@@ -1323,8 +1323,7 @@ int freeze_super(struct super_block *sb)if (sb->s_op->freeze_fs) { ret = sb->s_op->freeze_fs (sb);if (ret) { -printk (KERN_ERR -"VFS:Filesystem freeze failed\n" ); +pr_err ("VFS:Filesystem %s freeze failed\n" , sb->s_id); sb->s_writers.frozen =SB_UNFROZEN ;smp_wmb ();wake_up (&sb->s_writers.wait_unfrozen);
@@ -1364,8 +1363,7 @@ int thaw_super(struct super_block *sb)if (sb->s_op->unfreeze_fs) { error = sb->s_op->unfreeze_fs (sb);if (error) { -printk (KERN_ERR -"VFS:Filesystem thaw failed\n" ); +pr_err ("VFS:Filesystem %s thaw failed\n" , sb->s_id);up_write (&sb->s_umount);return error; }
Signature · 1 line
1.8.3.1
PATCH 2/2 fs: print a message when freezing/unfreezing filesystems
This helps hang troubleshooting efforts when only dmesg is available.
--- fs/super.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
fs/super.c +5 -6
@@ -1289,12 +1289,9 @@ int freeze_super(struct super_block *sb)return 0 ;/* sic - it's "nothing to do" */ } -if (sb->s_flags &MS_RDONLY ) { +if (sb->s_flags &MS_RDONLY )/* Nothing to do really... */ - sb->s_writers.frozen =SB_FREEZE_COMPLETE ; -up_write (&sb->s_umount); -return 0 ; - } + goto out;/* From now on, no new normal writers can start */ sb->s_writers.frozen =SB_FREEZE_WRITE ;
@@ -1335,8 +1332,10 @@ int freeze_super(struct super_block *sb)* This is just for debugging purposes so that fs can warn if it * sees write activity when frozen is set to SB_FREEZE_COMPLETE. */ +out: sb->s_writers.frozen =SB_FREEZE_COMPLETE ;up_write (&sb->s_umount); +pr_info ("VFS:Filesystem %s thawed\n" , sb->s_id);return 0 ; }EXPORT_SYMBOL (freeze_super);
@@ -1374,7 +1373,7 @@ out:smp_wmb ();wake_up (&sb->s_writers.wait_unfrozen);deactivate_locked_super (sb); - +pr_info ("VFS:Filesystem %s thawed\n" , sb->s_id);return 0 ; }EXPORT_SYMBOL (thaw_super);
Signature · 1 line
1.8.3.1
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 1 line · Mateusz Guzik
This helps hang troubleshooting efforts when only dmesg is available.
[]
fs/super.c
Quoted 1 line
fs/super.c +0
-0
[]
fs/super.c int freeze_super(struct super_block *sb) · line 1294
Quoted 12 lines
fs/super.c +2
-5@@ -1289,12 +1289,9 @@ int freeze_super(struct super_block *sb)return 0 ;/* sic - it's "nothing to do" */ } -if (sb->s_flags &MS_RDONLY ) { +if (sb->s_flags &MS_RDONLY )/* Nothing to do really... */ - sb->s_writers.frozen =SB_FREEZE_COMPLETE ; -up_write (&sb->s_umount); -return 0 ; - } + goto out;
Should this block really be part of this patch? If so, please add a small note to the commit log.
I'm not sure making the reader goto out for this bit of the code needs doing as there are multiple blocks with return above here.
Trivially, when there is a comment in a single line like this, then braces are often used too.
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 9 lines · Joe Perches
Quoted 1 line · Mateusz Guzik
This helps hang troubleshooting efforts when only dmesg is available.
[]
fs/super.c
Quoted 1 line
fs/super.c +0
-0[]
Quoted 4 lines
fs/super.c +0
-0@@ -1289,12 +1289,9 @@ int freeze_super(struct super_block *sb)return 0 ;/* sic - it's "nothing to do" */ }
Quoted 12 lines
fs/super.c int freeze_super(struct super_block *sb) · line 1294
Quoted 8 lines
fs/super.c +2
-5-if (sb->s_flags &MS_RDONLY ) { +if (sb->s_flags &MS_RDONLY )/* Nothing to do really... */ - sb->s_writers.frozen =SB_FREEZE_COMPLETE ; -up_write (&sb->s_umount); -return 0 ; - } + goto out;Should this block really be part of this patch? If so, please add a small note to the commit log.
This is the same code which you can find at the end of the function. I added the label so that I can write freeze printk only once.
Quoted 4 lines
I'm not sure making the reader goto out for this bit of the code needs doing as there are multiple blocks with return above here.
There are 2. One just returns an error and second one does not change s_writers.frozen, so unreezing of something like this would fail.
However, the case which got modified results in unfreezable fs, so not printing the message for this case would result in unmatched entries in the log.
Quoted 3 lines
Trivially, when there is a comment in a single line like this, then braces are often used too.
Thanks, I'll send a modified patch later. Waiting for some other comments.
Signature · 1 line
Mateusz Guzik
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 2 lines · Mateusz Guzik
This is the same code which you can find at the end of the function. I added the label so that I can write freeze printk only once.
Yes, but for a read-only filesystem is the message useful?
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 6 lines · Joe Perches
Quoted 2 lines · Mateusz Guzik
This is the same code which you can find at the end of the function. I added the label so that I can write freeze printk only once.
Yes, but for a read-only filesystem is the message useful?
I see no harm in printing it and it seems more correct given that s_writers.frozen changed, but I'm fine either way.
btw, I just noticed a copy-pasto - the same message for freezing and unfreezing found its way into the patch. sigh.
Signature · 1 line
Mateusz Guzik
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
In reply to Mateusz GuzikQuoted 1 line · Mateusz Guzik
This helps hang troubleshooting efforts when only dmesg is available.
I really don't think that spamming dmesg every time a filesystem is frozen or thawed is a good idea. This happens a *lot* when systems are using snapshots, and for the most part nobody cares about freeze/thaw cycles because they almost always work just fine.
I'd think that /proc/self/mounts would be a much better place to indicate that the fs is frozen. After all, that's where we tell people whether the filesystem is ro or rw, and frozen is just a temporary, non-invasive ro state...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 7 lines · Dave Chinner
Quoted 1 line · Mateusz Guzik
This helps hang troubleshooting efforts when only dmesg is available.
I really don't think that spamming dmesg every time a filesystem is frozen or thawed is a good idea. This happens a *lot* when systems are using snapshots, and for the most part nobody cares about freeze/thaw cycles because they almost always work just fine.
The problems people are worried about are when there is freezing and unfreezing taking place in combination with a machine-level suspend/resume cycle, no? What if we only spammed dmesg when doing a suspend/resume?
- Ted
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 9 lines · Theodore Ts'o
Quoted 7 lines · Dave Chinner
Quoted 1 line · Mateusz Guzik
This helps hang troubleshooting efforts when only dmesg is available.
I really don't think that spamming dmesg every time a filesystem is frozen or thawed is a good idea. This happens a *lot* when systems are using snapshots, and for the most part nobody cares about freeze/thaw cycles because they almost always work just fine.
Quoted 4 lines
The problems people are worried about are when there is freezing and unfreezing taking place in combination with a machine-level suspend/resume cycle, no? What if we only spammed dmesg when doing a suspend/resume?
Filesystems are not frozen by power management based suspend/resume cycles. That uses sync and then assumes that it can't race with with the filesystem doing metadata writeback while the suspend code is creating the freeze image. Hence there won't by any syslog spam because freeze is not used. ;)
Sure, I've been advocating that suspend should use freezing rather than sync for years, but that's never happened...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
In reply to Dave ChinnerQuoted 8 lines · Dave Chinner
Quoted 1 line · Mateusz Guzik
This helps hang troubleshooting efforts when only dmesg is available.
I really don't think that spamming dmesg every time a filesystem is frozen or thawed is a good idea. This happens a *lot* when systems are using snapshots, and for the most part nobody cares about freeze/thaw cycles because they almost always work just fine.
I agree it may get noisy.
Quoted 5 lines
I'd think that
/proc/self/mountswould be a much better place to indicate that the fs is frozen. After all, that's where we tell people whether the filesystem is ro or rw, and frozen is just a temporary, non-invasive ro state...
Except you can't inspect /proc/self/mounts when the only thing you got is dmesg, so this does not really help my case.
That said, I'll try to come up with a different solution.
Poorly reported side-effects of frozen I/O are only a part of the real problem which is hung task detector being able to typically report backtraces of "victims" only. I came up with printks becuase these are a cheap way and would help us out in a lot of cases.
So general idea is to support providing callbacks when setting tasks to TASK_UNINTERRUPTIBLE which could either tell the user what's up directly or would perform some heuristics. I'll post this in a separate thread later, maybe with PoC.
Thanks,
Signature · 1 line
Mateusz Guzik
Jan Kara
reviewerRe: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 21 lines · Mateusz Guzik
Quoted 8 lines · Dave Chinner
Quoted 1 line · Mateusz Guzik
This helps hang troubleshooting efforts when only dmesg is available.
I really don't think that spamming dmesg every time a filesystem is frozen or thawed is a good idea. This happens a *lot* when systems are using snapshots, and for the most part nobody cares about freeze/thaw cycles because they almost always work just fine.
I agree it may get noisy.
Quoted 5 lines
I'd think that
/proc/self/mountswould be a much better place to indicate that the fs is frozen. After all, that's where we tell people whether the filesystem is ro or rw, and frozen is just a temporary, non-invasive ro state...Except you can't inspect
/proc/self/mountswhen the only thing you got is dmesg, so this does not really help my case.
Quoted 6 lines
That said, I'll try to come up with a different solution.
Poorly reported side-effects of frozen I/O are only a part of the real problem which is hung task detector being able to typically report backtraces of "victims" only. I came up with printks becuase these are a cheap way and would help us out in a lot of cases.
I was tracking down a couple of times what the hell is freezing the
filesystem (and not unfreezing it) and I agree with Mateusz it would be nice if we could tell after the fact who froze the fs. Maybe we could store that information in superblock and dump it during emergency thaw?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 34 lines · Jan Kara
Date: Thu, 15 May 2014 12:01:57 +0200
From: Jan Kara <[email protected]>
To: Mateusz Guzik <[email protected]>
Cc: Dave Chinner <[email protected]>, [email protected],
[email protected], Josef Bacik <[email protected]>, Jan Kara <[email protected]>, Al Viro <[email protected]>, Eric Sandeen <[email protected]>
Subject: Re: [PATCH 2/2] fs: print a message when freezing/unfreezing
filesystemsQuoted 23 lines · Mateusz Guzik
Quoted 8 lines · Dave Chinner
Quoted 1 line · Mateusz Guzik
This helps hang troubleshooting efforts when only dmesg is available.
I really don't think that spamming dmesg every time a filesystem is frozen or thawed is a good idea. This happens a *lot* when systems are using snapshots, and for the most part nobody cares about freeze/thaw cycles because they almost always work just fine.
I agree it may get noisy.
Quoted 5 lines
I'd think that
/proc/self/mountswould be a much better place to indicate that the fs is frozen. After all, that's where we tell people whether the filesystem is ro or rw, and frozen is just a temporary, non-invasive ro state...Except you can't inspect
/proc/self/mountswhen the only thing you got is dmesg, so this does not really help my case.That said, I'll try to come up with a different solution.
Quoted 8 lines
Quoted 4 lines
Poorly reported side-effects of frozen I/O are only a part of the real problem which is hung task detector being able to typically report backtraces of "victims" only. I came up with printks becuase these are a cheap way and would help us out in a lot of cases.
I was tracking down a couple of times what the hell is freezing the
filesystem (and not unfreezing it) and I agree with Mateusz it would be nice if we could tell after the fact who froze the fs. Maybe we could store that information in superblock and dump it during emergency thaw?
I like that idea and if we agree to create procfs file to list frozen file system this might be one of the useful information to report there.
-Lukas
Quoted 3 lines
Honza
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
In reply to Jan KaraQuoted 4 lines · Jan Kara
I was tracking down a couple of times what the hell is freezing the
filesystem (and not unfreezing it) and I agree with Mateusz it would be nice if we could tell after the fact who froze the fs. Maybe we could store that information in superblock and dump it during emergency thaw?
Saving it in the superblock would require changing a bunch of file systems. What if we store this information in memory, and print it out under certain conditions (i.e., after a soft lockup detection, or upon request of some magic sysrq request)?
Or we could create a tunable threshold and print a message after a file system has been frozen more than a particular specified duration, with that duration set conservatively to something like 60 or 120 seconds by default.
- Ted
Jan Kara
reviewerRe: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 10 lines · Ted Tso
Quoted 4 lines · Jan Kara
I was tracking down a couple of times what the hell is freezing the
filesystem (and not unfreezing it) and I agree with Mateusz it would be nice if we could tell after the fact who froze the fs. Maybe we could store that information in superblock and dump it during emergency thaw?Saving it in the superblock would require changing a bunch of file systems. What if we store this information in memory, and print it out under certain conditions (i.e., after a soft lockup detection, or upon request of some magic sysrq request)?
By 'superblock' I meant 'struct super_block' ;) So we are in agreement I
believe.
Quoted 4 lines
Or we could create a tunable threshold and print a message after a file system has been frozen more than a particular specified duration, with that duration set conservatively to something like 60 or 120 seconds by default.
I was thinking about this as well but all these "warn after X seconds"
warnings tend to have quite a few false positives in practice so dumping this in emergency-thaw sysrq handler or exposing the information somewhere in proc (e.g. mountinfo) would look like a better option to me.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 2/2] fs: print a message when freezing/unfreezing filesystems
Quoted 6 lines · Jan Kara
Quoted 4 lines
Saving it in the superblock would require changing a bunch of file systems. What if we store this information in memory, and print it out under certain conditions (i.e., after a soft lockup detection, or upon request of some magic sysrq request)?
By 'superblock' I meant 'struct
super_block' ;) So we are in agreement I
believe.
Ah, yes, we're in agreement. I thought you were talking about the on-disk superblock.
Quoted 8 lines
Quoted 4 lines
Or we could create a tunable threshold and print a message after a file system has been frozen more than a particular specified duration, with that duration set conservatively to something like 60 or 120 seconds by default.
I was thinking about this as well but all these "warn after X seconds"
warnings tend to have quite a few false positives in practice so dumping this in emergency-thaw sysrq handler or exposing the information somewhere in proc (e.g. mountinfo) would look like a better option to me.
Well, we already have the soft lockup warning, which sometimes has some false positives, but in practice, if a process is runable but doesn't get to run in 2 minutes (the default is 20 seconds, but we've used 2 minutes to avoid the false positive problem on a super busy system), something is probably clearly wrong.
Similarly, if a process is trying to write to a frozen file system, and can't after two minutes, something is almost certainly wrong, or least, it's something a system administrator should know about it. We can argue over whether the default threshold should be 20 seconds, or 120 seconds, or 2 hours, but I think there would be agreement that for pretty much any configuration, there is some delay after which printing a message is actually the right thing to do. (Yes, "time that a process is waiting to write to a frozen file system != time the file system is frozen" --- the latter is easier to implement, but if people feel strongly about it, the former isn't that much more difficult.)
The problem with using an sysrq handler is the user has to know how to use it. If the user files a bug saying the system has mysteriously hung, the fact that the system log contains a hint as to what might be going on would be very useful for an enterprise distribiution's help desk. (Yes, this won't help if it's the root file system is the one that's been frozen, unless the customer has configured remote syslog. But for many cases, it might provide a vital clue that could save a lot of time and support costs.)
- Ted