synchronous mounts
Subsystems (3)
Linux is not syncing write() data for files on synchronously mounted filesystems, and it isn't syncing write() data for ext2/3 files which are operating under `chattr +S'.
Is this deliberate, or a bug?
This patch makes write()s data-synchronous. There's a fix-for-the-fix here for ext3. Other filesystems which are second-guessing the generic layer's behaviour may also end up doing a double sync with this patch.
Really, generic_osync_inode() needs to be front-ended by an inode_operations method.
mm/filemap.c +4 -2
@@ -2916,8 +2916,10 @@ unlock:/* For now, when the user asks for O_SYNC, we'll actually * provide O_DSYNC. */ -if ((status >=0 ) && (file->f_flags &O_SYNC )) - status =generic_osync_inode (inode,OSYNC_METADATA |OSYNC_DATA ); +if (status >=0 ) { +if ((file->f_flags &O_SYNC ) ||IS_SYNC (inode)) + status =generic_osync_inode (inode,OSYNC_METADATA |OSYNC_DATA ); + } err = written ? written : status; out:
fs/ext3/file.c +11 -14
@@ -61,22 +61,19 @@ static int ext3_open_file (struct inodestatic ssize_t ext3_file_write (struct file *file,const char *buf,size_t count,loff_t *ppos) { -int ret;struct inode *inode = file->f_dentry->d_inode; - ret =generic_file_write (file, buf, count, ppos); -if ((ret >=0 ) &&IS_SYNC (inode)) { -if (file->f_flags &O_SYNC ) { -/* -* generic_osync_inode() has already done the sync -*/ - }else { -int ret2 =ext3_force_commit (inode->i_sb); -if (ret2) - ret = ret2; - } - } -return ret; +/* +* Nasty: if the file is subject to synchronous writes then we need +* to force generic_osync_inode() to call ext3_write_inode(). +* We do that by marking the inode dirty. This adds much more +* computational expense than we need, but we're going to sync +* anyway. +*/ +if (IS_SYNC (inode) || (file->f_flags &O_SYNC )) +mark_inode_dirty (inode); + +return generic_file_write (file, buf, count, ppos); }struct file_operations ext3_file_operations = {
Hi,
Quoted 3 lines · Andrew Morton
Linux is not syncing
write()data for files on synchronously mounted filesystems, and it isn't syncingwrite()data for ext2/3 files which are operating under `chattr +S'.
In the past, chattr +S and mount -o sync always resulted in sync metadata with no guarantees about data.
I'm not sure this makes much sense, but it's what has always happened. For directories, the behaviour is fine, in particular as it gives us the same directory sync consistency semantics as synchronous BSD UFS.
It's not clear to me that chattr/mount sync options make _any_ sense for regular file metadata. Rather than tightening up the semantics, I'd actually prefer to restrict them so that they only apply to directories. Users who set the sync bits are usually doing so for applications like MTAs where it's directory syncing which is what matters: the apps typically fsync the files themselves, anyway.
Quoted 2 lines
Really,
generic_osync_inode()needs to be front-ended by aninode_operationsmethod.
Yes.
Cheers,
Stephen
Quoted 15 lines · Stephen C. Tweedie
Hi,
Quoted 3 lines · Andrew Morton
Linux is not syncing
write()data for files on synchronously mounted filesystems, and it isn't syncingwrite()data for ext2/3 files which are operating under `chattr +S'.In the past, chattr +S and mount -o sync always resulted in sync metadata with no guarantees about data.
I'm not sure this makes much sense, but it's what has always happened. For directories, the behaviour is fine, in particular as it gives us the same directory sync consistency semantics as synchronous BSD UFS.
Quoted 7 lines
It's not clear to me that chattr/mount sync options make _any_ sense for regular file metadata. Rather than tightening up the semantics, I'd actually prefer to restrict them so that they only apply to directories. Users who set the sync bits are usually doing so for applications like MTAs where it's directory syncing which is what matters: the apps typically fsync the files themselves, anyway.
OK, that makes sense. Thanks. The mount' and chattr' manpages need updating...
So shall we try to nail this down? Synchronous mounts and chattr +S provide synchronous semantics for directory contents, diretory metadata and directory inodes only. And fsync() will write out a file's data, metadata and inode?
If this is correct then there are a few places where ext2 is syncing stuff unnecessarily - file indirect blocks, etc. Not very important at this stage I guess.
-
Hi,
Quoted 5 lines · Andrew Morton
So shall we try to nail this down? Synchronous mounts and chattr +S provide synchronous semantics for directory contents, diretory metadata and directory inodes only. And
fsync()will write out a file's data, metadata and inode?
Sounds sane. UFS sync mounts also update the inode bitmaps synchronously, and in order, so that they can tell exactly when an inode is valid. I guess that we don't need that, at least on ext2, since the i_dtime gives us the necessary information automatically.
Quoted 3 lines
If this is correct then there are a few places where ext2 is syncing stuff unnecessarily - file indirect blocks, etc. Not very important at this stage I guess.
It's important for people running MTAs which expect dir sync
behaviour: admins have to pay the performance cost of all file updates being sync as well as the directory updates if they enable chattr +S on the dir.
--Stephen
Quoted 6 lines · Stephen C. Tweedie
It's not clear to me that chattr/mount sync options make _any_ sense for regular file metadata. Rather than tightening up the semantics, I'd actually prefer to restrict them so that they only apply to directories. Users who set the sync bits are usually doing so for applications like MTAs where it's directory syncing which is what matters: the apps typically fsync the files themselves, anyway.
I don't think it is correct to make that assumption.
When working on something likely to crash, I always remount my filesystems 'sync' with the intention to have the kernel immediately sync to disk anything and everything it is coded to do. Since the kernel is responsible to flushing data to disk, it makes perfect sense to have an option to sync not only metadata but data to disk immediately, if the user desires such.
Further, expecting all apps to fsync(2) files under the right circumstances is not reasonable. There are "normal" circumstances where someone expects non-syncing behavior of "cat foo bar > foobar", and then there are extentuating circumstances where another expects the shell to sync that command immediately. Should we rewrite cat/bash/apps to all fsync, depending on an option? Should we expect people to modify all their shell scripts to include "/bin/sync" for those times when they want data-sync? Such is not scalable at all.
It would make more sense to me to implement 'sync' as sync-everything, and add 'dirsync' or 'metasync' for syncing only directories or only metadata.
As it stands, it seems like redefining 'sync' to sync less data than is currently done is not only changing current behavior, but providing less to users overall.
Jeff
Signature · 3 lines
Jeff Garzik | Only so many songs can be sung Building 1024 | with two lips, two lungs, and one tongue. MandrakeSoft | - nomeansno
Quoted 5 lines · Jeff Garzik
As it stands, it seems like redefining 'sync' to sync less data than is currently done is not only changing current behavior, but providing less to users overall.
Persuasively argued. You appear to have your wish, as this patch was merged in -pre5.
A `dirsync' option does make sense though, for the reasons which Stephen outlined.
The whole handling of synchronous operations needs a rip-up-and-rewrite anyway. We're currently holding onto a stack of locks while waiting for the disk to spin round and round. It's a great scalability bottleneck
for multiple threads doing things in the same directory. This is
something I shall look at when the kernel versions turn odd.
-
Quoted 2 lines · Andrew Morton
A `dirsync' option does make sense though, for the reasons which Stephen outlined.
So we could then have:
- (no option) == async, which only syncs file + data on fsync() or O_SYNC (BSD calls this async, it may corrupt file systems because writes are out-of-order)
- dirsync, which syncs directories and metadata and causes ordered writes thereof (BSD calls this noasync), no chance of corrupting on-disk structure unrecoverably.
- sync, which syncs all filesystem operations (BSD calls this sync also), will have at most 1 dirty block at a time on non-journaled file systems(?)
I expect sync to be faster on journalled file systems in that case, because "in-order execution" to journal will probably cause linear writes, while on ext2, it will involve seeking.
Signature · 4 lines
Matthias Andree "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." Benjamin Franklin
Hi,
Quoted 3 lines · Jeff Garzik
When working on something likely to crash, I always remount my filesystems 'sync' with the intention to have the kernel immediately sync to disk anything and everything it is coded to do.
The kernel has, in my memory, never behaved like that on sync mounts. mount -o sync was always intended just to give people the BSD-style sync metadata updates that some users expected.
The "mount" man page is wrong on this one.
Quoted 4 lines
Since the
kernel is responsible to flushing data to disk, it makes perfect sense to have an option to sync not only metadata but data to disk immediately, if the user desires such.
If you want to sync _everything_, it's at least 5 seeks per write syscall when you're writing a new file: superblock, group descriptor, block bitmap, inode, data, and potentially inode indirect.
There's no point doing all that, especially since some of that data is redundant and will be rebuilt by e2fsck anyway after a crash.
Is it really such an important feature that we're willing to suffer a factor-of-100 or more slowdown for it?
Quoted 8 lines
Further, expecting all apps to fsync(2) files under the right circumstances is not reasonable. There are "normal" circumstances where someone expects non-syncing behavior of "cat foo bar > foobar", and then there are extentuating circumstances where another expects the shell to sync that command immediately. Should we rewrite cat/bash/apps to all fsync, depending on an option? Should we expect people to modify all their shell scripts to include "
/bin/sync" for those times when they want data-sync? Such is not scalable at all.
Not-scalable is doing 5000 seeks to write a 4MB file.
The behaviour you are talking about now, "cat foo bar > foobar" and expecting it to be intact on return, is *not the same thing*. The sync mount option is there to order metadata writes for predictable recovery of the directory structure. In the "cat" case, nobody cares what the inode is like during the write. All that is desired in that example is fsync-on-close, and it is insane to implement fsync-on-close by writing every single block of the file synchronously.
At ALS, an ext3 user asked why ext3 performance was entirely unusable under mount -o sync (he had a broken config which accidentally set an ext3 mount synchronous), whereas ext2 was OK. I only realised afterwards that this was because of ext3's ordered data writes:
whereas ext2 was just syncing the inodes and indirect blocks on write, ext3 was syncing the data too as part of the ordered data guarantees, and performance was totally destroyed by the extra seeks.
"sync to keep the fs structures intact" and "sync to keep this file intact" are two totally different things. In the latter case, we only care about the file contents as a whole, so fsync-on-close is far more appropriate. If we want that, lets add it as a new option, but I don't see the benefit in making o- sync do all file data writes 100% synchronously.
Cheers,
Stephen
Quoted 10 lines · Stephen C. Tweedie
Quoted 3 lines · Jeff Garzik
When working on something likely to crash, I always remount my filesystems 'sync' with the intention to have the kernel immediately sync to disk anything and everything it is coded to do.
The kernel has, in my memory, never behaved like that on sync mounts. mount -o sync was always intended just to give people the BSD-style sync metadata updates that some users expected.
The "mount" man page is wrong on this one.
No, that's always been a bug. Occasionally it gets brought up on lkml and people have talked about fixing it.
Quoted 9 lines
Quoted 4 lines
Since the
kernel is responsible to flushing data to disk, it makes perfect sense to have an option to sync not only metadata but data to disk immediately, if the user desires such.If you want to sync _everything_, it's at least 5 seeks per write syscall when you're writing a new file: superblock, group descriptor, block bitmap, inode, data, and potentially inode indirect.
Quoted 5 lines
There's no point doing all that, especially since some of that data is redundant and will be rebuilt by e2fsck anyway after a crash.
Is it really such an important feature that we're willing to suffer a factor-of-100 or more slowdown for it?
mount -o dirsync, if you don't want the slowdown.
I can write a fast, incorrect implementation of 'sync' too. Let's try for a correct implementation.
Quoted 11 lines
Quoted 8 lines
Further, expecting all apps to fsync(2) files under the right circumstances is not reasonable. There are "normal" circumstances where someone expects non-syncing behavior of "cat foo bar > foobar", and then there are extentuating circumstances where another expects the shell to sync that command immediately. Should we rewrite cat/bash/apps to all fsync, depending on an option? Should we expect people to modify all their shell scripts to include "
/bin/sync" for those times when they want data-sync? Such is not scalable at all.Not-scalable is doing 5000 seeks to write a 4MB file.
Quoted 8 lines
The behaviour you are talking about now, "cat foo bar > foobar" and expecting it to be intact on return, is *not the same thing*. The sync mount option is there to order metadata writes for predictable recovery of the directory structure. In the "cat" case, nobody cares what the inode is like during the write. All that is desired in that example is fsync-on-close, and it is insane to implement fsync-on-close by writing every single block of the file synchronously.
The "sync" mount option's purpose is and should be this simple:
"All I/O to the file system should be done synchronously."
If you want different behavior, use a different option.
I still do not understand why you seem to think modifying tons of programs and shell scripts is reasonable, in order to get "true" sync behavior.
Quoted 7 lines
At ALS, an ext3 user asked why ext3 performance was entirely unusable under mount -o sync (he had a broken config which accidentally set an ext3 mount synchronous), whereas ext2 was OK. I only realised afterwards that this was because of ext3's ordered data writes:
whereas ext2 was just syncing the inodes and indirect blocks on write, ext3 was syncing the data too as part of the ordered data guarantees, and performance was totally destroyed by the extra seeks.
Sure. Seems perfectly normal and expected for an fs mounted 'sync'.
Quoted 6 lines
"sync to keep the fs structures intact" and "sync to keep this file intact" are two totally different things. In the latter case, we only care about the file contents as a whole, so fsync-on-close is far more appropriate. If we want that, lets add it as a new option, but I don't see the benefit in making o- sync do all file data writes 100% synchronously.
The benefit is a correct implementation..
Jeff
Signature · 3 lines
Jeff Garzik | Only so many songs can be sung Building 1024 | with two lips, two lungs, and one tongue. MandrakeSoft | - nomeansno
May I suggest another way of thinking about this issue. ext3 allows one to journal everything, or just metadata. Why -avoid- this distrinction when it comes to the 'sync' mount option (and 'dirsync' or 'metasync')?
Signature · 3 lines
Jeff Garzik | Only so many songs can be sung Building 1024 | with two lips, two lungs, and one tongue. MandrakeSoft | - nomeansno
Quoted 6 lines · Stephen Tweedie
If you want to sync _everything_, it's at least 5 seeks per write syscall when you're writing a new file: superblock, group descriptor, block bitmap, inode, data, and potentially inode indirect.
There's no point doing all that, especially since some of that data is redundant and will be rebuilt by e2fsck anyway after a crash.
The file system cannot judge on what's needed, neither can the file system developer.
The developer however can offer the administrator
and/or application developer the corresponding interfaces. In the sense of portability, however, it may be rather useful to correspond to BSD semantics, so I vote to change sync and offer dirsync -- it might be useful to use BSD nomenclature though and offer a synonymous noasync option (which also has dirsync semantics as laid out in this thread before).
It does not matter whether you look at BSD or Linux man pages, either one documents "sync - All I/O to the file system should be done synchronously." It should behave like documented. Let's not introduce compatibility problems by fixing the documentation.
Quoted 2 lines
Is it really such an important feature that we're willing to suffer a factor-of-100 or more slowdown for it?
It depends on the task. If the machine regularly reads data from an external interface and records it to disk, like in a log, synchronous data writes may be important.
BTW, Wietse Venema was dazzled when he got to know that Linux values its syslog (defaults to all-synchronous writes) more than its mail (defaults to all-asynchronous writes that not even a file fsync() rectifies).
Quoted 1 line
Not-scalable is doing 5000 seeks to write a 4MB file.
If that's what the admin wants, let him do so.
Quoted 4 lines
what the inode is like during the write. All that is desired in that example is fsync-on-close, and it is insane to implement fsync-on-close by writing every single block of the file synchronously.
Yes, but you cannot generalize this example.
Quoted 3 lines
appropriate. If we want that, lets add it as a new option, but I don't see the benefit in making o- sync do all file data writes 100% synchronously.
Compatibility and reliability are good points.
Signature · 4 lines
Matthias Andree "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." Benjamin Franklin
Quoted 4 lines · Neil Brown
Andrew's patch fixes
O_SYNCin data=journal and data=ordered modes, but does not fix fsync in data=writeback mode.
Neil, thanks for spotting this.
patch-2.4.10-pre11 introduced a new per-inode dirty buffer queue, inode.i_dirty_data_buffers.
generic_commit_write() was changed so that it placed buffers on that queue, rather than i_dirty_buffers. A new function fsync_inode_data_buffers() was added for writing the new buffer list out.
That patch changed ext2 and reiserfs to use the new function. The patch was not copied to any mailing list. There was no announcement and no discussion. There are no comments in the code indicating the distinction between these lists and why we have two such.
This change broke fsync() for three in-kernel filesystems as well as the then-out-of-kernel ext3 and presumably any other buffer-based Linux filesystems.
Ah. google finds this, from April:
http://www.uwsg.iu.edu/hypermail/linux/kernel/0104.1/0810.html
which attempts to explain why the separate list was added. It appears to be bogus. There is no reason why, if correctly done, syncing a list of ext2 data buffers and indirects should be significantly slower than syncing just the data.
Here's a fix. A better fix would be to back out the unnecesary list and its support functions.
fs/minix/file.c +3 -1
@@ -30,8 +30,10 @@ struct inode_operations minix_file_inodeint minix_sync_file (struct file * file,struct dentry *dentry,int datasync) {struct inode *inode = dentry->d_inode; -int err =fsync_inode_buffers (inode); +int err; + err =fsync_inode_buffers (inode); + err |=fsync_inode_data_buffers (inode);if (!(inode->i_state &I_DIRTY ))return err;if (datasync && !(inode->i_state &I_DIRTY_DATASYNC ))
fs/sysv/file.c +3 -1
@@ -35,8 +35,10 @@ struct inode_operations sysv_file_inode_int sysv_sync_file (struct file * file,struct dentry *dentry,int datasync) {struct inode *inode = dentry->d_inode; -int err =fsync_inode_buffers (inode); +int err; + err =fsync_inode_buffers (inode); + err |=fsync_inode_data_buffers (inode);if (!(inode->i_state &I_DIRTY ))return err;if (datasync && !(inode->i_state &I_DIRTY_DATASYNC ))
fs/udf/fsync.c +1 -0
@@ -45,6 +45,7 @@ int udf_fsync_inode(struct inode *inode,int err; err =fsync_inode_buffers (inode); + err |=fsync_inode_data_buffers (inode);if (!(inode->i_state &I_DIRTY ))return err;if (datasync && !(inode->i_state &I_DIRTY_DATASYNC ))
fs/ext3/fsync.c +1 -0
@@ -62,6 +62,7 @@ int ext3_sync_file(struct file * file, s* we'll end up waiting on them in commit. */ ret =fsync_inode_buffers (inode); + ret |=fsync_inode_data_buffers (inode);ext3_force_commit (inode->i_sb);