RFC: O_PONIES semantics (well O_REWRITE)
The ext4 automatic-fsync-on-rename discussion has shown that many applications simply Do It Wrong when it comes to rewriting configuration files.
Some of the common failures are:
- program overwrites the old config file
- program writes a new file, but forgets to fsync before rename
- program writes the new file in /tmp, so the rename fails on some systems
- program writes a new file and fsyncs, but forgets to give the new file the same file ownership, permission and/or extended attributes as the old file
Magically taking care of filesystem semantics for every use may not be possible (no O_PONIES for you!), but I believe we can help the applications that just want to completely rewrite a file and atomically replace it.
The semantics for O_REWRITE would be:
1) When opening a file O_REWRITE, the file handle points at a freshly allocated, empty file. The original file is still available to programs that open the file without O_REWRITE.
2) O_REWRITE can only be used in conjunction with O_WRONLY, because the file descriptor is not associated with the original file (which has data), but with an empty inode.
3) The code that implements O_REWRITE (kernel? glibc?) makes sure that:
- the new file is on the same filesystem as the original file
- the new file is not linked (so it is automatically freed after a process or system crash)
- the new file's ownership, permissions and extended attributes match that of the original file
4) The application that opens a file O_REWRITE is required to rewrite the entire file.
5) On close(), the code that implements O_REWRITE makes sure that the file is atomically renamed, so that if a system crash happens, the user will see either the old or the new file contents, but never an empty file.
6) After close(), processes that open the file will get the new content. Processes that previously opened the file will hold on to the old inode and get old contents.
Here are my questions:
- Are these semantics useful for programs that want to replace config (or other) files with new content?
- Are these semantics sane?
- What would be the best place to implement these semantics?
Relying on application developers to get it right seems to not have worked out well, so I'm thinking kernel or glibc. Glibc has the advantage of it not being in the kernel, but implementing it in-kernel might give us the opportunity for performance enhancements, like reducing step (5) to merely enforcing ordering between filesystem operations, instead of requiring an fsync.
Signature · 1 line
All rights reversed.
Quoted 19 lines · Rik van Riel
The semantics for
O_REWRITEwould be:1) When opening a file
O_REWRITE, the file handle points at a freshly allocated, empty file. The original file is still available to programs that open the file withoutO_REWRITE.2)
O_REWRITEcan only be used in conjunction withO_WRONLY, because the file descriptor is not associated with the original file (which has data), but with an empty inode.3) The code that implements
O_REWRITE(kernel? glibc?) makes sure that:
- the new file is on the same filesystem as the original file
- the new file is not linked (so it is automatically freed after a process or system crash)
- the new file's ownership, permissions and extended attributes match that of the original file
Quoted 2 lines
4) The application that opens a file
O_REWRITEis required to rewrite the entire file.
This is all essentially open(O_CREAT|O_TRUNC|O_WRONLY)
Quoted 4 lines
5) On
close(), the code that implementsO_REWRITEmakes sure that the file is atomically renamed, so that if a system crash happens, the user will see either the old or the new file contents, but never an empty file.
This would be possible if the kernel set the i_size=0, but didn't send the filesystem the truncate until the file was closed and being flushed.
Quoted 3 lines
6) After
close(), processes that open the file will get the new content. Processes that previously opened the file will hold on to the old inode and get old contents.
What is the benefit of (6)? Of all these semantics this is the one that would cause the most confusion I think.
Quoted 8 lines
Here are my questions:
- Are these semantics useful for programs that want to replace config (or other) files with new content?
- Are these semantics sane?
- What would be the best place to implement these semantics?
The main question is - would any applications use O_REWRITE in the first place, or would it just make sense to have a helper function in glibc like e.g. mktemp that handles the "atomic update of config file" properly in the first place.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 20 lines · Andreas Dilger
Quoted 19 lines · Rik van Riel
The semantics for
O_REWRITEwould be:1) When opening a file
O_REWRITE, the file handle points at a freshly allocated, empty file. The original file is still available to programs that open the file withoutO_REWRITE.2)
O_REWRITEcan only be used in conjunction withO_WRONLY, because the file descriptor is not associated with the original file (which has data), but with an empty inode.3) The code that implements
O_REWRITE(kernel? glibc?) makes sure that:
- the new file is on the same filesystem as the original file
- the new file is not linked (so it is automatically freed after a process or system crash)
- the new file's ownership, permissions and extended attributes match that of the original file
Quoted 4 lines
Quoted 2 lines
4) The application that opens a file
O_REWRITEis required to rewrite the entire file.This is all essentially
open(O_CREAT|O_TRUNC|O_WRONLY)
With one big difference.
open(~/.myprog/myprog.conf, O_REWRITE|O_WRONLY) does not truncate the inode which myprog.conf lives in currently, but it opens a new inode instead.
This means that the old ~/.myprog/myprog.conf still exists, and will continue to exist until the program closes the O_REWRITE file handle - at which point the new contents will be attached to the filename.
Signature · 1 line
All rights reversed.
Quoted 28 lines · Rik van Riel
Quoted 20 lines · Andreas Dilger
Quoted 19 lines · Rik van Riel
The semantics for
O_REWRITEwould be:1) When opening a file
O_REWRITE, the file handle points at a freshly allocated, empty file. The original file is still available to programs that open the file withoutO_REWRITE.2)
O_REWRITEcan only be used in conjunction withO_WRONLY, because the file descriptor is not associated with the original file (which has data), but with an empty inode.3) The code that implements
O_REWRITE(kernel? glibc?) makes sure that:
- the new file is on the same filesystem as the original file
- the new file is not linked (so it is automatically freed after a process or system crash)
- the new file's ownership, permissions and extended attributes match that of the original fileQuoted 4 lines
Quoted 2 lines
4) The application that opens a file
O_REWRITEis required to rewrite the entire file.This is all essentially
open(O_CREAT|O_TRUNC|O_WRONLY)With one big difference.
Quoted 8 lines
open(~/.myprog/myprog.conf, O_REWRITE|O_WRONLY)does not truncate the inode which myprog.conf lives in currently, but it opens a new inode instead.This means that the old
~/.myprog/myprog.confstill exists, and will continue to exist until the program closes theO_REWRITEfile handle - at which point the new contents will be attached to the filename.
How is this any different than just having your application use mkostemp() to create a temporary dot file, then renaming it when done writing?
Trond
Hi,
Quoted 3 lines
How is this any different than just having your application use
mkostemp()to create a temporary dot file, then renaming it when done writing?
Creating a temporary file and renaming is what applications do now, but it has problems that Rik already mentioned:
- Apps forget to fsync() before rename()
- Apps forget to copy permissions over
- Apps forget to copy acls, selinux labels, and other xattrs over
- Apps create the temporary file in /tmp and then the rename() fails on systems where /tmp is a separate mount
It's not that it's not possible to do, it's that it's hard to do right. There are a huge number of hoops to jump through to just to properly save a file. And apps get it wrong. Most apps get it wrong in some way or another.
Also if it's in the kernel then filesystems could potentially hook into the process and prevent the early sync.
--Ray
Quoted 3 lines · Trond Myklebust
How is this any different than just having your application use
mkostemp()to create a temporary dot file, then renaming it when done writing?
That is exactly what it is. There are reasons for implementing it at a lower level in the system, though.
Implementing it in glibc:
- means applications get it right (today, many don't)
- allows for a performance optimization, moving the fsync into a specially spawned off temporary thread, so the main application doesn't stall
Implementing it in the kernel allows for some further performance and power optimizations, most notably:
- the sync could be turned into an ordering requirement, meaning it can be postponed and even obsoleted by a future version of the file
- the ability to postpone the write allows for better power saving
Signature · 1 line
All rights reversed.
Hi,
Quoted 8 lines · Rik van Riel
Here are my questions:
- Are these semantics useful for programs that want to replace config (or other) files with new content?
- Are these semantics sane?
- What would be the best place to implement these semantics?
IMO, people won't use this. And IMO, if we want to fix broken applications, we should do something transparent to them, i.e. something similar to what ext4 did.
Signature · 6 lines
Best Regards, Artem Bityutskiy (Артём Битюцкий) -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Rik van Riel
The ext4 automatic-fsync-on-rename discussion has shown that many applications simply Do It Wrong when it comes to rewriting configuration files.
I got the impression ext4 has automatic-fsync-on-rename-only-if-the-old-file-exists, which is a bit less reliable.
By the way, the kernel has some generic support for O_SYNC and O_DSYNC, and generic MS_SYNC mount option.
So I guess it could also have generic support for mount options "sync_on_rename" and "sync_on_close", instead of only doing it with ext4.
For example, this came up recently on the linux-mtd list which deals with flash filesystems. The ext4-like behaviour is being considerd in a flash filesystem. So if it's that important, maybe it would be even better to make it a generic VFS mount option for all filesystems.
Quoted 8 lines
Some of the common failures are:
- program overwrites the old config file
- program writes a new file, but forgets to fsync before rename
- program writes the new file in /tmp, so the rename fails on some systems
- program writes a new file and fsyncs, but forgets to give the new file the same file ownership, permission and/or extended attributes as the old file
It's also really hard to do those things from shell scripts, so they are almost never done there.
Quoted 5 lines
Glibc has the advantage of it not being in the kernel, but implementing it in-kernel might give us the opportunity for performance enhancements, like reducing step (5) to merely enforcing ordering between filesystem operations, instead of requiring an fsync.
I think the performance enhancement from order-without-sync might be useful, I'm not sure, but if so not just for this operation, which is still quite specialised.
-- Jamie
Quoted 11 lines · Jamie Lokier
Quoted 8 lines
Some of the common failures are:
- program overwrites the old config file
- program writes a new file, but forgets to fsync before rename
- program writes the new file in /tmp, so the rename fails on some systems
- program writes a new file and fsyncs, but forgets to give the new file the same file ownership, permission and/or extended attributes as the old fileIt's also really hard to do those things from shell scripts, so they are almost never done there.
That's a good point, but O_(PONIES|REWRITE) doesn't fix that problem, since shell scripts can't specify it, and shells can't do it automatically for all files created.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Hi,
Quoted 10 lines · Jamie Lokier
Quoted 9 lines
Quoted 8 lines
Some of the common failures are:
- program overwrites the old config file
- program writes a new file, but forgets to fsync before rename
- program writes the new file in /tmp, so the rename fails on some systems
- program writes a new file and fsyncs, but forgets to give the new file the same file ownership, permission and/or extended attributes as the old file
Quoted 6 lines
Quoted 2 lines
It's also really hard to do those things from shell scripts, so they are almost never done there.
That's a good point, but
O_(PONIES|REWRITE)doesn't fix that problem, since shell scripts can't specify it, and shells can't do it automatically for all files created.
Well, it's not really the shells that need to be able to do it, it's the helper utilities that shell scripts use.
For instance, sed should use O_REWRITE if -i is passed.
Still, you could imagine a new type of redirection operator that exposed O_REWRITE, e.g.,
cat >>> config-file.cfg << EOF
[config]
Value=foo
EOF
--Ray