Skip to contentSkip to conversation

RFC: O_PONIES semantics (well O_REWRITE)

DISCUSSION Rik van Riel 7 participants 10 messages Jun 11, 2009 – Jun 17, 2009 VFS

lore
Share & tools
lore
#0 lore raw

The ext4 automatic-fsync-on-rename discussion has shown that many applications simply Do It Wrong when it comes to rewriting configuration files.

Some of the common failures are:
- program overwrites the old config file
- program writes a new file, but forgets to fsync before rename
- program writes the new file in /tmp, so the rename fails on some systems
- program writes a new file and fsyncs, but forgets to give the new file the same file ownership, permission and/or extended attributes as the old file

Magically taking care of filesystem semantics for every use may not be possible (no O_PONIES for you!), but I believe we can help the applications that just want to completely rewrite a file and atomically replace it.

The semantics for O_REWRITE would be:

1) When opening a file O_REWRITE, the file handle points at a freshly allocated, empty file. The original file is still available to programs that open the file without O_REWRITE.

2) O_REWRITE can only be used in conjunction with O_WRONLY, because the file descriptor is not associated with the original file (which has data), but with an empty inode.

3) The code that implements O_REWRITE (kernel? glibc?) makes sure that:
- the new file is on the same filesystem as the original file
- the new file is not linked (so it is automatically freed after a process or system crash)
- the new file's ownership, permissions and extended attributes match that of the original file

4) The application that opens a file O_REWRITE is required to rewrite the entire file.

5) On close(), the code that implements O_REWRITE makes sure that the file is atomically renamed, so that if a system crash happens, the user will see either the old or the new file contents, but never an empty file.

6) After close(), processes that open the file will get the new content. Processes that previously opened the file will hold on to the old inode and get old contents.

Here are my questions:

- Are these semantics useful for programs that want to replace config (or other) files with new content?

- Are these semantics sane?

- What would be the best place to implement these semantics?

Relying on application developers to get it right seems to not have worked out well, so I'm thinking kernel or glibc. Glibc has the advantage of it not being in the kernel, but implementing it in-kernel might give us the opportunity for performance enhancements, like reducing step (5) to merely enforcing ordering between filesystem operations, instead of requiring an fsync.

Signature · 1 line
All rights reversed.
#1 lore raw
Quoted 19 lines · Rik van Riel

The semantics for O_REWRITE would be:

1) When opening a file O_REWRITE, the file handle points at a freshly allocated, empty file. The original file is still available to programs that open the file without O_REWRITE.

2) O_REWRITE can only be used in conjunction with O_WRONLY, because the file descriptor is not associated with the original file (which has data), but with an empty inode.

3) The code that implements O_REWRITE (kernel? glibc?) makes sure that:
- the new file is on the same filesystem as the original file
- the new file is not linked (so it is automatically freed after a process or system crash)
- the new file's ownership, permissions and extended attributes match that of the original file

Quoted 2 lines

4) The application that opens a file O_REWRITE is required to rewrite the entire file.

This is all essentially open(O_CREAT|O_TRUNC|O_WRONLY)

Quoted 4 lines

5) On close(), the code that implements O_REWRITE makes sure that the file is atomically renamed, so that if a system crash happens, the user will see either the old or the new file contents, but never an empty file.

This would be possible if the kernel set the i_size=0, but didn't send the filesystem the truncate until the file was closed and being flushed.

Quoted 3 lines

6) After close(), processes that open the file will get the new content. Processes that previously opened the file will hold on to the old inode and get old contents.

What is the benefit of (6)? Of all these semantics this is the one that would cause the most confusion I think.

Quoted 8 lines

Here are my questions:

- Are these semantics useful for programs that want to replace config (or other) files with new content?

- Are these semantics sane?

- What would be the best place to implement these semantics?

The main question is - would any applications use O_REWRITE in the first place, or would it just make sense to have a helper function in glibc like e.g. mktemp that handles the "atomic update of config file" properly in the first place.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#6 lore raw
In reply to Rik van Riel

Hi,

Quoted 8 lines · Rik van Riel

Here are my questions:

- Are these semantics useful for programs that want to replace config (or other) files with new content?

- Are these semantics sane?

- What would be the best place to implement these semantics?

IMO, people won't use this. And IMO, if we want to fix broken applications, we should do something transparent to them, i.e. something similar to what ext4 did.

Signature · 6 lines
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw
In reply to Andreas Dilger
Quoted 20 lines · Andreas Dilger
Quoted 19 lines · Rik van Riel

The semantics for O_REWRITE would be:

1) When opening a file O_REWRITE, the file handle points at a freshly allocated, empty file. The original file is still available to programs that open the file without O_REWRITE.

2) O_REWRITE can only be used in conjunction with O_WRONLY, because the file descriptor is not associated with the original file (which has data), but with an empty inode.

3) The code that implements O_REWRITE (kernel? glibc?) makes sure that:
- the new file is on the same filesystem as the original file
- the new file is not linked (so it is automatically freed after a process or system crash)
- the new file's ownership, permissions and extended attributes match that of the original file

Quoted 4 lines
Quoted 2 lines

4) The application that opens a file O_REWRITE is required to rewrite the entire file.

This is all essentially open(O_CREAT|O_TRUNC|O_WRONLY)

With one big difference.

open(~/.myprog/myprog.conf, O_REWRITE|O_WRONLY) does not truncate the inode which myprog.conf lives in currently, but it opens a new inode instead.

This means that the old ~/.myprog/myprog.conf still exists, and will continue to exist until the program closes the O_REWRITE file handle - at which point the new contents will be attached to the filename.

Signature · 1 line
All rights reversed.
#3 lore raw
Quoted 28 lines · Rik van Riel
Quoted 20 lines · Andreas Dilger
Quoted 19 lines · Rik van Riel

The semantics for O_REWRITE would be:

1) When opening a file O_REWRITE, the file handle points at a freshly allocated, empty file. The original file is still available to programs that open the file without O_REWRITE.

2) O_REWRITE can only be used in conjunction with O_WRONLY, because the file descriptor is not associated with the original file (which has data), but with an empty inode.

3) The code that implements O_REWRITE (kernel? glibc?) makes sure that:
- the new file is on the same filesystem as the original file
- the new file is not linked (so it is automatically freed after a process or system crash)
- the new file's ownership, permissions and extended attributes match that of the original file

Quoted 4 lines
Quoted 2 lines

4) The application that opens a file O_REWRITE is required to rewrite the entire file.

This is all essentially open(O_CREAT|O_TRUNC|O_WRONLY)

With one big difference.

Quoted 8 lines

open(~/.myprog/myprog.conf, O_REWRITE|O_WRONLY) does not truncate the inode which myprog.conf lives in currently, but it opens a new inode instead.

This means that the old ~/.myprog/myprog.conf still exists, and will continue to exist until the program closes the O_REWRITE file handle - at which point the new contents will be attached to the filename.

How is this any different than just having your application use mkostemp() to create a temporary dot file, then renaming it when done writing?

Trond

#4 lore raw

Hi,

Quoted 3 lines

How is this any different than just having your application use mkostemp() to create a temporary dot file, then renaming it when done writing?

Creating a temporary file and renaming is what applications do now, but it has problems that Rik already mentioned:

- Apps forget to fsync() before rename()
- Apps forget to copy permissions over
- Apps forget to copy acls, selinux labels, and other xattrs over
- Apps create the temporary file in /tmp and then the rename() fails on systems where /tmp is a separate mount

It's not that it's not possible to do, it's that it's hard to do right. There are a huge number of hoops to jump through to just to properly save a file. And apps get it wrong. Most apps get it wrong in some way or another.

Also if it's in the kernel then filesystems could potentially hook into the process and prevent the early sync.

--Ray

#7 lore raw
In reply to Rik van Riel
Quoted 3 lines · Rik van Riel

The ext4 automatic-fsync-on-rename discussion has shown that many applications simply Do It Wrong when it comes to rewriting configuration files.

I got the impression ext4 has automatic-fsync-on-rename-only-if-the-old-file-exists, which is a bit less reliable.

By the way, the kernel has some generic support for O_SYNC and O_DSYNC, and generic MS_SYNC mount option.

So I guess it could also have generic support for mount options "sync_on_rename" and "sync_on_close", instead of only doing it with ext4.

For example, this came up recently on the linux-mtd list which deals with flash filesystems. The ext4-like behaviour is being considerd in a flash filesystem. So if it's that important, maybe it would be even better to make it a generic VFS mount option for all filesystems.

Quoted 8 lines

Some of the common failures are:
- program overwrites the old config file
- program writes a new file, but forgets to fsync before rename
- program writes the new file in /tmp, so the rename fails on some systems
- program writes a new file and fsyncs, but forgets to give the new file the same file ownership, permission and/or extended attributes as the old file

It's also really hard to do those things from shell scripts, so they are almost never done there.

Quoted 5 lines

Glibc has the advantage of it not being in the kernel, but implementing it in-kernel might give us the opportunity for performance enhancements, like reducing step (5) to merely enforcing ordering between filesystem operations, instead of requiring an fsync.

I think the performance enhancement from order-without-sync might be useful, I'm not sure, but if so not just for this operation, which is still quite specialised.

-- Jamie

#8 lore raw
Quoted 11 lines · Jamie Lokier
Quoted 8 lines

Some of the common failures are:
- program overwrites the old config file
- program writes a new file, but forgets to fsync before rename
- program writes the new file in /tmp, so the rename fails on some systems
- program writes a new file and fsyncs, but forgets to give the new file the same file ownership, permission and/or extended attributes as the old file

It's also really hard to do those things from shell scripts, so they are almost never done there.

That's a good point, but O_(PONIES|REWRITE) doesn't fix that problem, since shell scripts can't specify it, and shells can't do it automatically for all files created.

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#9 lore raw

Hi,

Quoted 10 lines · Jamie Lokier
Quoted 9 lines
Quoted 8 lines

Some of the common failures are:
- program overwrites the old config file
- program writes a new file, but forgets to fsync before rename
- program writes the new file in /tmp, so the rename fails on some systems
- program writes a new file and fsyncs, but forgets to give the new file the same file ownership, permission and/or extended attributes as the old file

Quoted 6 lines
Quoted 2 lines

It's also really hard to do those things from shell scripts, so they are almost never done there.

That's a good point, but O_(PONIES|REWRITE) doesn't fix that problem, since shell scripts can't specify it, and shells can't do it automatically for all files created.

Well, it's not really the shells that need to be able to do it, it's the helper utilities that shell scripts use.

For instance, sed should use O_REWRITE if -i is passed.

Still, you could imagine a new type of redirection operator that exposed O_REWRITE, e.g.,

cat >>> config-file.cfg << EOF
[config]
Value=foo
EOF

--Ray

#5 lore raw
In reply to Trond Myklebust
Quoted 3 lines · Trond Myklebust

How is this any different than just having your application use mkostemp() to create a temporary dot file, then renaming it when done writing?

That is exactly what it is. There are reasons for implementing it at a lower level in the system, though.

Implementing it in glibc:
- means applications get it right (today, many don't)
- allows for a performance optimization, moving the fsync into a specially spawned off temporary thread, so the main application doesn't stall

Implementing it in the kernel allows for some further performance and power optimizations, most notably:
- the sync could be turned into an ordering requirement, meaning it can be postponed and even obsoleted by a future version of the file
- the ability to postpone the write allows for better power saving

Signature · 1 line
All rights reversed.