Skip to contentSkip to conversation

aio_fsync() a directory ?

DISCUSSION Christoph Hellwig 6 participants 7 messages Apr 24, 2013 – Apr 25, 2013 6 messages not in the archive VFS

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

This change is completely contrary to real world behaviour. No modern filesystem I know of implements this behaviour as the default, and performance with the coresponding mount options (usually -o dirsync on Linux) is terrible as it forces a write out of the log (or corresponding action on non-log based filesystems) and in the common case of volatile write caches a cache flush.

Retrospectively claiming this as standards behaviour in a "clarification" is utterly wrong.

Quoted 119 lines · Xavier Roche

Just to close this thread, Geoff Clare has summarized the changes regarding aio_fsync() and relationship wrt. directory entry data.

<http://austingroupbugs.net/view.php?id=672>

"In the April 18 teleconference it was agreed that the standard should mandate that directory operations are always synchronized on conforming file systems, and should include warnings about non-conforming configurations. The proposed changes are as follows.

Changes to XBD...

At page 94 line 2581-2588 section 3.376 change:

For read, when the operation has been completed or diagnosed if unsuccessful. The read is complete only when an image of the data has been successfully transferred to the requesting process. If there were any pending write requests affecting the data to be read at the time that the synchronized read operation was requested, these write requests are successfully transferred prior to reading the data.

For write, when the operation has been completed or diagnosed if unsuccessful. The write is complete only when the data specified in the write request is successfully transferred and all file system information required to retrieve the data is successfully transferred.

to:

For read operations, when the operation has been completed or diagnosed if unsuccessful. The operation is complete only when an image of the data has been successfully transferred to the requesting process. If there were any pending write requests or (if the file is a directory) directory modifications affecting the data to be read at the time that the synchronized read operation was requested, these requests are successfully transferred prior to reading the data.

For write operations and directory modification operations, when the operation has been completed or diagnosed if unsuccessful. The operation is complete only when the written data or (if the file is a directory) modified directory entries have been successfully transferred to storage and all file system information required to retrieve them is successfully transferred.

At page 107 line 2859 add a new XBD 4.2 section (and renumber the current 4.2 and all later 4.x sections):

4.2 Directory Operations

All file system operations that read a directory or that modify the contents of a directory (for example creating, unlinking, or renaming a file) shall be completed as defined for synchronized I/O data integrity completion (see section 3.376).

<small>Note: Although conforming file systems are required to
perform all directory modifications as synchronized I/O operations, some file systems may support non-conforming configurations (for example via mount options) where directory modifications are not synchronized. Applications that rely on directory modifications being synchronized should only be used with such file systems in their conforming configuration(s).</small>

Changes to XSH...

At page 574 line 19833 section aio_fsync() change the APPLICATION USAGE section from:

None.

to:

Refer to fdatasync() and fsync().

At page 815 line 27215 section fdatasync() append to the first paragraph:

If the file is a directory, an implicit fdatasync() is already performed on every I/O operation (see XBD 4.2) and consequently if fdatasync() is called explicitly it shall take no action and shall return the value 0.

At page 815 line 27232 section fdatasync() change the APPLICATION USAGE section from:

None.

to:

Although conforming file systems are required to complete all directory modifications as defined for synchronized I/O data integrity completion, some file systems may support non-conforming configurations (for example via mount options) where directory modifications are not synchronized. When the file system is configured in this way, calls to fdatasync() on directories may cause I/O operations to be synchronized, rather than being a no-op.

At page 954 line 31987 section fsync() add a new paragraph to the APPLICATION USAGE section:

Since conforming file systems are required to complete all directory modifications as defined for synchronized I/O data integrity completion (see XBD 4.2), calling fsync() on a directory only synchronizes the file attributes such as timestamps. However, some file systems may support non-conforming configurations (for example via mount options) where modifications to directory contents are not synchronized. When the file system is configured in this way, calls to fsync() on directories may cause directory contents to be synchronized in addition to file attributes.

Changes to XRAT...

At page 3444 line 115531 add a new XRAT A.4.2 section (and renumber the current A.4.2 and all later A.4.x sections):

A.4.2 Directory Operations

Quoted 10 lines

Earlier versions of this standard did not make clear that all directory modifications are performed as synchronized I/O operations, although that is the historical behavior and was always intended. Applications have no need to call fdatasync() or fsync() on a directory unless they want to synchronize the file attributes (using fsync()), provided the directory is on a conforming file system. However, since applications may wish to use fdatasync() or fsync() to synchronize directory modifications on non-conforming file systems, implementations are required to support fdatasync() on directories as a no-op on conforming file systems."

---end quoted text---

#7 lore raw
Quoted 8 lines · Christoph Hellwig

This change is completely contrary to real world behaviour. No modern filesystem I know of implements this behaviour as the default, and performance with the coresponding mount options (usually -o dirsync on Linux) is terrible as it forces a write out of the log (or corresponding action on non-log based filesystems) and in the common case of volatile write caches a cache flush.

Retrospectively claiming this as standards behaviour in a "clarification" is utterly wrong.

As I understand the wording, the original intent was to have all file operations synchronized (ie. operation committed permanently) at the "beginning" of the specification. The updated one suggests that an implementation may be non-conformant, and allow fsync() on a directory entry.

I'm also a bit puzzled by the default synchronization guarantee (which would lead an I/O per open/rename/etc. operation ?) ; even if many developers just do not care about filename synchronization. [ they are probably assuming that the fsync() on the file itself will also flush the related directory entrie(s) ]

maintainer
#8 lore raw
Quoted 7 lines · Xavier Roche
Quoted 6 lines · Christoph Hellwig

This change is completely contrary to real world behaviour. No modern filesystem I know of implements this behaviour as the default, and performance with the coresponding mount options (usually -o dirsync on Linux) is terrible as it forces a write out of the log (or corresponding action on non-log based filesystems) and in the common case of volatile write caches a cache flush.

Quoted 8 lines
Quoted 2 lines

Retrospectively claiming this as standards behaviour in a "clarification" is utterly wrong.

As I understand the wording, the original intent was to have all file operations synchronized (ie. operation committed permanently) at the "beginning" of the specification. The updated one suggests that an implementation may be non-conformant, and allow fsync() on a directory entry.

Original intent of _what_? With all due respect, original intent of Austin Group is none of our concern - "we had always intended to make that promise on your behalf, now we'd simply got around to saying so clearly" inspires many things, but "oh, sure, it's binding for us now" is *not* one of those. Suggestions to do anatomically impossible things, OTOH...

#9 lore raw

Take this up with the POSIX committee. Complaining on implementors' mailing lists won't have any effect on the future text of standards.

#10 lore raw
Quoted 2 lines · Roland McGrath

Take this up with the POSIX committee. Complaining on implementors' mailing lists won't have any effect on the future text of standards.

Absolutely. I'd like to mention that the austin-group-l mailing-list is opened to anyone wishing to contribute (with proper technical arguments, of course), and AFAICS opened to remarks (this bug was opened following one of my remarks - so you may consider this as my fault :p)

#11 lore raw
Quoted 9 lines · Xavier Roche
Quoted 2 lines · Roland McGrath

Take this up with the POSIX committee. Complaining on implementors' mailing lists won't have any effect on the future text of standards.

Absolutely. I'd like to mention that the austin-group-l mailing-list is opened to anyone wishing to contribute (with proper technical arguments, of course), and AFAICS opened to remarks (this bug was opened following one of my remarks - so you may consider this as my fault :p)

I'm guilty of a similar mistake, with issue #663, and would appreciate any supporting arguments against the ridiculous change in the standard they want to make and pretend is just a clarification of historic practice. The proposed change also renders glibc non-conforming so I think it's on-topic.

Rich

#12 lore raw

I have followed up with the Austin Group, here:

http://austingroupbugs.net/view.php?id=672#c1545

and would appreciate further review and followup as needed. To add further comments to that page, one must be an Austin Group member; if you're not one already, you can easily become one by using the "Signup for a new account" link on that page.