ext3-2.4-0.9.4

An update to the ext3 filesystem for 2.4 kernels is available at

	http://www.uow.edu.au/~andrewm/linux/ext3/

The diffs are against linux-2.4.7 and linux-2.4.6-ac5.

The changelog is there. One rarely-occurring but oopsable bug was fixed and several quite significant performance enhancements have been made. These are in addition to the performance fixes which went into 0.9.3.

Ted has put out a prelease of e2fsprogs-1.23 which supports filesystem type `auto' in /etc/fstab, so it is now possible to switch between ext3- and non-ext3-kernels without changing any configuration.

It is recommended that users of earlier ext3 releases upgrade to 0.9.4.

For people who are undertaking performance testing, it is perhaps useful to point out that ext3 operates in one of three different journalling modes, and that these modes have very different functionality and very different performance characteristics. Really, you need to test all three and balance the functionality which each mode offers against the throughput which you obtain in your application.

The modes are:

data=writeback

This is classic metadata-only journalling. File data is written back to the main fs lazily. After a crash+recovery the fs's structural integrity is preserved, but the *contents* of files can and will contain old, stale data. Potentially hundreds of megabytes of it.

This is the fastest mode for normal filesystem applications.

data=ordered

The fs ensures that file data is written into the main fs prior to committing its metadata. Hence after a crash+recovery, your files will contain the correct data.

This is the default operating mode and throughput is good. It adds about one second to a four minute kernel compile when
compared with ext2. Under heavier loads the difference
becomes larger.

data=journal

All data (as well as to metadata) is written to the journal before it is released to the main fs for writeback.

This is a specialised mode - for normal fs usage you're better off using ordered data, which has the same benefits of not corrupting data after crash+recovery. However for applications which require synchronous operation such as mail spools and synchronously exported NFS servers, this can be a performance win. I have seen dbench figures in this mode (where the files were opened O_SYNC) running at ten times the throughput of ext2. Not that this is the expected benefit for other applications!

Looking at the above issues, one may initially think that the post-recovery data corruption is a serious issue with writeback mode, and that there are big advantages to using journalled or ordered data.

However, even in these modes the affected files may be shorter-than-expected after recovery, because the app hadn't finished writing them yet. And usually, a truncated file is just as useless as one which contains garbage - it needs to be deleted.

It's not really as simple as that - for small (< a few hundred k) files, it tends to be the case that either the whole file is intact after a crash, or none of it is. This is because the journalling mechanism starts a new transaction every five seconds, and a typical open/write/close operation usually fits entirely inside this window.

There is also a security issue to be considered: a recovered writeback-mode filesystem will expose other people's old data to unintended recipients.

Hopefully this description will help people make their deployment choices. If not, assistance is available on the [email protected] mailing list.

-

In ordered and journal mode, are meta data operations, namely creating a file, rename(), link(), unlink() "synchronous" in the sense that after the call has returned, the effect of this call is never lost, i. e., if link(2) has returned and the machine crashes immediately, will the next recovery ALWAYS recover the link?

Or will ext3 still need chattr +S?

Does it still support chattr +S at all?

Synchronous meta data operations are crucial for mail transfer agents such as Postfix or qmail. Postfix has up until now been setting chattr +S /var/spool/postfix, making original (esp. soft-updating) BSD file systems significantly faster for data (payload) writes in this directory than ext2.

Note: I'm not on the ext3-users list. Please Cc: back replies.

Matthias Andree

No, they're not synchronous by default. After recovery they will either be wholly intact, or wholly absent.

Yes, if the app doesn't support O_SYNC or fsync(). I believe that MTA's *do* support those things.

Yes.

If postfix is capable of opening the files O_SYNC or of doing fsync() on them then the `chattr +s' is no longer necessary - unlike ext2, when the O_SYNC write() or the fsync() return, the directory contents (as well as the inode, bitmaps, data, etc) will all be tight on disk and will be restored after a crash.

This should speed things up considerably, especially with journalled-data mode. I need to test and characterise this some more to come up with some quantitative results and configuration recommendations.

BTW, if you have more-than-modest throughput requirements, don't even *think* of mounting the fs with `mount -o sync'. Our performance in this mode is terrible :(

I have a hack somewhere which fixes this as much as it can be fixed, but I didn't even bother committing it. It's feasible, but tiresome.

A better solution is to fix some lock inversion problems in the core kernel which prevent optimal implementation of data-journalling filesystems. I don't really expect this to occur medium-term or ever.

A middle-ground solution may be to add an fs-private `osync' mount option, so all files are treated similarly to O_SYNC, which would work well.

-

...

You seem to be missing the point, because I wasn't verbose enough, so I will try to rephrase this and explain. This may turn out to be a feature request. :-}

Before going into detail, MTAs do know about fsync(). ext3 synching relevant directory parts as part of fsync() is a great achievement. Finally, more than five years after initial complaints, Linux is SLOWLY getting somewhere for speeding up reliable MTA operation.

But that's the smaller piece. Common MTAs such as Postfix or qmail rename or link files into place (their queues, the mail spool). With the advent of journalling came the atomicity of rename operations. That's also a great achievement.

However, the remaining problem is being synchronous with respect to open (fixed for ext3 with your fsync() as I understand it), rename, link and unlink. With ext2, and as you write it, with ext3 as well, there is currently no way to tell when the link/rename has been committed to disk, unless you set mount -o sync or chattr +S or call sync() (the former is not an option because it's far too expensive).

The official statement by Dr. Wietse Venema (who wrote Postfix) is, Postfix REQUIRES synchronous directory updates (open, rename, link, unlink, in order of decreasing importance). Wietse refuses to wrap all these calls for Linux.

Similar assumptions hold for qmail.

So, what would help the common MTA? osync wouldn't, MTAs know how to use fsync(). dirsync or bsdstyle or however it's called, as chattr and mount options, would help. This option should make all directory operations (open/creat/fsync, rename, link, unlink, symlink, possibly close) synchronous in respect to affected directory and meta data while leaving application data (payload) operations asynchronous (applications can then choose when to call fsync() to flush the data to disk).

A much better file system for an MTA might be ext3fs with data=journalled and dirsync mount/chattr option. Would you deem it possible to get such an option done before ext3fs 1.0.0?

I hope this makes the requirements of this particular group of applications clear.

Thanks again to everyone involved with the ext3fs development.

Matthias Andree

Not relying on non-supported semantics to save your ass.

Rename() is atomic in the sense that you either see the old name or the new name, but I don't know of systems which guarantee atomicity across a system crash.

In fact, knowing how hard disks work mechanically, only journaling filesystems could have an extention to make this work. Ie. this is NOT something you can rely on ;)

regards,

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

This is not about failing hard disks. It is about premature acknowledgment of something which has not happened at that time.

Linux cannot possibly fix all incomplete protocols, specifications and implementation, but it can fix its own behaviour.

Everything is about speed, and allowing the MTA to use a (weaker) dirsync rather than allsync option would speed things up without sacrificing reliability.

MTA reliability is NOT about failing disk drives. If it falls over, you notice that. If files are in the wrong directory or not there at all, you don't necessarily notice until someone complains.

Please don't get in the way of finally fixing things just because someone might have a broken item that could endanger your data. I have a huge magnet here...

The competition is there and it has names: BSD + ufs + softupdates, Solaris + logging ufs. Read MTA mailing lists before obstructing.

Thanks.

Matthias Andree

So you didn't read what I was writing.

Let me explain it to you slowly:

Disks. Write. One. Write. At. A. Time.

A rename often needs as many as 4 or 5 writes, ergo, you CANNOT do a rename atomically without journaling and transactions.

BSD + softupdates is physically incapable of doing what you suggest it does. This can be proven from the lack of transactions and the way hard disks work physically.

regards,

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

Rik is right. It isnt just about premature notification - its about atomicity. At the point you are notified the data has been queued for disk I/O. Even on traditional BSD ufs with synchronous metadata you still had points where a crash left the rename partially complete and nothing but a log or an atomic update system is going to fix that.

All of which are - not unsuprisingly - using a log. In fact Solaris logging ufs and ext3 are very similar ideas - adding a log to an existing fs.

Alan

Softupdates isn't using logging. Furthermore, even the journaling filesystems won't all guarantee that the various parts of a rename() operation will all be in the same transaction.

An MTA which relies on this is therefore Broken(tm).

cheers,

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

umm.. I'd certainly hope that they do guarantee this.

The only operations which can't trivially fit into a single transaction are write() and truncate().

-

MTAs rely on TRULY, ULTIMATELY AND DEFINITELY SYNCHRONOUS directory updates, nothing else. And because they do so, and most systems have them, and MTAs are portable, they choose chattr +S on Linux. And that's a performance problem because it doesn't come for free, but also with synchronous data updates, which are unnecessary because there is fsync().

That's already the complete story about MTAs on Linux.

If Linux HAD a mode (it doesn't) to have just synchronous directory updates, MTAs could stop using chattr +S and be faster.

MTAs do NOT care how the file system is internally managed, they only rely on the rename operation having completed physically on disk before the "my rename call has returned 0" event. They expect that with the call returning the rename operation has completed ultimately, finally, for good, definitely and the old file will not reappear after a crash.

(Note that the atomicity addressed in the man pages and Unix specifications is a different one: it deals with the visibility of the changes in the system, not with the functioning of the file system.)

That's why *BSD + softupdates is still recommended over Linux for pure mail transfer agents by people.

This still implies the drive doesn't lie to the OS about the completion of write requests: write cache == off.

Matthias Andree

And thus they are broken, all caps don't make that less true.

"and most systems have them"...

So they rely on undocumented and non standadisized semantics of some implementations. I'd call this buggy.

Christoph

Whip me.  Beat me.  Make me maintain AIX.

chattr +S and atomic updates hitting disk then returning to the app will give the same performance. You can also fsync() the directory.

Actually the old file re-appearing after the crash is irrelevant. It will have a previously logged message id. And if you are not doing message id histories then you have replay races at the SMTP level anyway

Write cache off is not a feature available on many modern disks. You already lost the battle before you started.

Alan

Losing the battle of brain-dead hardware is not a problem... :-)

SCSI hard disks are expected to follow the specifications. But, may be, you are referring to IDE disks, only ...

With SCSI, you can enable write caching and also ask the device to signal completion of actual write to the media by setting the FUA bit in the SCSI command block (not available in WRITE(6), but available in WRITE(10)).

Gérard.

Christoph Hellwig schrieb am Donnerstag, den 26. Juli 2001:

If each in the set of "supported systems" document this behaviour for themselves, there is no bug. I didn't check however for systems other than FreeBSD 4.x and Linux. And "Linux support" forces these semantics with chattr +S, at a high price.

Go tell your opinion to those people that refuse to wrap their rename/link calls with open()/fsync() calls to the respective parents, particularly Daniel J. Bernstein, Wietse Z. Venema, among others. I don't possibly know all MTAs.

You will encounter these or similar questions/objections:

1. what systems apart from Linux need this kind of Pampers?

2. manual lookups of parent directories cause additional overhead better avoided in performance critical systems.

You would not be the first one to tell them...

I've pointed things out to Mr Bernstein before. His normal replies are not helpful and generally vary between random ravings and threatening to sue people who publish things on web pages he disagrees with.

Alan

The MTA depends on behaviour which is undefined. Now you want to go blame the OS ?

If you care about your email, probably you should either teach these people about standards like POSIX or SuS (and tell them to not rely on undefined behaviour) or switch to an MTA which isn't broken in various ways ;)

cheers,

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

POSIX and SuS are actually not helpful here. They don't cover how to force namespace to disk, only data and metadata for the file. So you can portably stick your data onto disk, portably be sure its on disk, but not portably be sure the directory entries are on disk.

Alan

In article <[email protected]>,

Now, now, Alan. He has strong opinions, I'll agree, but I've never see him threaten to _sue_.

Also, I think he eventually agreed on the logic of fsync() on the directory, and we even had a bug report (quickly fixed) for reiserfs because it got confused by it.

Of course, knowing Dan, I suspect the fsync() is accompanied by several lines of derogatory comments about the need for it (not that I've checked).

Everybody tends to agree that synchronous IO is stupid and slow, but some people are just so fixated with "That is how it has been done for 20 years..".

Logging filesystems together with explicit logging points (namely, "fsync()") are very obviously a superior answer from a technical standpoint, but that doesn't impact the emotional arguments ("but I want things to stay the same!").

Linus

Ask Alexey about the end of the syncookie "debate"

[snip]

In looking at the synchronous directory options, I'm unsure as to the 'real' status wrt fsync() on a directory:
1) Does fsync() of a directory work on most/all current FS?
2) Does it work on 2.2.x as well as 2.4.x?

Rick Nelson
"... being a Linux user is sort of like living in a house inhabited
by a large family of carpenters and architects. Every morning when
you wake up, the house is a little different. Maybe there is a new
turret, or some walls have moved. Or perhaps someone has temporarily
removed the floor under your bed." - Unix for Dummies, 2nd Edition
	-- found in the .sig of Rob Riggs, [email protected]

Modulo bugs, yes.

Now, there's another issue, of course: if you have an important mail-spool on some of the less tested filesystems, I would consider you crazy regardless of fsync() working ;). I don't think anybody has ever verified that fsync() (or much anything else wrt writing) does the right thing on NTFS, for example.

Yes. However, there may be performance issues. As with just about anything, we didn't start optimizing things until it became a real issue, and in some cases at least historically the filesystems fell back on just doing a whole "fsync_dev()" if they had nothing better to do.

I think later 2.2.x kernels (ie the ones past the point where Alan took over) probably have the fsync() optimizations at least for ext2.

Linus

Great, that was a big concern

Caveat Emptor ;-)

That should be recent enough - I push 2.2.19 for shm support and security reasons anyway - though I see alot of folk on 2.2.16/17.

Are the optimizations more than writing out only changed blocks? Has anyone any information on the performance differences between optimized vs non-optimized?

Thanks, I'm feeling much better about getting this support added

Rick Nelson
Life'll kill ya                         -- Warren Zevon
Then you'll be dead                     -- Life'll kill ya

In the for what it is worth department, I spent the day with Daniel after the kernel summit meeting a while back, we talked file systems for about 6 or 7 hours. While I'll plead guilty to getting mad at him (his ego is up there with mine :-), I came away impressed with his knowledge. I get the feeling that he thinks deeply about the problems he works on, he's probably right a lot of the time, *and* as with many deep thinkers, he has a problem communicating his ideas.

This is a common problem, and I'm not sure Daniel is fully aware of it. One cannot expect other people to have done the same thinking and have the same context, and when they do not, it is easy to get frustrated. I think that some of Daniel's "ravings" are probably just frustration that the other person "doesn't get it".

That doesn't mean that Daniel is the right hand of God or anything, I've seen him do some stupid things but I've seen all of us do some stupid things, so that doesn't mean much. I think Daniel does way more smart things than stupid things, and not all of us can claim that (sort of like half of the drivers are below average, noone likes that idea either).

What I'm trying to say is that I think Daniel is one of the good guys, even though his user interface could stand improvement (a common thing amongst smart people) and it looks like it would be smart to figure out how to work with him.

Just my opinion...

---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

there's a work-in-progress called ReiserSMTP[1] which rewrites some bits of qmail so it works better with ReiserFS, although i can imagine that it would improve things on Linux as a whole.

this is getting off-topic, but since the various parties involved (Linux vs djb/Wietse/etc[2]) are probably never going to agree on semantics, i'm wondering if it's possible to ask them to write the software in such a way that it's possible to 'drop in' your own functions relevant for sync'ing. then the MTA writers can go and use their traditional filesystem assumptions and Linux users can produce very small patches to support the correct behaviour under Linux.

it would be _nice_ if the ext3 guys would be more willing to implement directory-syncing on link/rename/etc, though, even as an option. a 'mount -o dirsync' would be enough; no need for chattr +D stuff. Linux tends to have a bad name as a platform as an MTA just because of all this, which is a shame. it would be nice if a fix is possible. *nudge nudge Mr. Morton* :)

[1] http://www.jedi.claranet.fr/reisersmtp.html

[2] hey, this might be the first time they agree on anything!

#ozone/algorithm <[email protected]>          - trust.in.love.to.save

No, the behaviour is defined on certain systems. Not sure if that comprises all supported systems.

I'm not blaming anybody besides Linux which does not offer the "noasync" (FreeBSD) compromise between sync and async. I don't see any reason why this option cannot be there. Is it too expensive too implement? No-one said so.

I cannot tell if and how the MTA authors checked all their supported OSs how they handle metadata updates.

Wee. And then, I tell the system to comply with that as well, don't I? ;)

Matthias Andree

Arrg, I take it all back, I'm taking about Daniel Phillips not Daniel Bernstein. I tend to agree with Alan about Mr Bernstein.

Thanks to Richard Gooch for pointing out that I'm asleep at the switch.

---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

It stopped due to flakiness on the part of all parties including myself, the programmer, and the sponsor, but it would be nice if a sponsor and programmer came along to restart it.

No, Linus is right and the MTA guys are just wrong. The mailers are the place to fix things, not the kernel. If the mailer guys want to depend on the kernel being stupidly designed, tough. Someone should fix their mailer code and then it would run faster on Linux than on any other platform.

Hans

Heh, very interesting, but you seem to have created a collage of two different Daniels ;-)

--
Daniel

Perhaps I didn't understand the requirement.

I believe that `dirsync' would provide synchronous metadata operations (ie: the metadata is crashproofed on-disk when the syscall returns), but non-sync data. Correct?

Whereas mount -o sync' or chattr +S' would provide synchronous metadata operations PLUS synchronous data, so when write() returns, the data which was written is crashproofed.

Is that your understanding of the difference?

If so, then with `dirsync', the application would have to open the file O_SYNC (which would make the whole thing pointless!) or it would run fsync() when it had finished writing the file.

So what it boils down to is that dirsync will improve the efficiency of applications which do a bunch of small writes and then an fsync.

If, however, the application is capable of doing a nice big write() (setvbuf!) then really, the two things will be pretty much the same.

Wait and see how the benchmarks turn out, yes?

One problem at present is that an application could be in the middle of a nice big write(), but another thread comes up and does a synchronous creat(). That will force a commit right in the middle of the write(). It would be better (I think) if the write's transaction were allowed to run to completion and the creat() caller blocks until the write() finishes - this way the write(), the creat() and anything else which happened during the write() would all be written out in a single compound transaction.

Alas, we cannot run a transaction handle for more than a single page in write() because of locking inversion problems with i_sem and the lock_page outside ->writepage(). i_sem is trivial to fix, but writepage is not. It has not really proven to be a problem yet, but it would be nice to be able to _guarantee_ that writes up to a particular size (100k, say) were 100% atomic.

-

Hi,

I'm one of those icky application programmers attempting to make reliable software across different versions of Unix.

We need to get data to disk portably, quickly, and reliably.

I love it when I see things like: "No, Linus is right and the MTA guys are just wrong."

This sort of attitude is just ridiculous. Unix had a defined set of semantics. This might have been stupid semantics, but it had them. Then journalling filesystems, softupdates, and Linux async updates came along and destroyed those semantics, preventing those of us who want to write reliable applications using the filesystem from doing so. At least Oracle doesn't change the definition of COMMIT.

When I contacted the Linux JFS team about the semantics of link(), I was told that there is _no way_ of forcing a link() to disk. Not an fsync() on the file, not an fsync() on the directory, just _not possible_.

Great.

Then we come to ext2. "Oh, just call fsync() on the directory and you'll be fine." Well, wait, a second, if ext2 isn't ordering the metadata writes, a crash at the wrong time (whether or not I've called fsync()) may lose directory entries---even directory entries unrelated to the files I'm doing operations on! Greeeeat.

Thus why all reasonably paranoid MTAs and other mail programs say "use chattr +S on ext2"---we need ordered metadata writes.

Ok, journalled filesystems are better. At least crashes aren't going to affect random files on disk. But since link() and the like don't force a commit, we need some way---some reasonably portable way---of getting that on disk. On softupdates, calling fsync() on a file forces all directory entries pointing to that file to disk. This is pretty reasonable. 1 fsync() call.

Why do we all cringe when we're told to call fsync() on the directory? Several reasons:
. not needed on any other variety of Unix
. two fsync() calls force two different syncronization points: the
application is forcing ordering on the OS that may not be needed. (Thus performance doesn't "fly" when you need multiple fsyncs.)
. directory may have other modifications going on that we're not
interested in

You want to help performance? Give us an fsync() that works on multiple file descriptors at once, or an async fsync() call. Don't make us fight the OS on getting data to disk.

Larry

The stuff you people seem to insist on, however, most definately isn't part of the defined set of semantics.

If you believe otherwise, feel free to point out the relevant sections in POSIX / SuS / ...

regards,

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

And even if it's "inherited wisdom", you cannot simply tell those people "don't rely on that" if - as claimed - you can't even force a link() to disk.

The standard is only useful if it specifies how to get data safely on disk - it is quite explicit for fsync(), but you evidently cannot fsync() a link().

Matthias Andree

As Linus said, fsync() on the directory.

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

Relying on that to work on other operating systems is no better than demanding synchronous meta data writes: relying on undocumented behaviour.

If we spake about Linux-specific applications, that'd be okay, but we speak about portable applications, and the diversity is bigger than useful. Speed is not the only problem the OS has to solve.

Matthias Andree

I guess many MTAs have a small libc inside of them exactly in order to handle things like this without fouling up the core code too much.

Time to make your favorite MTA use link_slowly() ;)

cheers,

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

You are blurring the boundaries between "undocumented behavior" and "OS-specific behavior". fsync() on a directory to sync metadata is a defined (according to my copy of fsync(2)), Linux-specific behavior. It is also very reasonable IMHO and in keeping with the traditional Unix notion of directories as lists of files.

I argue that using defined Linux behavior to implement what you want on Linux systems _is_ better than relying on undocumented behavior, and I think most people would agree. If you don't do this you have not really ported the software to Linux; you instead have some standards compliant software that "kinda usually works on Linux". You could argue that no one should localize their software to different versions of Unix, but you would be by far in the minority.

http://www.google.com/search?q=autoconf

Writing portable Unix software has always meant some degree of system-specific accomodation. It's a bummer but it's life; otherwise Unix wouldn't evolve.

miket

No-one claims that fsync() the directory is a bad interface - it's non-portable however. Actually, chattr +S is well-documented - it just doesn't work on ReiserFS or Minix for now, and it may be unnecessarily slow on ext2.

As pointed out more than once, "synchronous meta data" is documented e. g. for FreeBSD, so in at least these two cases, the box relies on documented behaviour.

How can autoconf figure if you need to fsync() the directory? Apart from that, which Unix MTA uses autoconf?

Remember, the whole discussion is about getting rid of the need for chattr +S and offering the admin the chance to mount or flag a directory for synchronous meta data updates.

Matthias Andree

Zmailer uses autoconf, Exim also has some nice
tool to make itself build for the right OS using
the right interfaces.

regards,

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

Simple! Grep the fsync(2) manpage ;)

Ketil the joker

My point was not that they should be using autoconf; I don't know if they are or not. My point was that they should use existing published interfaces that are reasonable, rather than push for guarantees that impose new requirements on filesystems. And even without autoconf it's not hard to figure out what system you're running on.

    rename(tmpfile, spoolfile);
#ifdef __linux___
    fsync(tmpdir);
    fsync(spooldir);
#endif
    /* transaction is complete */

Right; and I'm arguing that the way to get rid of the need for chattr +S is to incorporate directory fsync() in the MTAs, not to cram more features into the filesystems.

Problem: MTA needs to know when rename() has been forced to disk.

Solution 1: MTA authors use fsync(dirfd) on Linux.

Analysis: This is not the most portable solution, but it should work on any FS that supports Linux semantics. You can't expect such semantics on FAT and other filesystems that are just supported for compatibility reasons. But you could, say, switch filesystems for performance reasons, and not have your MTA start mysteriously failing, because you are using the official, documented API to do what you want to do (at the very least you would be in a much stronger position when pushing a bug fix :-).

Solution 2: Linux semantics are changed so that rename() returns only when the data hits the disk. All filesystems are expected to implement this change.

Analysis: This sucks. It precludes some filesystem design choices, prevents users from making a speed/reliability tradeoff, and makes each filesystem more complex.

Solution 3: Some filesystems implement synchronous directory updates for renames, using filesystem-specific feature flags, chattr, etc.

Analysis: I wouldn't want to try to dictate anything to the FS authors, but this solution seems inferior to me. Each filesystem would have to implement such a flag to become "MTA compatible". Why add a complex feature to the filesystem when it can already be accessed via a userspace API? It will be more complex for administrators too -- they will have to know which filesystems implement the synchronous directory metadata.

There are lots of filesystems out there. Why not use an interface they should all support rather than ask for per-filesystem, filesystem-specific improvements?

miket

First off, would you care to quote chapter and verse of these
"defined semantics" ? Do you mean the BSD source?

Traditional FFS/UFS achieves "safety" at a terrible cost to performance. I can barely stand the wait to untar XFree86 on Solaris8 on a PII-333, even with UFS logging -- I'd rather use my Pentium 166 laptop running Linux! ext2 solved this performance issue many years ago by recognizing that the FFS metadata scheme was not really safe either; instead the intelligence was put into e2fsck, and where necessary, the applications. (Do I hear faint echoes of the "lint" v. "cc" design criterion ... ?)

The infrastructure is now in place to solve these problems in ext3, without imposing a least-common-denominator approach that degrades overall system performance. In these instances "Linus is right" when he notes that (1) the proposed immediate solution does not really solve the problem, and (2) once in there, developers will rely on its precise semantics, making them difficult to get right later on, and providing no incentive to do so. In many such instances "undefined" behavior is the best intermediate solution.

As one can see from the "gkernel-commit" traffic, Andrew Morton has not only taken away useful information from this thread, he's already halfway to a solution, in just a day, because Matthias Andree took the time to describe the functional requirements instead of just whining that "it's not like BSD."

And that's precisely the type of thing we want -- unused features should not impact the rest of the system.

Regards,

Bill Rugolsky

Well, some systems are even documented that way, so there's nothing with "depend on the kernel being stupidly designed", but "depend on what mount(8) says".

MTA authors don't play games, they also write that their software relies on this behaviour, as laid out.

Matthias Andree

The key word here is "some systems".

"MTA authors don't play games" ?!?!

I wonder how that explains things like QMQP or the next-to-useless bounce messages generated by Notes ;)

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

Documenting their code won't make it fast or well designed.

Hans

Hi,

Not quite: dirsync would provide synchronous metadata operations on directories, but would make no guarantees about other file types. That way we don't have the cost of doing sync updates to the inodes or indirect blocks of regular files --- fsync() is adequate to do that on demand.

Of course, fsync() is also sufficient to do syncing of directory operations on demand, but it's a bit heavyweight for that purpose, hence the request for dirsync (either as a chattr flag or as a mount option.)

Almost --- it's the same for create+write+close+fsync, but not for rename or for unlink (in which case there's not necessarily going to be a data fsync to force the directory operation out to disk.)

Cheers,
Stephen

Sorry.

You're missing the point, with this as the previous mail. The MTA is not going to change from one unsupported/incompatible interface (that only Linux suffers from) and if it did, it would still do the wrong thing.

MTAs often run multiple processes, and if these all open the same directory and sync it while others have changes open that don't need a sync at that time and will sync later, you're getting no further than with chattr +S or mount -o sync.

It's not about atomicity itself, but about
first. write. all. required. blocks. for. a. certain. change.
physically. to. disc. and. only. after. this. do. return. from.
rename, link, unlink. function. calls.

I'm aware of phase-tree concepts ("single block write switches from one consistent state to another") and I'm aware that ext3fs and reiserfs do feature atomic renames (after crash recovery).

That a drive might fall over or the power might fail before all writes of a certain rename operation have completed is harmless UNLESS you lied to someone that the operation was already complete (when it wasn't).

You misunderstood me. I'm not talking about atomicity.

Matthias Andree

No. Atomic update systems and logs can by no means fix premature acknowledgements:

Proof:

Assume the OS has a phase tree kind of thing or log that requires just a single-block write for an atomic rename.

Assume an MTA calls rename(), and the OS by whatever means notifies it of completion, but actually, the data is only queued, not written.

Assume The MTA receives the acknowledgement (e. g. rename call returned), sends a "250 mail action complete" packet across the network.

Assume the machine sends the network packed, but not the queued disk block and then crashes.

--> The single block is lost, the rename operation is lost, but the operation had been acknowledged. Consequence: the mail is lost. q. e. d.

All this boils down to:

1. The OS _MUST_ know when a write operation has been physically committed to non-volatile storage.

2. The OS _MUST_ _NOT_ acknowledge the (assumedly synchronous operation) any earlier. (This may well include switching off drive write buffering.)

If the OS cannot fulfill these two basic requirements, I can save all the log or FS atomicity efforts because they don't get me anywhere.

The problem is not that the operation can fail, the problem IS premature acknowledgement. Even with atomic updates, as shown above.

Note, of course there is no premature acknowledgement for the Linux-default asynchronous directory update. There IS for -o sync or chattr +S -- and that's what MTAs to to guarantee data integrity, and that's why I'm still suggesting dirsync or something to remedy the negative data write performance of full-sync.

If the OS tell me "write completed" when it means "I queued your data for writing", it is BROKEN.

That's my point.

And since the common POSIX OS lacks a dedicated notification feature for e. g. rename, MTAs have no other choice than to rely on "has completed when the syscall returns".

BTW, my Linux rename(2) man page doesn't document EIO condition, FreeBSD 4.3-STABLE and SUS v2 do.

Matthias Andree

We're working on that, see the "[PATCH] 64 bit scsi read/write" thread on linux-fsdevel. About half of it is devoted to investigating the detailed semantics of physical write completion.

Yes, for now that's how you have to do it.

Right now the interface for determining that the operation has actually completed is "sync". Yes, that sucks but with journalling or atomic commit it's not nearly as expensive as you might think. My early flush patch does nearly the equivalent of sync, 10 times a second and it actually improves performance (it does not attempt to do this under high load of course).

We *should* have something like sys_sync_dev(majorminor) or sys_sync_fs(mountpoint) (whatever that would look like). For phase-tree the semantics are that the call doesn't return until the metaroot of the then-current "branching" tree is known to be safely on disk. (Side note: it's ok to allow subsequent updates on the same filesystem to procede while an outstanding sync_dev is waiting for confirmation from the block layer, because these don't affect the filesystem state the sync_fs is waiting on.)

As I understand it, Ext2 allows much the same semantics. While we do need to do something about exposing a more elegant interface, with Ext3 you should be ok with +S and a "sync" just before you report to the world that the mail transaction is complete. Ext3 does *not* leave a lot of dirty blocks hanging around in normal operation, so sync is not nearly as slow as it is with good old Ext2.

Sounds like a man page bug.

--
Daniel

eek.

In fully-journalled data mode, we write everything to the journal in a linear chunk, wait on it, write a commit block, wait on that and then release all the just-journalled data into the main filesystem for conventional bdflush/kupdate writeback in twenty seconds time.

Calling anything which uses fsync_dev() would cause all that writeback data to be written out and waited on, with the consequential seeking storm. Disastrous.

Note that fsync() is OK - in full data journalling mode nothing is ever attached to i_dirty_buffers.

-

That wasn't my impression, particularly not with data=journalling which can drop data into the log. It's just: why sync the world if synching directories does the job and relevant data is synched manually with fsync()?

However, how big are chances that these interfaces will spread outside of Linux? That's the crucial point for portable applications. If it's a kernel <-> libc interface, OK, no problem, but if it's a user-space interface, it might easily become a useless invention because no-one uses it in real life. You don't support multiple interfaces in a portable application because that's a maintenance disaster and often causes reliability problems because on different platforms, code takes different paths, so applications won't usually choose limited-use interfaces (such as sendfile).

BTW, your Message-ID is unqualified == on a collision course in mail duplicate killers.

Whoops, ok, no, this is not particularly sync-friendly. On the other hand, I don't think your seek storm would be as bad as all that. You can still feed enough blocks to the elevator to give it something to chew on. On the third hand, since you are still using the generic flushing machinery I can see you'd have quite a lot of work to do to control the flushing accurately in this way.

Somewhere in there is a beautiful optimization trying to get out...

--
Daniel

On Thu, Jul 26, 2001 at 09:42:37PM +1000, Andrew Morton wrote:

If postfix is capable of opening the files O_SYNC or of doing fsync() on them then the `chattr +s' is no longer necessary - unlike ext2, when the O_SYNC write() or the fsync() return, the directory contents (as well as the inode, bitmaps, data, etc) will all be tight on disk and will be restored after a crash.

This should speed things up considerably, especially with journalled-data mode. I need to test and characterise this some more to come up with some quantitative results and configuration recommendations.

Postfix does an fsync on file before closing them, it then does a rename and expects once rename as returned, the renamed actually occured --- even if the fs crashes. It also expects if you fsync a file, then it will appear in the parent directory with certainty and not say /lost+found after fsck on reboot.

Without +s under ext2, you can loose file(s) in /lost+found because open+write+fsync+close works and ensures the data is on disk, but the parent directory doesn't get synced to disk, so it might get lost.

--cw

OK, I've taken a closer look at this. ext3 has picked up some cruft from ext2's sync handling which it does not need in the least.

It will be fairly straightforward and a useful cleanup to provide the following semantics for either synchronous mounts or `chattr +S' directories:

* All metadata operations (rename, unlink, link, symlink, etc) will be synchronous. So when the system call returns, the data is crash-proofed.

* All write()s will be synchronous. So when the write() system call returns, the data written and all associated metadata will be crash-proofed.

O_SYNC and fsync() will not be necessary - in fact they'll slow things down slightly by forcing an unnecessary and probably empty commit.

If you crash in the middle of a write, you may end up with a truncated file on recovery.

This is in fact the behaviour right now, but the performance is not good.

The performance problem at present is that large write()s have unnecessary commits in the middle of them. This is due to the abovementioned cruft in ext3_get_block() and the things it calls.

We'd prefer not - we're trying to stabilise things quite sternly at present. However that doesn't prevent work on 1.1.0 :)

Seems like a worthwhile thing to do - I'll cut a branch and do this. It'll take a couple of weeks - as usual, most of the work is in development and use of test tools... But I can't predict at this time when we'll merge it into the mainline fs.

Yes, it was useful - thanks.

-

So the summary of all this is, as I understand it: for ext3fs 1.0, treat it with chattr +S and the like as if it was ext2fs, it may or may not be faster with "mount -o data=journalled" and is well worthwhile for an MTA to try, a weaker sync option may be introduced after ext3fs 1.0.

Sounds good.

I'm dropping the ext3-users mailing list for now since this is getting more general.

However, since the ReiserFS team also showed interest in a similar functionality, and they don't yet support chattr, would it be useful to specify a "D" option for chattr already?

I have a suggestion: if D is set, but S isn't, no effect. If S is set, but D is unset, treat S as in the past. If S is set, and D is set, directory updates are synchronous like with S, but data updates are asynchronous in spite of S.

This way, booting a kernel without chattr "D" flag support or mounting the file system as ext2 would have it default to the safer everything-synchronously mode.

Matthias Andree

chattr is an ext[23]-specific thing. reiserfs could certainly support a similar thing if they have a few bits spare in the inode.

I don't think this would be needed until really proven necessary - for data, fsync() should work for all filesystems.

There would be one benefit in splitting sync from datasync, and that is for applications which do not write() their data in large enough chunks.

When I fix the get_block thing, O_SYNC, chattr +S' and mount -o sync' will provide good, fast synchronous write()s - the fs will run a commit at the end of the write(). That's just fine as long as the app is writing its data in goodly chunks. If it is is using 4k or 8k chunks (eg: default stdio) then throughput will suffer. That would be rather silly of it though.

-

In article <[email protected]>,

Congratulations. You have been brainwashed by Dan Bernstein.

Use fsync() on the directory.

Logical, isn't it?

Linus

Following the announcement on lkml, I have started using ext3 on one of my servers. Since the server in question is a farily security-sensitive box, my /usr partition is mounted read only except when I remount rw to install packages.

I converted this partition to run ext3 with the mount options "nodev,ro,data=writeback,defaults" figuring that when I need to install new packages etc, that I could just mount rw as before and that metadata-only journalling would be ok for this partition as it really sees very little write activity.

When I try to remount it r/w I get a log message saying:

Jul 27 09:54:29 henry kernel: EXT3-fs: cannot change data mode on remount

...even if I give the full mount option list with the remount instruction.

I can, however, remount it as ext2 read-write, but when I try to remount as ext3 (even read only) I get the same problem.

Wierdly, "mount" lists it as being still an ext3 partition even though it has been remounted as ext2. I can't umount /usr because kjournald is currently listed as using the partition.

The box in question is more-or-less RedHat 7.1, with ext3-2.4-0.9.4, kernel 2.4.7 and with the following relevant package versions:

mount-2.11g-4
util-linux-2.11f-3
e2fsprogs-1.22-2

...all from rawhide rpms.

Sean

hmm.. The mount option handling there is a bit bogus.

What we *should* do on remount is check that the requested journalling mode is equal to the current mode. ext3 won't allow you to change the journalling mode on-the-fly.

So... you will have to omit the `data=xxx' portion of the mount options when remounting. It's being invisibly added by /bin/mount.

/bin/mount tries to be smart. If, for example you have

/dev/hdf12 /mnt/hdf12 ext3 noauto,ro,data=writeback 1

in /etc/fstab and then type

mount /dev/hdf12 -o remount,rw

then /bin/mount runs off and looks up the fstab entry and inserts the mount options. However if you instead type

mount /dev/hdf12 /mnt/hdf12 -o remount,rw (1)

then /bin/mount does *not* look up the fstab entry, and the remount succeeds.

ho-hum. For the while you'll have to fiddle with the mount
usage to get things working right. Equation (1) above will
work fine. Or apply the appended patch.

You can't switch between ext2 and ext3 with a remount - unmount is needed.

That sounds very weird. Could you please describe the steps you took to create this state?

Sometimes /etc/mtab gets out of sync - especially for the root fs. It's more reliable to look in /proc/mounts

Here's the fix for the data= handling on remount:

Index: fs/ext3/super.c
===================================================================
RCS file: /cvsroot/gkernel/ext3/fs/ext3/super.c,v retrieving revision 1.31

diff -u -r1.31 super.c
@@ -513,12 +513,6 @@
 
 			if (want_value(value, "data"))
 				return 0;
-			if (is_remount) {
-				printk ("EXT3-fs: cannot change data mode "
-						"on remount\n");
-				return 0;
-			}
-
 			if (!strcmp (value, "journal"))
 				data_opt = EXT3_MOUNT_JOURNAL_DATA;
 			else if (!strcmp (value, "ordered"))

@@ -529,9 +523,18 @@
 				printk ("EXT3-fs: Invalid data option: %s\n",
 					value);
 				return 0;
+			}
+			if (is_remount) {
+				if ((*mount_options & EXT3_MOUNT_DATA_FLAGS) !=
+							data_opt) {
+					printk("EXT3-fs: cannot change data "
+						"mode on remount\n");
+					return 0;
+				}
+			} else {
+				*mount_options &= ~EXT3_MOUNT_DATA_FLAGS;
+				*mount_options |= data_opt;
 			}
-			*mount_options &= ~EXT3_MOUNT_DATA_FLAGS;
-			*mount_options |= data_opt;
 		} else {
 			printk ("EXT3-fs: Unrecognized mount option %s\n",
 					this_char);

Indeed.

Thought so. I tried both ways just to be sure.

Interesting, and (almost) 100% true

sean@henry:~$  sudo mount /dev/sda8 /usr -oro,nodev,data=writeback,remount
mount: you must specify the filesystem type
sean@henry:~$ sudo mount /dev/sda8 /usr -oro,nodev,data=writeback,remount -text3
mount: /usr not mounted already, or bad option
sean@henry:~$ sudo mount /dev/sda8 /usr -oro,nodev,remount -text3
sean@henry:~$ mount
/dev/sdb6 on / type ext3 (rw)
none on /proc type proc (rw)
/dev/sda1 on /boot type ext2 (ro,nosuid,nodev)
/dev/sdc6 on /home type ext3 (rw,nosuid,nodev,data=ordered)
/dev/sda8 on /usr type ext3 (ro,nodev)
/dev/sda5 on /var type ext3 (rw,nosuid,nodev,sync,data=journal)
none on /dev/pts type devpts (rw,gid=5,mode=620)

It succeeds as long as I don't specify the journal type.

Wierd. This certainly looked to all the world as though it worked for me. Thus:

sean@henry:~$ sudo mount /dev/sda8 /usr -oro,nodev,remount -text2

...doesn't give me an error, but:

sean@henry:~$ mount 
/dev/sdb6 on / type ext3 (rw)
none on /proc type proc (rw)
/dev/sda1 on /boot type ext2 (ro,nosuid,nodev)
/dev/sdc6 on /home type ext3 (rw,nosuid,nodev,data=ordered)
/dev/sda8 on /usr type ext3 (ro,nodev)
                       ^^^^
/dev/sda5 on /var type ext3 (rw,nosuid,nodev,sync,data=journal)
none on /dev/pts type devpts (rw,gid=5,mode=620)

See above.

sean@henry:~$ cat /proc/mounts
/dev/root / ext3 rw 0 0
/proc /proc proc rw 0 0
/dev/sda1 /boot ext2 ro,nosuid,nodev 0 0
/dev/sdc6 /home ext3 rw,nosuid,nodev 0 0
/dev/sda8 /usr ext3 ro,nodev 0 0
/dev/sda5 /var ext3 rw,nosuid,nodev,sync 0 0
none /dev/pts devpts rw 0 0

sean@henry:~$ cat /etc/mtab   
/dev/sdb6 / ext3 rw 0 0
none /proc proc rw 0 0
/dev/sda1 /boot ext2 ro,nosuid,nodev 0 0
/dev/sdc6 /home ext3 rw,nosuid,nodev,data=ordered 0 0
/dev/sda8 /usr ext3 ro,nodev 0 0
/dev/sda5 /var ext3 rw,nosuid,nodev,sync,data=journal 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0

Its almost as if mount is just silently ignoring the "-t" option when I specify ext2.

I'll try this when its safe to reboot the box.

Thanks very much for your help.

Sean

Regardless of possible weirdness in a "smart" behaviour of 'mount' what one exactly buys running a journaling file system on a _read only_ partition? fsck times will be the same (unless you crashed when installing new software :-).

Michal

Several things:

1. The simple case of remounting an fs read-only is easy, since no
writes means no journal

2. The software suspend case is horrible. Right now mixing a
journalling fs and swsuspend tends to cause disk corruption because journalling fs's write to disk when told to mount read only

3. Failed drives. Here the journalling mount overrides the read only
request and the machine locks up preventing data recovery except by copying the whole 80Gb disk image to another disk

Been there, it sucks

4. Snapshots. Making read only snapshots can be very useful, and there
you want the replay of the log to be into the page cache but not written back to physical media until its marked read-write

Alan

On Fri, Jul 27, 2001 at 09:46:57PM +0100, Alan Cox wrote:

2. The software suspend case is horrible. Right now mixing a
journalling fs and swsuspend tends to cause disk corruption because journalling fs's write to disk when told to mount read only

this is hard to fix... the fs needs to replay things to make things consistent, and in many cases doing an 'in-memory' replay isn't an option (ie. remember which stuff needs to replayed and read from the journal instead of disk when required to do so)

4. Snapshots. Making read only snapshots can be very useful, and there
you want the replay of the log to be into the page cache but not written back to physical media until its marked read-write

R/O snapshots are best done in the fs if possible, al la WAFL. Something like that for resierfs or TUX2 would rule so much (you more-or-less need need a tree-based fs and reference counting for all the magic bits). In fact, doing it as the fs layer means you could have r/w snapshots with COW semantics.

--cw

You dont want r/w snapshots for archiving. An archive of a previous date is worthless if it can't be absolutely utterly and definitively read only.

It is hard to do well, but its an important item. One possiiblity is to do it by replaying the log to a r/w snapshot (in ram) over a r/o snapshot (on stable media)

On Fri, Jul 27, 2001 at 10:23:44PM +0100, Alan Cox wrote:

You dont want r/w snapshots for archiving. An archive of a previous date is worthless if it can't be absolutely utterly and definitively read only.

sure, for archiving you don't, but for other purposes you might

RO is easier and what most people want, this is all WAFL gives right now

RW has it's uses too, especially if you can clone /foo/bar to /foo/blem and such like, a cheaper more elegant way of cp -Rupdl I guess

It is hard to do well, but its an important item. One possiiblity is to do it by replaying the log to a r/w snapshot (in ram) over a r/o snapshot (on stable media)

you can probably get away without the need for replay... just build and in-memory extent list of blocks to would otherwise have been rewritten and the journal offsets, before you read a block, you check to see if you need to get from journal first

obviously you need to make sure you get the last insatce of each block in the journal should there be more than one

--cw

[email protected] (Alan Cox) wrote on 27.07.01 in <[email protected]>:

Not for archiving, but when you want to run something and then throw it away again, for example. You could do that by just holding onto a ro snapshot and then replacing the rw tree with it later, but by having two rw trees you don't need to stop your regular operations.

For this to really be useful, you'd want it as an inheritable per-process thing, similar to aviro's namespace thing.

MfG Kai

....

No. I don't see why I should opendir() a directory, fsync()
that handle, and closedir() the handle. I would definitely prefer:

lsync(dirpath)

This could, even, behave like lstat() with the path: if the last name segment is symlink, the sync is done on the i-node data of symlink, not on what it (possibly) points to.

I didn't check if POSIX folks have thought of that.

/Matti Aarnio

And it wouldn't even be enough. Who guarantees you that the parent directory of this directory has been written to disk and we won't lose the entry pointing to this directory on a crash ?

Nice idea. Of course, fsync(file) also has the obligation to make sure all the metadata of the file is written to disk. Lots of people seem to be convinced this also includes the metadata needed to _reach_ the file all the way from the root of the filesystem...

Nice addition. Easier to use than fsync() - no need to open the file - and probably easier to implement in the kernel because this way we'll be handing the whole path to the kernel, whereas fsync() would have the dubious task of finding out how this file can be traced all the way down from the root of the filesystem.

regards,

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

Date: Mon, 30 Jul 2001 22:23:29 -0300 (BRST)
From: Rik van Riel <[email protected]>
[...]

[...]
Nice addition. Easier to use than fsync() - no need to open the file - and probably easier to implement in the kernel because this way we'll be handing the whole path to the kernel, whereas fsync() would have the dubious task of finding out how this file can be traced all the way down from the root of the filesystem.

It's not as good as fsync() just doing what it's suppose to do. You'll force applications that want to issue multiple link()s to issue multiple lsync()s, forcing the kernel to serialize all of the disk writes when the application just wants one file (and all of it's associated filenames) to disk.

Yes, I understand that implementing fsync() so that it syncs all names to reach the file is difficult. But if you want the best performance, you don't want to make applications issue multiple calls each of which force their own synchronous writes.

Not to mention us whiny application writers won't be happy throwing lsync()s all over the place.

Larry

The thing about filesystems, and how dimmly MTAs (should) consider some performance tweaks is something I have tried to describe at ZMailer's manual in part about its the queue:

http://www.zmailer.org/zman/zadm-queues.html

On Tue, Jul 31, 2001 at 01:25:06AM -0400, Lawrence Greenfield wrote:
...

I quite agree.

Filesystems are not, unfortunately, rollbackfull logged and committable databases, even if we like to use them often in that way.

An MTA with a fundamental design point of not using any privileged programs (no suid anything!) and least esoteric technology possible (for wide portability) can only use message submission means available to it everywhere -- implementing the queue inside a database system
is definitely a possibility. Possibly yielding higher performance
than one using filesystem for it, but at what cost ??
(I am thinking of SleepyCat DB multiaccess transaction supported
version.)

/Matti Aarnio

Well it depends on which file system you are talking about. NTFS is for all intents and purposes a rollbackfull logged and committable (relational) database and a file system at the same time. It's a shame M$ don't release the specs for it, otherwise it would be just what you are looking for. - It will take us forever to reverse engineer the journalling part of NTFS. You can see how long it is taking us just to get the actual file system part.. and journalling on top of that is going to be even worse. (Of course once we have the file system part there is nothing to stop us doing our own thing with respect to journalling but that's a different discussion.)

Anton

Best regards,

Anton

Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Not portable -> won't happen usually.

Matthias Andree

Btw, you don't have to do opendir() - that just wastes time. Just do something like

	int lsync(char *path)
	{
	        int err, fd;
	        fd = open(path, 0);
	        if (fd >= 0) {
	                err = fsync(fd);
	                close(fd);
	        }
	        return err;
	}

and you're done. But it won't do the symlink thing...

Linus

If I understand SUS v2 correctly, fsync() must sync meta data corresponding to the file.

If Linux ext2 doesn't to that, it might be a good idea to change that so it does.

Matthias Andree

Where can I find a common definition for "meta data."

For example, I consider meta data to be things kept in the inode only (size, timestamps, permissions). Indirect blocks, maybe. But, considering how, in the unix world, file names are NOT associated with files, I have never considered file names to be meta data. Instead, file names is a set of data associated with special files known as "directories." So, it is obvious, to me, that expecting fsync to sync changes to directory entries is silly.

Obviously, however, you have a different definition of what meta data is.

Does SUS2 provide a definition for meta data?

A quick glance at the webside didn't turn anything up for me, but I would not be surprised that I may have missed it.

mrc

     Mike Castle      [email protected]      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

On Tue, Jul 31, 2001 at 11:29:47PM +0200, Matthias Andree wrote:

If I understand SUS v2 correctly, fsync() must sync meta data corresponding to the file.

If Linux ext2 doesn't to that, it might be a good idea to change that so it does.

Define 'meta-data' --- linux sync's any inode and/or bitmap changes, fsyn on a file will ensure it is intact but not that it can't get lost.

--cw

Syntactically correct, but quite useless IMHO ;)

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

No, I asked Wietse Venema what assumptions Postfix makes. Since he refuses to fsync() directories, he has Postfix set chattr +S to enforce the semantics he expects. No problem here.

Why go all the lengths to look up each single directory path component again just to fsync() stuff that doesn't belong to you and that you don't want synched, possibly the entire device?

Chase up to the root manually, because Linux' ext2 violates SUS v2 fsync() (which requires meta data synched BTW), as has been pointed out (and fixed in ReiserFS and ext3)?

Admittedly, MTAs are (supposed to be) (per command of RFC-1123) more paranoid than the average application - and per lack of standard whether rename/link & Co. need to be synchronous or asynchronous, this is a problem for the MTA.

So, please tell my why Single Unix Specification v2 specifies EIO for rename. Asynchronous I/O cannot possibly trigger immediate EIO.

Matthias Andree

[snip fsync() on directory ... on second thought this isn't enough]

Agreed. fsync() on the file needs to write the meta data, this includes the directory and (if needed) the parent directories all the way up to the root.

Crap. An asynchronous rename() can hit the situation where it cannot read the disk when searching for the directory it wants to move the file to.

rename(/from/a/b/file, /to/d/f/file) can fail when the system gets an IO access on reading "d".

regards,

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

It also specifies EIO as possible for write().

Are you saying that, since SUS2 specifies that write() is capable of returning EIO, and asynchronous I/O cannot possibly trigger immediate EIO, that all calls to write() should by synchronous?

mrc

     Mike Castle      [email protected]      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

No, I'm wondering about the semantics. Of course, write() can be synchronous (O_SYNC or fs mounted sync e. g.).

I'm using ext3-0.9.4 with linux-2.4.7 / 2.4.8-pre1 and get some hangs on my dual P2-350:

hangs when logging in. It even happens during boot before my MTA is started.

I have a single ext3 partition which is exported by kernel-nfs-server.

As soon as I do an Alt-SysRq-S forced sync the hang goes away and everything works normal.

If you need further information send me an eMail. SGIs kdb is already compiled in so if we need it ...

BYtE
Philipp

  / /  (_)__  __ ____  __ Philipp Hahn
 / /__/ / _ \/ // /\ \/ /
/____/_/_//_/\_,_/ /_/\_\ [email protected]

Hi,

Please quote chapter and verse --- my reading of SUS shows no such requirement.

fsync is required to force "all currently queued I/O operations associated with the file indicated by file descriptor fildes to the synchronised I/O completion state." But as you should know, directory entries and files are NOT the same thing in Unix/SUS.

Are we expected to fsync the metadata belonging to just the file itself? Or all symlinks to the file? Or all hard links? Answer, as best I can determine --- just the file. That's all SUS talks about. There can be many ways of reaching that file in the directory hierarchy, or there can be none, but fsync() doesn't talk at all about the status of those dirents after the sync.

ext3 happens to provide the guarantee, but that's coincidental and does not imply that I think of it as being "fixed". It's just changed behaviour relative to ext2.

Yes it can --- we may need to read metadata to complete the rename, and such reads can fail.

Cheers,
Stephen

But does fsync() have any meaning if it doesn't ensure the file is visible within the filesystem?

This all comes back to the fact that old UFS's made directory operations syncronous, at a substantial cost in performance. Writing the directory data wasn't necessary for them, because it was already commited when the creat() call returned.

I can easily understand people's asthetic objection to having fsync touch the directory object as well as the file, however what meaning does fsync() have it it doesn't - under linux, it tells usermode "yes, your object is committed, but it might be in lost+found next time you want it", and with the syncronous UFS implementations, it tells usermode "yes, your object is committed, you can find it where you left it (unless the directory was corrupted)".

---
Andrew McNamara (System Architect)

connect.com.au Pty Ltd
Lvl 3, 213 Miller St, North Sydney, NSW 2060, Australia
Phone: +61 2 9409 2117, Fax: +61 2 9409 2111

It goed on with "All I/O operations are completed as defined for synchronised I/O file integrity completion.", whatever it all means.

For fdatasync() it says:
"The fdatasync() function forces all currently queued I/O operations associated with the file indicated by file descriptor fildes to the synchronised I/O completion state.", which is just the same as it says for fsync().

It also says:
"The functionality is as described for fsync() (with the symbol _XOPEN_REALTIME defined), with the exception that all I/O operations are completed as defined for synchronised I/O data integrity completion."

It doesn't mention meta-data.

I have no idea what it all means.

Kurt

Read on: "All I/O operations are completed as defined for synchronised I/O _file_ integrity completion.". To show what that means, see the glossary.

http://www.opengroup.org/onlinepubs/007908799/xbd/glossary.html#tag_004_000_291

"synchronised I/O data integrity completion

[...]

* For write, when the operation has been completed or diagnosed if unsuccessful. The write is complete only when the data specified in the write request is successfully transferred and all file system information required to retrieve the data is successfully transferred.

File attributes that are not necessary for data retrieval (access time, modification time, status change time) need not be successfully transferred prior to returning to the calling process.

synchronised I/O file integrity completion

Identical to a synchronised I/O data integrity completion with the addition that all file attributes relative to the I/O operation (including access time, modification time, status change time) will be successfully transferred prior to returning to the calling process."

As I understand it, the directory entry's st_ino is a file attribute necessary for data retrieval and also contains the m/a/ctime, so it must be flushed to disk on fsync() as well.

Well, if there's not a single dirent, you cannot retrieve the data, so I'd assume at least one dirent needs to be flushed as well. If there's a simple way to get unflushed dentries to disk (hard links included), flush them. Not sure about symlinks, but since they don't share the inode number, that might be rather difficult for the kernel (I didn't check):

touch 1 ; ln 1 2 ; ln -s 1 3 ; ls -li

 303464 -rw-r--r--   2 emma     users           0 Aug  2 10:56 1
 303464 -rw-r--r--   2 emma     users           0 Aug  2 10:56 2
 303466 lrwxrwxrwx   1 emma     users           1 Aug  2 10:56 3 -> 1

Matthias Andree

NOTE: _file_ attributes

As long as you have an open fd, no directory entry is needed for data retrieval. In fact some fds never have a directory entry (e.g. sockets - but these don't support fsync anyway) or do not have a directory entry in their user-visble interface (e.g. posix shm).

And m/a/ctime is in the inode of the file, not in the directory enrty. (at least for usual UNIX filesystems).

Of course you can, you can pass and fd for an unliked file everywhere using AF_LOCAL descriptor passing.

Christoph

Of course it doesn't work. We've performed a software upgrade.

But this assumes the system doesn't crash, while
fsync() seems meant more as a protection against
the system going down unexpectedly ...

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

There is something to that. However taking this argument to it's logical extreme I have you have to not only sync every directory in the current path of the file. You have to sync your online file index, because search engines is how we find things right?

Since the filename in unix is not part of the files metadata it is a perfectly sane semantic for fsck to drop the file into /lost+found, if no one cared enough about the index/directory to update it.

In the general case if you have the guarantee that a filesystem does safe directory updates. So unless someone does an unlink you won't loose your old link. For most cases it doesn't matter as your directory entry for the file is much older than the file itself, and has been already synched. MTA's are the exception to this where there good filename is written only after the file is written.

The only other argument that seems to come from the MTA case is that syscalls are slow, and a pain and programmers don't want to make two or three syscalls just to do this. Heck if you are doing a sync you are waiting for a disk which is slow. So speed doesn't really count.

There is probably an argument in there somewhere about batching up I/O, so you have to wait a minimum amount of time for your sync. But until someone benchmarks, and tries a few different approatches I won't believe that you need a kernel change even for that.

My question is what does fsync do on directories in other unix's. It would be really strange if it didn't behave similiarly to linux. If forget wether it was AIX or HP-UX where doing a grep foo * would also grep through "." . So at least open works on other peoples directores.

Eric

Hi,

Interesting. Do you have the ability to hook up a serial console? If so, "alt-sysrq-T" to capture a backtrace of all the blocked processes would be a great help. Thanks.

Cheers,
Stephen

I believed you've summarized the SUS requirements very well. Apart from legalistic arguments, SUS quite clearly states that fsync should not return until you are sure of having recorded not only the file's data, but the access path to it. I interpret this as being able to "access the file by its name", and being able to guess by looking in lost+found doesn't count. I don't see the point in niggling about that.

So, it seems clear that an fsync which leaves any window of vulnerability where an interruption can leave a file unlinked is not SUS-compliant.

This is a legalistic argument. I don't think we should be looking for loopholes in SUS here. To achieve SUS compliance there are two reasonable courses: "fix SUS" or "fix sys_fsync". Since what SUS clearly wants here seems emminently reasonable, I'd suggest putting the energy that's currently going into this thread into fixing fsync instead.

*All* hard links? No, there is no general way to do that. However, any hard links[1] in the path used to open the file - yes. There is always a chain of parent dentries held locked in the dcache for any open file.

I don't know why it is hard or inefficient to implement this at the VFS level, though I'm sure there is a reason or this thread wouldn't exist. Stephen, perhaps you could explain for the record why sys_fsync can't just walk the chain of dentry parent links doing fdatasync? Does this create VFS or Ext3 locking problems? Or maybe it repeats work that Ext3 is already supposed to have done?

The prescription for symlinks is, if you want them safely on disk you have to explicitly fsync the containing directory.

[1] In Ext2, all filename dirents are "hard links", i.e., there is no way to tell which of the two names is the original after creating a new hard link.

--
Daniel

Parent directory can be renamed. Which grandparent should we sync? New one? Old one? Both? BTW, how about file itself getting renamed during fsync()?

See the problem? And no, blocking all renames while fsync() happens is not an answer - it's a DoS.

s/Ext2/UNIX/.

Either one, or both, it doesn't matter since the application has not forced any serialization on this and can't assume any.

It doesn't matter. If the application wants to race that way, let it. We're talking about ensuring access to the fsynced fd's inode.

We would have done our duty by fsyncing the inodes one at a time working up the dentry chain towards the root, and not trying to lock the whole chain. If something happens while we're doing that it's an application race.

--
Daniel

On Thu, 02 Aug 2001, Daniel Phillips wrote:

[file name must be flushed on fsync()]

Well, the course was that I asked whether ext3 would do synchronous directory updates, and some people jumped in and said that one should fsync() the parent directory, however, since we figure from SUS, that's invalid.

After some forth and back, we finally figured that at least ext2 is implementing fsync() improperly.

So this part is covered.

The other thing is, that Linux is the only known system that does asynchronous rename/link/unlink/symlink -- people have claimed it might not be the only one, but failed to name systems.

So we need to assume that Linux is the only system that does asynchronous rename/link/unlink/symlink, however a directory fsync() is believed to be rather expensive.

Still, some people object to a dirsync mount option. But this has been the actual reason for the thread - MTA authors are refusing to pamper Linux and use chattr +S instead which gives unnecessary (premature) sync operations on write() - but MTAs know how to fsync().

Yes, and it doesn't matter, since MTAs don't use symlinks (symlinks waste inodes on most systems).

Matthias Andree

How the fuck it's expensive? It does _exactly_ the same as file fsync() - literally the same code. It doesn't write blocks that don't belong to directory. It doesn't write blocks that are clean. IOW, it does the minimal work possible.

fsync()ing the dir is not the minimal work possible, if e. g. temporary files are open that don't need their names synched. Fsync()ing the directory syncs also these temporary file NAMES that other processes may have open (but that they unlink rather than fsync()).

Assume:

open -> asynchronous, but filename synched on fsync() rename/link/unlink(/symlink) -> synchronous

This way, you never need to fsync() the directory, so you never sync() entries of temporary files. You never lose important files (because the application uses fsync() and the OS synchs rename/link etc.).

Matthias Andree

Do you read what you are writing? How can a "synchronous" operation for rename/link/unlink/symlink NOT also write out "temporary" files in the same directory? How does calling fsync() on the directory IF YOU REQUIRE SYNCHRONOUS DIRECTORY OPERATIONS differ from making the specific operations synchronous from within the kernel???

The only difference I can see is that making these specific operations ALWAYS be synchronous hurts the common case when they can be async (see Solaris UFS vs. Linux benchmark elsewhere in this thread), while requiring an fsync() on the directory == only synchronous operation when it is actually needed, and no "extra" performance hit.

The only slight point of contention is if you have very large directories which span several filesystem blocks, in which case it _would_ be possible to write out some blocks synchronously, while leaving other blocks dirty. In practise however, you will either only be modifying a small number of blocks (at the end of the directory) because an MTA usually only creates files and doesn't delete them, and the actual speed of syncing several blocks at one time is not noticably different than syncing only one.

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Only dirty blocks are written back to disk, i.e. the parts of the directory that were modified by adding/removing names. It should be pretty efficient.

"chattr +S" is about as portable as adding fsync(parent), or even worse as it only works on an ext2 file system. So I'm assuming that this is just a nice exercise in annoying people.

Jan

Can people please try to understand? Can people please start to THINK before flaming?

I did not say that open() is to be synchronous. I did not write ANYTHING of fsync()ing directories, I'm trying to get rid of this requirement.

Thus, if the kernel does rename/link synchronously, you'd never ever fsync() a directory. To synch a filename to disk, you'd just fsync() the filedescriptor (with a SUS compliant system, that is, i. e. ext3 or reiserfs, but not ext2).

Now, if someone opens a temporary file, and nukes it later -- unlink() --, and doesn't want it visible, he never calls fsync() for the file.

However, if some other process then fsync()s the directory, you start synching the temporary file dirent -> unnecessary, is nuked later on with an unlink().

That's why fsync() on the directory is on no account the minimum work.

In case you haven't noticed, this is about reliability without need to fsync() the directory that doesn't all belong to your single, stupid process but may have lots of asynchronous data of other processes - temporary files for instance. You synch() that as well, which is unnecessary and brings down other processes' performance.

In case you haven't noticed the other issue:

The whole thread is a FEATURE REQUEST for a dirsync mount option, for MTAs and other software which requires reliable file systems, where the name is negotiable. It aims to REDUCE OVERHEAD since chattr +S which is the only workaround for synch-dirs - and it synchs synchronous files and writes as well, and rendering things slower than necessary, since write() can be buffered until you fsync() (and you want that to cut off seek times).

Call the option bsd_slow_dirs if you like, I don't care. Given the option, the administrator/user has the choice, currently, he hasn't. He cannot possibly change all applications ported from other Unices.

Note: hindering this option doesn't get Linux anywhere. Pure file system benchmarks are not worth a single bit of entropy unless Linux is benchmarked chattr +S -- it's unreliable otherwise.

I cannot remember how often I explained this during the course of this thread. Every other day, some ignorant comes out of its cavern and discusses the whole thing over and over again.

And, once again, fsync()ing the directory is not an option for portable applications. It's unnecessary on every other system (until someone shows a production-ready system which by default has asynchronous directory updates as well, but no-one has so far.)

Actually given that this thread keeps coming up, but no one does anything about it. I'm tempted to suggest we remove chatrr +S support from ext2. Then there will be enough pain that someone will fix the MTA instead of moaning that kernel is slow...

That should be an easy patch to make...

Eric

They'd just drop Linux from the list of supported OS's, Linux will disappoint people who trusted it, nothing is gained. Deliberate breakage will not happen, because it would not help anyone except people with twisted minds.

NO-ONE, including you, has come up with SERIOUS objections against a dirsync option, except "is it really so much slower than chattr +S? show figures" -- ext3 is being tuned to be fast in spite of chattr +S.

Reconsider your position.

Stop trolling please.

Matthias Andree

Who cares? There are more than enough sane mailer around..

Talk is cheap. Code up a non-invasive dirsync option and submit it to Linus. I don't see any reason why it won't be accepted..

Christoph

Of course it doesn't work. We've performed a software upgrade.

There are some other uses for a fully synchronous disk accesses so I'm not going to run out and do it. The point is that work arounds for strange programs is not a right, it is a nice optional feature.

Clear objects against dirsync.
- Extra code maintenance, makes the fs less reliable
(A reason for removing even synchrouns fs operations BTW).
- Unnecessary. fsync(dir) works today.
- dirsync is unlikely to be faster than fsync(file); fsync(dir) [not chattr +S] You really need something that can say remember these 5 syscalls, and sync the all their changes to disk togeter to really get an improvement in sync speed.
- I don't see anyone volunteering to write the code.

Nope. Right now I would rather
a) Patch the mail programs to do the needed fsync(dir)
b) Totally remove synchrous disk updates from my OS, and make life really painful.
Before adding a dirsync option.

It wasn't trying to troll, just get this conversation on some productive grounds. I think supporting the MTA's is good, so long as it is a two way relationship.

If someone went out and tried using fsync(dir) and then saying it sucked we could definentily have more peace.

Using dirsync, and chattr +S hide the real problems that need to get fixed. Getting a good reliable, and high performance way to commit actions to a filesystem.

We already have one work around on linux that will work reliably. So now let's see if we can get a functional high performance solution.

And oh btw, new, functional high performance solutions are not portable because they haven't been implemented in every operating system. Full understanding of the problem, and the solutions are two new for the implementations to have gotten around.

Eric

Let's inject a little reality into this discussion. Filesystems are used for something other than running MTA's written by stubborn "purists".

Solaris: Dell 600 MHz PIII 128MB RAM, largely quiescent:
Solaris 8 mu4, UFS with logging

Linux: VA Linux 800 MHZ PIII, 128MB RAM, largely quiescent

         RedHat Linux 7.1 w/ kernel-2.4.6-2.4 (2.4.6-ac5 + ext3-0.9.3).

660MB XFree86-4.1 build tree, cache primed with du -s in each case.

Here's something that we developers probably all do frequently: copy a tree using hard links, so that we can patch it.

[solaris] find . | wc
33027 33027 1251671
[solaris] time find . -depth | cpio -pdul ../foo
0 blocks
363.46s real 0.84s user 10.13s system

Plain ext2:

[linux]# time find . -depth | cpio -pdul ../foo
0 blocks

real 0m3.823s user 0m0.240s sys 0m3.570s

Mounted ext3, ordered data mode.

[linux] time find . -depth | cpio -pdul ../foo
0 blocks

real 0m5.106s user 0m0.200s sys 0m3.700s

Mounted ext3, -o sync:

[root@ead51 bar]# time find . -depth | cpio -pdul ../foo 0 blocks

real 1m28.483s user 0m0.470s sys 0m4.410s

=====================================================

Solaris8 UFS:   363.5 seconds
ext2:             3.8 seconds
ext3:             5.1 seconds
ext3 -o sync:    88.5 seconds

Got it?

Obviously, the last is the result of the poor interaction of ext3+sync in 0.9.3, but Andrew Morton has already fixed that. I will try again with 0.9.5 when I have a chance to upgrade that machine.

I have no idea where BSD falls, but the basic point stands: unused features should not penalize other applications. Andrew Morton has figured out how to do this efficiently with ext3, and many kudos to him for doing the work. Absent that, why should I have to go get a cup of coffee every time I want to patch a tree, just so some MTA can make naive assumptions?

Regards,

Bill Rugolsky

Hi,

Not true. There are tons of others.

The issue was that synchronous directory updates are *optional* on many systems (Linux included), but that Linux's support for that is really inefficient since it ends up syncing file metadata updates too (and it's much more efficient to use fsync for that.)

Who? People who have discussed this in the past have certainly not objected to my knowledge. It would clearly help situations like this (as would a dirsync chattr option.)

Irrelevant. We're talking about what makes sensible semantics, not
what assumptions any specific application makes. It makes no sense to say that dirsync won't affect symlinks just because some existing applications don't rely on that!

Cheers,
Stephen

Yup, and this should be fixed imho.

And this is a feature, not a bug.

A directory fsync() might be expensive on non-Linux filesystems...

So what you mean is that MTA authors refuse to pamper Linux through use of fsync of the directory, but can accept to "pamper" Linux through use of chattr +S?! This seem ridiculous. It seems equally ridiculous to demand that Linux should pamper for MTA authors that can't implement fsync on the directory instead of writing BSD-specific code.

[snip]

To me this seems mostly like a way of saying "Hey, we've finally found a way to make Linux look really bad compared to BSD-systems; let's complain instead of writing alternative code that suits Linux systems better than this code does." A lot like all the discussions on threads, ueally.

Then again, I'm probably just extra grouchy today because it rained when I rode my bike to work.

/David Weinehall
_ _
// David Weinehall <[email protected]> /> Northern lights wander \\

//  Project MCA Linux hacker        //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

This is just completely true. One wonders why we seem to enjoy getting screwed this way. We shouldn't be patching these MTAs or hacking Linux to act like BSD. We should be avoiding these MTAs.

Somebody can create a big MTA list, listing the good and bad ones. Then we get the Linux-hostile MTAs out of the Linux distributions, demanding compliance like we do for filesystem layout. We also hunt down Linux-related web pages that mention these MTAs and get the pages changed or removed. The point is to make these MTAs just disappear, never to be seen again. Nice MTAs get promoted.

On Fri, Aug 03, 2001 at 03:59:02PM -0400, Albert D. Cahalan wrote:
[snip]

Think we could just get their authors to 'disappear'? It might be more cost effective, and I can think of at least one example where removing the author would have other benefits beyond MTAs. :) :)

Oh, you should make a start avoiding any MTAs because that way, this list would get rid of one trouble maker after all.

Don't feed the trolls.

That wasn't intended to be a troll, though I do realize that it could cause some noise -- including your post. Plenty of noise is already being generated trying to accommodate hostile MTA authors.

Seriously, consider:

1. there are MTA authors that actively promote BSD over Linux
2. Linux users and distributions promote their MTA software

There is no sense in this. It is masochism and suicide. It is worse than a waste of time to accommodate these MTAs.

Getting back on topic... while non-inherited ext2 attributes might be nice, I'm sure the ext2/VFS authors don't need to be pestered about it, and certainly not because of some lame software making non-standard assumptions about filesystem behavior.

I do not endorse this behaviour (particularly, qmail not supporting softupdates is rather ridiculous), but I understand that MTA authors would rather want to rely on fsync() also bringing related meta data do disk (as ext3 and reiserfs for Linux 2.4 already do even across a rename()!) than to add dir=open("directory"); fsync(dir); close(dir) all over the place.

What would they be good for? Make MTA that have in the past achieved reliable behaviour with chattr +S unreliable?

Well, the software documents its requirements and assumptions. I don't see anything nonstandard with relying on fsync(). If ext2fs doesn't meet the assumptions without chattr +S or mount -o sync, but allows to enforce this behaviour chattr +S, deliberately breaking ext2 attributes inheritance will make Linux deliberately unsuitable for this MTA -- or at least, slow it down through the need to use mount -o sync.

Deliberately breaking things just to show somebody else "you cannot even rely that chattr behaviour is invariant" is ridiculous and definitely not the right way to go.

If the MTA author chooses chattr +S over fsync-directory, what's wrong with that?

Matthias Andree

Hi,

Umm, this is a specification. It is *supposed* to be interpreted legalistically!

But that is just an interpretation. There's nothing in the spec which forces that interpretation.

fsync forces the data to disk. There may be one or more pathnames which the application also relies on, and if the application does care about those, the application will have to ensure that they are synced too.

Look, I agree that your interpretation is useful. It's just not an unambiguous requirement of the spec.

Cheers,
Stephen

I'm saying that, when the intent is clear as it is in this case then trying to find loopholes in the form of expression is not useful. Better to argue that SUS is wrong than to pretend it didn't say what it did.

Well, look at the name "lost+found". It's lost, maybe we found the data but the name is gone for good. On the other hand, if we start with the same on-disk state that fsck had before creating the lost+found, but use a journal to recover the name, it *does* count because we have a deterministic mechanism for making fsync's promise come true.

OK, fine, this damn English language and all that. Do we agree that "access" means "be able to open by name"?

--
Daniel

If we didn't agree on this, Linux would have had an open_by_inode() system call long ago.

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

The intent is perfectly clear regarding the data. It is totally undefined regarding names.

That's fine --- it satisfies the SUS requirements.

That's an implementation decision, not a comment on the spec.

The word "access" isn't used there in the spec, so it doesn't matter. The spec only refers to "all file system information required to retrieve the data." Integrity of the data is the only thing guaranteed, not integrity of the namespace.

Cheers,
Stephen

In fact this interpretation would have a severe performance impact for any implementation.

If you include "metadata of the 'path'" in "all filesystem information..." then you have to basically sync each and every element on the path(s) to that file. This means walking each dentry in the alias list for the inode, and then walk from each of those to the root sync'ing along the way.

That would be a rediculious requirement.

Later,
David S. Miller
[email protected]

No, until recently the device/inode number pair used to work very nicely for both Coda and knfsd when they wanted to access a file. But it only works from within the kernel where you can use iget. It's just that with some of the newer filesystems the inode numbers are no longer unique, so it became something more like device/inum/opaque handle (i.e. iget4).

As far as the fsync semantics are concerned. Yeah, they would be useful, but only to avoid one directory fsync call that will tell the kernel _exactly_ what the process wants instead of letting it figure it out by itself. The argument I saw in this thread that fsync(dir) has too much overhead because it might sync unrelated data is not very useful, because that unrelated data will be synced anyways when it's not a journalling fs.

Name to file object mapping is not part of the metadata associated with a file. It is the contents of the directory and can only be modified by directory operations, not operations on the file or filehandle. open(file, O_CREAT) is really split up into create(parent, file) an operation on the parent directory, and an open(file) operation which returns the filehandle.

I also don't see why a rename operation, which operates on the source and destination parent directories would have to not only look up the file object but also somehow register with all open filehandles for that object that both olddir and newdir need to be written back to disk during the fsync as well. Or should that go the other way around, where the filehandle has to chase down which directory it was renamed to?

Taken from another part of this thread Alexander and Daniel,

Using the dentry chain is not reliable, for instance instead of moving dentries around Coda simply unhashes dentries when state on the server changes. Another process (perhaps on a different machine) might have moved one of the ancestor directories from one location to another, or even relinked/unlinked the file that we're working with (ln old new; rm old) in which case the dentry path is lost, but ideally you'd expect fsync to sync the 'new' name if it supposedly keeps the namespace consistent.

Working on a distributed filesystem with somewhat weaker than UNIX semantics might have skewed my vision. In Coda not every client will be able to figure out which are all of the possibly paths that can lead to a file object. And although we currently try our best to block hardlinked directories they could possibly exist, making the problems even worse.

Jan

As I read the (excerpts from the) SUS, this isn't required, only that at least one namespace path from the root to the fsynced file is preserved. I can imagine an efficient implementation for this.

--
Daniel

We are talking about "after waking up from an unexpected interruption". So, is this still relevant?

No, it's not just that. It's not enough to fsync just the parent, the parent's parent may have been relinked and not comitted. Also, the kernel has the advantage of the locked chain of dcache entries available to it. Finally, there is the saving of multiple syscall overhead, which I didn't mention first time around because it's not a lot compared to media access overhead.

Yes, and I gave a detailed explanation earlier of why such overhead will not amount to much.

SUS doesn't just pronounce on the file metadata. Quoting from earlier in the thread:

----------

----------

They don't both have to, either one will be good enough. However, "neither" is not good enough, according to SUS.

Could you be more specific about this, are you saying there are cases where there is no valid parent link from a dcache entry?

I doesn't matter which one it syncs.

We don't need all the paths, and not any specific path, just a path.

--
Daniel

That isn't the only cruft there from the "Good Old Days":

"Battery backups will keep your server alive, letting you park the disk to avoid a head crash, when the power goes out."

What the hell? Self-parking drives predate qmail by quite a long time.

Thanks for clarifying, I'm asking Wietse to figure if Postfix's queue file format is sufficient to check integrity.

To add to that confusion and alternatvies:
HAVE: async, sync
SUGGEST: 1. BSD dirsync, 2. "weak" unlink/symlink dirsync and have fsync() track and sync pending link/rename as well, 3. make just symlink dirsync, 4. be confused of all the options.

Where this could be set: directory inode flag, mount option, process flag (like umask), include file.

Seriously, if fsync() syncs the effects of link and rename as well, there's no need to make them synchronous unconditionally except if one wants to offer a "traditional BSD synchronous directory semantics" mode.

Matthias Andree

Are the Linux "fsync() the directory" semantics documented anywhere? I mean other than in the source code and on mailing lists.

It might be easier to convince MTA authors to support these semantics if there were an "official" document describing them and giving some guarantee that Linux will contine to support them in the future. It would be nice to see this described in linux/Documentation or in the fsync() man page or both. Without this, it is hard for a software author to know that Linux's behavior here is not just an implementation artifact.

- Pat

P.S. Is fdatasync() on a directory guaranteed to do anything? Just curious.

Hi,

Not necessarily. If the file has been hard-linked and then the original link removed, we don't have a valid dcache entry any more (and yes, this is a common construct for some applications --- it is often used to work around NFS atomicity problems.)

An MTA using such a construct and expecting fsync to do the right thing will *not* work if you follow the dcache chain on fsync as you propose here.

Exactly, because fsync makes absolutely no gaurantees about the namespace. So a lost+found path is quite sufficient.

Cheers,
Stephen

The NFS client hasn't been interrupted, and it's filehandle will still identifies the file object by device/inode.

So that would be the file data, and it's on-disk inode information, indirect blocks etc. All the information that the file system needs to retrieve the data is then available, i.e. what is required for iget() to succeed.

Ok, iget isn't exported to userspace, but fsck will place the file in a user reachable location.

Ehh, sync only olddir and you just lost any path leading to the file. Sync only newdir and the file is reachable from two locations, but it's linkcount is too low.

No the dcache entry could have a 'stale' fileobject associated with it that has been superceded by a different object. This dentry is unhashed, so that the next lookup will instantiate a new dentry which references the new object. So syncing the stale object is useless, because it doesn't really exist anymore, but the kernel (and actually the userspace daemon on the client) doesn't know what the new object is until it is accessed.

Even if that path leads to a name that got removed, thereby forcing the object into lost+found? I thought the MTA did something like,

fd = open(tmp/file)
write(fd)
fsync(fd)
link(tmp/file, new/file)
fsync(fd) *1
unlink(tmp/file)

*1 If this fsync only syncs the path leading to tmp/file, and the unlink tmp/file is written back to disk, which is likely because we're only creating/syncing stuff in tmp. Now, until new/file is written there is no path information leading to the file anymore which makes this as 'safe' as not syncing path name information at all.

Now if the application would use the directory sync, it can actually tell the kernel that that new/file name is the interesting one to keep and that tmp doesn't even need to be written to disk at all.

Jan

But in that case, the file was opened using the hard link, then the link was deleted. Fine. The user is trying to tell us it's ok to lose the linked file. Whether or not it can be accessed through another path is irrelevant.

OK, this case where the walk to the root should fail is a "duh", and exposes a corner case SUS didn't cover (at least not in the excerpts I saw). But this case is a userland race, the right thing to do is just stop the walk. This doesn't detract from the value of doing the walk in the important case that the chain is intact.

Dunno, I think that's a statement that should be held up for further scrutiny.

--
Daniel

Interesting, but at best relevant only to network mounts.

In spite of your interesting network filesystem example I'm not willing to accept that access by inode is enough. It's going pretty far out on a limb, don't you agree? I doubt that SUS explicitly allows the inum to be interpreted as "information that the file system needs to retrieve the data".

Hmmph. It used to have a name, now it doesn't, and somebody did an fsync on the file, which returned indicating success. Do you think that's right?

You sync the one the dcache entry points at.

SUS didn't say the filesystem integrity had to be perfect, just that "all file system information required to retrieve the data is successfully transferred".

This happens because the dentry was unlinked, but not when it is renamed, right? In that case it's ok to fail the walk IMHO, the user explicitly left a window when it's ok to interpret the fsynced file as "gone".

We'd better get confirmation from the MTA expert in the thread.

Nice clear example! Yes, in essence we would have synced the original path twice. If this is what the MTA is really doing I'm willing to join the "MTA is broken" camp.

Yep. Or if it used rename, which updates the dcache entries, instead of link/unlink.

--
Daniel

I might be way off base here; if so tell me.
Let's litter some sample code with fsync():

	fd = open("tmp/x",O_CREAT|O_WRONLY);
	...
	fsync(fd);
	rename("tmp/x","spool/x");
	fsync(fd);
	close(fd);

This looks safe and correct, given your semantics. It is, unless the link count of that file rises above 1, from say a mail admin quite reasonably doing

ln tmp/x peek/x

in the interval specified by "...". In that case, it's not required that "tmp/x" or "spool/x" ever hit disk. So the code is only correct if the file only has a single link, or we specify ordered meta-data updates for open/rename/link/... Following Stephen's argument, is "peek/x" any better than "lost+found"? With more than one admin?

On older non-BSD systems (SYS3?,SYSV 3.x?) that don't do rename(), files can't be moved without bumping their link counts:

	fd = open("tmp/x",O_CREAT|O_WRONLY);
	...
	fsync(fd);
	link("tmp/x","spool/x");
	fsync(fd);  /* <- possibly a NOP with your semantics */
	unlink("tmp/x");
	fsync(fd);
	close(fd);

Again, this will fail to preserve your desired semantics on crash, unless we have ordered meta-data updates, or the stronger synchronous link() requirement.

I think the semantics that you propose might be marginally useful, but I don't think SUS requires it; my understanding (and that of a close friend and former POSIX reviewer) has always been that inodes are distinct from directory entries, and that fsync() preserves the fields that stat() returns: mode, uid, gid, size, {a,c,m}time.

Regards,

Bill Rugolsky

In fact, make that

ln tmp/x peek/x
<- MTA does the fsync()
less peek/x
<- MTA closes the file.
rm peek/x
(peek gets written asynchronously to disk)
*CRASH*

Then we are back to "lost+found", nothing gained.

Regards,

Bill Rugolsky

Here is the relevant mail,

On Mon, Jul 30, 2001 at 01:11:32PM -0400, Lawrence Greenfield wrote:
} BSD softupdates allows you to call fsync() on the file, and this will } sync the directories all the way up to the root if necessary. }
} Thus BSD fsync() actually guarantees that when it returns, the file } (and all of it's filenames) will survive a reboot.
}
} Sendmail does:
} fd = open(tmp)
} write(fd)
} fsync(fd)
} rename(tmp, final)
} fsync(fd)
}
} Cyrus IMAP does:
} fd = open(tmp)
} write(fd)
} fsync(fd)
} link(tmp, final1)
} link(tmp, final2)
} link(tmp, final3)
} fsync(fd)
} close(fd)
} unlink(tmp)
}
} The idea that Linux fsync() doesn't actually make the file survive } reboots is pretty ridiculous.

As you can see, the 'sync a path leading to the file' semantics from SuS don't work in the Cyrus IMAP case as is specifically requires to have _all_ paths committed to disk before fsync returns.

MTA's that want to do reliable deliveries using multiple processes (or on a networked filesystem) tend to not use rename because it implicitly unlinks the target if it already exists and this could lead to loss of mail that was already considered as being delivered.

Jan

MTA authors don't share this. lost+found is "invisible" for the application that created the file.

I have yet to meet a distribution which scans lost+found at boot time and syslogs found files or sends root a mail.

So, effectively, lost+found will NOT be sufficient. Discarding file names at will is not a good thing.

Matthias Andree

The whole idea is to have a switch to turn on BSD-style synchronous directory update semantics. Nothing more, nothing you would not be able to get rid off. In fact, you can mount file systems async on BSD as well, but you'd better not have the machine crash. Irrecoverable file system damage can result. As a compromise, softupdates are nearly as fast as async, but FS damage is guaranteed to be recoverable.

In either case (async or soft-updates), files can end up in lost+found after the control had been returned to the application that called open or link.

Matthias Andree

It's rather my imagination that tracking hard links might be easier than symlinks because hard links share the inode number. A more advanced (and complex) implementation might prove the imagination wrong. I don't want to consider which one is more efficient.

Matthias Andree

It's a maintenance issue.

You effectively start wrapping up all relevant syscalls and have system-specific interfaces. One wants the directory fsync()ed, the other offers a special other trick to get the data flushed... what useful is portability then if systems are so different?

No wonder if the application chooses fully-synchronous operation on Linux.

Matthias Andree

To fill in more of the table, Qmail does:

fd = open(tmp)
write(fd)
fsync(fd)
link(tmp,final)
close(fd)

...and Postfix does:

fd = open(final)
write(fd)
(should be an "fsync(fd)" here, but I cannot find it) fchmod(fd,+execute)
fsync(fd)
close(fd)

Postfix apparently uses the execute bit to indicate that delivery is complete. I am probably misreading the source (version 20010228 Patchlevel 3), but I do not see any fsync() between the write and the fchmod. Surely it is there or this delivery scheme is not reliable on any system, since without an intervening fsync() the writes to the data and the permissions can happen out of order.

Anyway, it is certainly true that it is largely useless to have fsync() commit only one path to a file; many applications expect to be able to force a simple link(x,y) to be committed to disk.

I know this thread is already much too long, but I am still not sure I understand the conclusions. I *think* it is true that:

1) People disagree about what SuS mandates, but at least a few critical developers (e.g., sct) say it definitely does not require synchronizing directory entries for fsync().

2) It would be fairly easy and efficient for fsync() to chase one chain of directory entries up to the root, but a lot harder and slower to find and commit all of them.

3) Most (?) core developers, including Linus (?), would not object to "dirsync" as a mount option and/or directory attribute, but somebody has to rise to the occasion and create the patches.

Is this an accurate summary?

- Pat

http://cr.yp.to/qmail/faq/reliability.html

Not really. The error code if fsync() or close failed are propagated back to the caller who then decides what to do. smtpd.c nukes the file. postdrop.c/sendmail.c do not, but the pickup daemon will see that the file had problems on sync and discard it.

I'm asking Wietse off-list how reliable this approach is and will report back privately. It should be fairly reliable.

BSD FFS + softupdates sync all file names, traversing from the mount point down to the actual directory entries that need to be synched.

For BSD FFS + softupdates, this is already done.

It looks so to me. After the MTA behaviour has been dug up, the dirsync option could be even weaker if fsync() behaved like FFS + softupdates:
sync the directory entries, including those of link and rename, as well.

The only things to consider would be unlink and symlink. symlinks are tough since you cannot open() them. Not sure about unlink, looks as if there's really no way apart from fsync(2)ing the directory or sync(2)ing the world for these two unless there's a dirsync option coming up.

Matthias Andree

You should teach the guy about MTAs one day.

Softupdates and other filesystems are perfectly safe with mailers which work with the filesystem instead of prescribing the filesystem authors how they should do their work ;)

Oh, most of those other mailers are RFC compliant too, but that's a separate issue ;)

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

...which is consistent. Qmail is assuming that the link() is synchronous, as it was back in the "Good Old Days" of stock FFS.

That is not the problem. The problem is that the system could start flushing blocks to disk after the call to fchmod and before the call to fsync. If so, the system could write the mode bits first and then crash before writing the data, leaving the execute bit set on the file but without valid data within. This could result in a corrupted mail message.

To avoid this, Postfix *must* do fsync() or fdatasync() after the write() and before the fchmod()+fsync(); that will insure that the execute bit implies valid ("committed") data in the file. I was unable to find any such call to fsync() or fdatasync(), but as I mentioned, I am probably simply misreading the code.

...and the Linux developers continue to insist that this is "stupid". Ah, the joys of gaps in standards.

Ideally, this would be an option you could set per-application (as opposed to per-directory or per-mountpoint), because we are really talking about allowing Linux to support applications written for BSD file system semantics. It is not obvious to me what the best implementation for that would be, though. Maybe just a compile-time option to choose the appropriate open/link/rename/etc. operations.

- Pat

Which, from my reading of the archives, even BSD folk say is a "Bad Thing(tm)."

mrc

     Mike Castle      [email protected]      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

...which is consistent, looking at the other things qmail does in strange ways ;)

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

Debian Woody ...

Zilvinas Valinskas

"Patrick J. LoPresti" <[email protected]> writes:

[snip sendmail/cyrus/qmail/postfix]

Just in case anyone cares here's what exim does (AFAICS)...

int fd1 = open(f1);
write(fd1);
fsync(fd1);

int fd2 = open(tmp);
write(fd2);
fsync(fd2);
rename(tmp, f2); // Good at this point.

So that seems to rely on all dir operations being sync.

Ps. I did a patch for exim to do the dir sync though...

http://www.and.org/exim-3.31-dirfsync.patch

# James Antill -- [email protected]
:0:
* ^From: .*james@and\.org
/dev/null