Skip to contentSkip to conversation

Status of fsync() wrt mail servers

DISCUSSION [email protected] 5 participants 12 messages Sep 10, 2003 – Sep 12, 2003

lore
Share & tools
lore
#0 lore raw

Hello,
I'm in the process of researching OSes and filesystems for a new mail system. I'm hoping to use linux+reiserfs+postfix, and I'm wondering where reiserfs stands wrt to fsync(). Does reiserfs provide a mechanism to have truly synchronous writes with a single fsync() call? Thanks

I've read a very long linux-kernel thread[1] from 2001 where Matthias Andree was petitioning for changes in the fsync() behavior, and I'm having trouble following what happened since then.

Thanks

[1] http://lists.insecure.org/lists/linux-kernel/2001/Jul/3545.html

Signature · 2 lines
Cameron Moore
[ The early bird gets the worm, but the second mouse gets the cheese. ]
#1 lore raw
2003-09-10T01:29:53 [email protected]:
Quoted 4 lines

I'm in the process of researching OSes and filesystems for a new mail system. I'm hoping to use linux+reiserfs+postfix, and I'm wondering where reiserfs stands wrt to fsync(). Does reiserfs provide a mechanism to have truly synchronous writes with a single fsync() call? Thanks

I'm not really fond of the phrase "truly synchronous writes"; it can be read different ways by different people.

What postfix demands (if you wish to adhere strictly to some peoples' interpretations of RFCs) is that when fsync returns, the filesystem guarantees that even if there's a crash an instant after, the file, data as well as metadata, will be intact when the machine comes up again. This is in support of a desire to positively commit to the sender that the receiving MTA has accepted receipt for a message.

-Bennett

#2 lore raw
Quoted 9 lines · Bennett Todd
2003-09-10T01:29:53 [email protected]:
Quoted 4 lines

I'm in the process of researching OSes and filesystems for a new mail system. I'm hoping to use linux+reiserfs+postfix, and I'm wondering where reiserfs stands wrt to fsync(). Does reiserfs provide a mechanism to have truly synchronous writes with a single fsync() call? Thanks

I'm not really fond of the phrase "truly synchronous writes"; it can be read different ways by different people.

Quoted 7 lines

What postfix demands (if you wish to adhere strictly to some peoples' interpretations of RFCs) is that when fsync returns, the filesystem guarantees that even if there's a crash an instant after, the file, data as well as metadata, will be intact when the machine comes up again. This is in support of a desire to positively commit to the sender that the receiving MTA has accepted receipt for a message.

This is what reiserfs does, the metadata is on disk after an fsync, including any renames.

-chris

#3 lore raw
2003-09-10T07:14:34 Chris Mason:
Quoted 11 lines · Bennett Todd
Quoted 7 lines

What postfix demands (if you wish to adhere strictly to some peoples' interpretations of RFCs) is that when fsync returns, the filesystem guarantees that even if there's a crash an instant after, the file, data as well as metadata, will be intact when the machine comes up again. This is in support of a desire to positively commit to the sender that the receiving MTA has accepted receipt for a message.

This is what reiserfs does, the metadata is on disk after an fsync, including any renames.

Metadata, yes, I've got that. How about the data? Does return from fsync guarantee that the data will be intact as well?

-Bennett

#4 lore raw
Quoted 10 lines · Bennett Todd
2003-09-10T07:14:34 Chris Mason:
Quoted 9 lines · Bennett Todd
Quoted 7 lines

What postfix demands (if you wish to adhere strictly to some peoples' interpretations of RFCs) is that when fsync returns, the filesystem guarantees that even if there's a crash an instant after, the file, data as well as metadata, will be intact when the machine comes up again. This is in support of a desire to positively commit to the sender that the receiving MTA has accepted receipt for a message.

Quoted 5 lines
Quoted 2 lines

This is what reiserfs does, the metadata is on disk after an fsync, including any renames.

Metadata, yes, I've got that. How about the data? Does return from fsync guarantee that the data will be intact as well?

Yes

-chris

#5 lore raw
Quoted 11 lines · * [email protected] (Chris Mason) [2003.09.10 07:31]:
Quoted 10 lines · Bennett Todd
2003-09-10T07:14:34 Chris Mason:
Quoted 9 lines · Bennett Todd
Quoted 7 lines

What postfix demands (if you wish to adhere strictly to some peoples' interpretations of RFCs) is that when fsync returns, the filesystem guarantees that even if there's a crash an instant after, the file, data as well as metadata, will be intact when the machine comes up again. This is in support of a desire to positively commit to the sender that the receiving MTA has accepted receipt for a message.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

This is what reiserfs does, the metadata is on disk after an fsync, including any renames.

Metadata, yes, I've got that. How about the data? Does return from fsync guarantee that the data will be intact as well?

Yes

Thanks for hashing this out while I was asleep. :-) Guess I'll go morph into a die-hard Reiser fan now. Thanks again

Signature · 2 lines
Cameron Moore
$\="Hacker";$,="another ";print"Just ","Perl ";
#6 lore raw
Quoted 15 lines · Cameron Moore
Quoted 14 lines · * [email protected] (Chris Mason) [2003.09.10 07:31]:
Quoted 13 lines · Bennett Todd
2003-09-10T07:14:34 Chris Mason:
Quoted 11 lines · Bennett Todd
Quoted 7 lines

What postfix demands (if you wish to adhere strictly to some peoples' interpretations of RFCs) is that when fsync returns, the filesystem guarantees that even if there's a crash an instant after, the file, data as well as metadata, will be intact when the machine comes up again. This is in support of a desire to positively commit to the sender that the receiving MTA has accepted receipt for a message.

This is what reiserfs does, the metadata is on disk after an fsync, including any renames.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Metadata, yes, I've got that. How about the data? Does return from fsync guarantee that the data will be intact as well?

Yes

Thanks for hashing this out while I was asleep. :-) Guess I'll go morph into a die-hard Reiser fan now. Thanks again

The whole perpose of fsync, is to flush the data to the disk. That works even with ext2, but it has the possibility of not flushing the meta-data.

With a journaled filesystem and fsync, you will have the data and meta-data on the disk after the call returns.

Isn't that part of Posix or sus?

#7 lore raw
Quoted 24 lines · * [email protected] (Mike Fedyk) [2003.09.10 16:32]:
Quoted 15 lines · Cameron Moore
Quoted 14 lines · * [email protected] (Chris Mason) [2003.09.10 07:31]:
Quoted 13 lines · Bennett Todd
2003-09-10T07:14:34 Chris Mason:
Quoted 11 lines · Bennett Todd
Quoted 7 lines

What postfix demands (if you wish to adhere strictly to some peoples' interpretations of RFCs) is that when fsync returns, the filesystem guarantees that even if there's a crash an instant after, the file, data as well as metadata, will be intact when the machine comes up again. This is in support of a desire to positively commit to the sender that the receiving MTA has accepted receipt for a message.

This is what reiserfs does, the metadata is on disk after an fsync, including any renames.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Metadata, yes, I've got that. How about the data? Does return from fsync guarantee that the data will be intact as well?

Yes

Thanks for hashing this out while I was asleep. :-) Guess I'll go morph into a die-hard Reiser fan now. Thanks again

Quoted 7 lines

The whole perpose of fsync, is to flush the data to the disk. That works even with ext2, but it has the possibility of not flushing the meta-data.

With a journaled filesystem and fsync, you will have the data and meta-data on the disk after the call returns.

Isn't that part of Posix or sus?

I'm not an expert on this, but my reading of the linux-kernel discussion I cited was that ext3 (at least at that revision point) only guaranteed that metadata would be written to disk when you fsync()'d a file. You had to do a second fsync() on the parent directory to guarantee that the file's data was written to disk.

Signature · 2 lines
Cameron Moore
[ Smoking cures weight problems... eventually. ]
#8 lore raw
Quoted 18 lines · Cameron Moore
Quoted 17 lines · * [email protected] (Mike Fedyk) [2003.09.10 16:32]:
Quoted 9 lines · Cameron Moore
Quoted 5 lines · * [email protected] (Chris Mason) [2003.09.10 07:31]:
Quoted 2 lines · Bennett Todd

Metadata, yes, I've got that. How about the data? Does return from fsync guarantee that the data will be intact as well?

Yes

Thanks for hashing this out while I was asleep. :-) Guess I'll go morph into a die-hard Reiser fan now. Thanks again

The whole perpose of fsync, is to flush the data to the disk. That works even with ext2, but it has the possibility of not flushing the meta-data.

With a journaled filesystem and fsync, you will have the data and meta-data on the disk after the call returns.

Quoted 7 lines
Quoted 1 line

Isn't that part of Posix or sus?

I'm not an expert on this, but my reading of the linux-kernel discussion I cited was that ext3 (at least at that revision point) only guaranteed that metadata would be written to disk when you fsync()'d a file. You had to do a second fsync() on the parent directory to guarantee that the file's data was written to disk.

Ok, I've read through part of the thread, but I remember reading it before, so...

What Matthias is asking for is to have any directory operation within the same filesystem to be on the disk when the directory operation call has completed. At the time, the only way to get that was to mount the filesystem in sync mode. That meant that any operation on that filesystem wouldn't return until it was on the disk, including data writes.

The drawback of that is that each write() (typically 4k) call would wait until it was on the disk, and that's very slow. What Matthias wanted was a combination of sync mode, but only for directory operations. That's where ext3's dirsync mount option came from.

With fsync() you write the file like normal (it's not guaranteed to be on the disk yet) where the call is buffered in memory, and it can be written out or not yet depending on memory pressure (virtual memory terms). Basically at this point it is in memory. When fsync() is called, all of the buffered data is sent to the disk, and the call doesn't return until the disk signals that it has received the data. You get that with or without dirsync.

During the processing of a message the MTA will do several renames, moves, and other calls that manipulate its directory entry. Without dirsync, it is up to the filesystem and memory pressure to determine when the meta-data from those calls actually makes it to the disk. (5 seconds with ext3 and 30 seconds with reiserfs3). With dirsync, once the directory operation call is made, it will not return to the userspace program until the meta-data has made it the disk (because during the rename and directory operation calls, there is no data only meta-data which is filesystem accounting data (directory entries and etc.)) Or more likely made it to the journal in a journaling filesystem, which is all that is needed to make the gurantee that all state will be kept intact after the journal recovery (which is automatic at boot time)

I don't know if reiserfs has a similar option (and are there modes for the other posix filesystems that this could be moved up to the vfs level?)

So nothing about the effect of fsync() was mentioned, only that with -o sync it was pointless, since each write() call was already syncronous, and without -o sync, you would have the data, but not nessicarily know what its delivery state is (if the crash is at the wrong time).

Anyone please point out any errors I may have made...

Thanks,

Mike

#9 lore raw
Quoted 2 lines · Mike Fedyk

During the processing of a message the MTA will do several renames, moves, and other calls that manipulate its directory entry.

Different MTAs implement their queue differently. Postfix doesn't rename the file into place, unlike qmail, it just drops a file, fsync()s it and that's it.

Quoted 8 lines

Without dirsync, it is up to the filesystem and memory pressure to determine when the meta-data from those calls actually makes it to the disk. (5 seconds with ext3 and 30 seconds with reiserfs3). With dirsync, once the directory operation call is made, it will not return to the userspace program until the meta-data has made it the disk (because during the rename and directory operation calls, there is no data only meta-data which is filesystem accounting data (directory entries and etc.))

Does reiserfs3.6 support dirsync? I thought it was ext3-specific until now.

Please take care to distinguish (file) meta data from directory data.

Quoted 4 lines

So nothing about the effect of fsync() was mentioned, only that with -o sync it was pointless, since each write() call was already syncronous, and without -o sync, you would have the data, but not nessicarily know what its delivery state is (if the crash is at the wrong time).

Basically, what we know is that with Linux 2.4, ext3fs, reiserfs and XFS will flush all pending transactions (per file system) that were requested prior to a synchronous operation (fsync, sync, umount, ...) out to disk.

This can heftily bite your back if you're running your MTA's queue on a large file system that has other sustained write load (logging, data bases, ...).

I recently helped one qmail user debug this; the symptom was that the first mail in a burst of mails took 2 seconds to queue, subsequent mails were queued much quicker (70 ms). He was using ext3fs, and had one huge / (root) file system and so the "synch the whole file system" behaviour made his qmail-queue synch *all* his dirty blocks to disk...

Signature · 3 lines
Matthias Andree

Encrypt your mail: my GnuPG key ID is 0x052E7D95
#10 lore raw
Quoted 3 lines · Matthias Andree

Does reiserfs3.6 support dirsync? I thought it was ext3-specific until now.

That was what I was asking too.

Quoted 2 lines

Please take care to distinguish (file) meta data from directory data.

Hmm, it seems to me, that all meta-data relating to the file fsync() was called on should be sent to the disk and waited for by the call.

Quoted 9 lines

Basically, what we know is that with Linux 2.4, ext3fs, reiserfs and XFS will flush all pending transactions (per file system) that were requested prior to a synchronous operation (fsync, sync, umount, ...) out to disk.

This can heftily bite your back if you're running your MTA's queue on a large file system that has other sustained write load (logging, data bases, ...).

Quoted 5 lines

I recently helped one qmail user debug this; the symptom was that the first mail in a burst of mails took 2 seconds to queue, subsequent mails were queued much quicker (70 ms). He was using ext3fs, and had one huge / (root) file system and so the "synch the whole file system" behaviour made his qmail-queue synch *all* his dirty blocks to disk...

Can you be sure the MTA wasn't calling sync() just to be sure (Many MTAs are funny in that they think the spool is on a seperate disk and filesystem). fsync() shouldn't be flushing anything not relating to the file it was called on (that includes directory entries related to the file also IMHO).

Also, if the MTA wasn't running as root, it shouldn't be able to make sync() affect the entire system. Is there anything that says that sync() can't just flush the user's buffers unless you're running as root or with some CAP_ capability?

Mike

#11 lore raw
Quoted 9 lines · Mike Fedyk
Quoted 5 lines

I recently helped one qmail user debug this; the symptom was that the first mail in a burst of mails took 2 seconds to queue, subsequent mails were queued much quicker (70 ms). He was using ext3fs, and had one huge / (root) file system and so the "synch the whole file system" behaviour made his qmail-queue synch *all* his dirty blocks to disk...

Can you be sure the MTA wasn't calling sync() just to be sure (Many MTAs are funny in that they think the spool is on a seperate disk and filesystem).

For qmail and Postfix I can be. sync(8) isn't remotely useful, because it's allowed to return before completion.

Quoted 3 lines

fsync() shouldn't be flushing anything not relating to the file it was called on (that includes directory entries related to the file also IMHO).

It "should", but current implementations on Linux do exactly that: flush everything. Maybe you've got better luck with BSD softupdates, but that's going to munch disk I/O big time next time you reboot after a
crash: fsck needed. It runs niced in the background so the machine boots up, but the box won't satisfy higher I/O demands. Looks like a "ex duobus malis" game.

Quoted 2 lines

Also, if the MTA wasn't running as root, it shouldn't be able to make sync() affect the entire system.

I'd like to see your plans that prevent DoS by local users...

One machine's light load is another one's DoS attack.

Quoted 2 lines

Is there anything that says that sync() can't just flush the user's buffers unless you're running as root or with some CAP_ capability?

Does the kernel track "whose dirty buffer is this" (uid_t) at all?

Signature · 3 lines
Matthias Andree

Encrypt your mail: my GnuPG key ID is 0x052E7D95