Appropriate use of sync() from user space?
Hi all:
Quick summary: We have a vendor who is claiming that it is required for their userspace program to execute sync(), and I am looking for some sort of authoritative document or person to refer them to that will state that this belief is incorrect and/or that this architecture is not acceptable in a Unix environment.
I checked Google and the archives and didn't find anything appropriate. Unfortunately, the word "sync" is very popular. :-)
We have users who have been experiencing 3 to 5 minutes "freezes" for a particular command which often times out and fails. I traced this down from the commercial userspace program (IBM Rational ClearCase / "cleartool mkview") that they are executing to a backend "view_server" process (also IBM Rational ClearCase) that is running sync() as a means of synchronizing their database to disk before proceeding, and VMware using a "large" memory mapped file to back it's virtual "RAM". The sync() for my computer normally completes in 7 to 8 seconds. The sync() for some of our users is taking 5 minutes or longer. This can be demonstrated simply by typing "time sync" from the command line at intervals. The time itself is relevant because if it finishes before a timeout elapses - the operation works (albeit slowly). If the timeout elapses, the operation fails.
The vendor stated that sync() is integral to their synchronization process to ensure all files reach disk before they are accessed, and that this is not a defect in their product. We have a work around - run "sync" before calling their command, and this generally avoids the failures.
I think the use of sync() in this regard is a hack. According to POSIX.1 and the Linux man pages, it seems clear to me that sync() does not guarantee data integrity (bytes guaranteed to have reached disk) - and it also seems clear that forcing all system data to flush out in response to a minor command is over kill. Like cutting down the forest to harvest fruit from a single tree.
I'm wondering what you think.
Thanks!
Signature · 1 line
Mark Mielke<[email protected]>
Quoted 6 lines · Mark Mielke
The vendor stated that
sync()is integral to their synchronization process to ensure all files reach disk before they are accessed, and that this is not a defect in their product. We have a work around - run "sync" before calling their command, and this generally avoids the failures.
Quoted 8 lines
I think the use of
sync()in this regard is a hack. According to POSIX.1 and the Linux man pages, it seems clear to me thatsync()does not guarantee data integrity (bytes guaranteed to have reached disk) - and it also seems clear that forcing all system data to flush out in response to a minor command is over kill. Like cutting down the forest to harvest fruit from a single tree.I'm wondering what you think.
Totally agree.
The susv3 man page for sync() is pretty clear: "The writing, although scheduled, is not necessarily complete upon return from sync()."
They should probably be using msync()/fsync()/fdatasync() which only affect the specified files and are supposed to wait for the data to hit the storage device. Of course this would require them to do something for each file they touch rather than once at the end of the whole operation.
Chris
Signature · 5 lines
Chris Friesen Software Developer GENBAND [email protected] www.genband.com
Hello,
Quoted 29 lines · Mark Mielke
Quick summary: We have a vendor who is claiming that it is required for their userspace program to execute
sync(), and I am looking for some sort of authoritative document or person to refer them to that will state that this belief is incorrect and/or that this architecture is not acceptable in a Unix environment.I checked Google and the archives and didn't find anything appropriate. Unfortunately, the word "sync" is very popular. :-)
We have users who have been experiencing 3 to 5 minutes "freezes" for a particular command which often times out and fails. I traced this down from the commercial userspace program (IBM Rational ClearCase / "cleartool mkview") that they are executing to a backend "
view_server" process (also IBM Rational ClearCase) that is runningsync()as a means of synchronizing their database to disk before proceeding, and VMware using a "large" memory mapped file to back it's virtual "RAM". Thesync()for my computer normally completes in 7 to 8 seconds. Thesync()for some of our users is taking 5 minutes or longer. This can be demonstrated simply by typing "time sync" from the command line at intervals. The time itself is relevant because if it finishes before a timeout elapses - the operation works (albeit slowly). If the timeout elapses, the operation fails.The vendor stated that
sync()is integral to their synchronization process to ensure all files reach disk before they are accessed, and that this is not a defect in their product. We have a work around - run "sync" before calling their command, and this generally avoids the failures.
Quoted 6 lines
I think the use of
sync()in this regard is a hack. According to POSIX.1 and the Linux man pages, it seems clear to me thatsync()does not guarantee data integrity (bytes guaranteed to have reached disk) - and it also seems clear that forcing all system data to flush out in response to a minor command is over kill. Like cutting down the forest to harvest fruit from a single tree.
Actually the manpage is wrong. Linux waits for all data to be safely on
disk before sync returns. So calling sync is a correct way (although inefficient at times) to achieve data integrity. What kernel version are you using? Different kernel versions are differently efficient when doing sync(2) and quite some effort went to make sync less prone to livelocks in recent kernels...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 35 lines · Jan Kara
Quoted 29 lines
Quick summary: We have a vendor who is claiming that it is required for their userspace program to execute
sync(), and I am looking for some sort of authoritative document or person to refer them to that will state that this belief is incorrect and/or that this architecture is not acceptable in a Unix environment.I checked Google and the archives and didn't find anything appropriate. Unfortunately, the word "sync" is very popular. :-)
We have users who have been experiencing 3 to 5 minutes "freezes" for a particular command which often times out and fails. I traced this down from the commercial userspace program (IBM Rational ClearCase / "cleartool mkview") that they are executing to a backend "
view_server" process (also IBM Rational ClearCase) that is runningsync()as a means of synchronizing their database to disk before proceeding, and VMware using a "large" memory mapped file to back it's virtual "RAM". Thesync()for my computer normally completes in 7 to 8 seconds. Thesync()for some of our users is taking 5 minutes or longer. This can be demonstrated simply by typing "time sync" from the command line at intervals. The time itself is relevant because if it finishes before a timeout elapses - the operation works (albeit slowly). If the timeout elapses, the operation fails.The vendor stated that
sync()is integral to their synchronization process to ensure all files reach disk before they are accessed, and that this is not a defect in their product. We have a work around - run "sync" before calling their command, and this generally avoids the failures.Quoted 6 lines
I think the use of
sync()in this regard is a hack. According to POSIX.1 and the Linux man pages, it seems clear to me thatsync()does not guarantee data integrity (bytes guaranteed to have reached disk) - and it also seems clear that forcing all system data to flush out in response to a minor command is over kill. Like cutting down the forest to harvest fruit from a single tree.
Quoted 7 lines
Actually the manpage is wrong. Linux waits for all data to be safely on
disk before sync returns. So calling sync is a correct way (although inefficient at times) to achieve data integrity. What kernel version are you using? Different kernel versions are differently efficient when doing sync(2) and quite some effort went to make sync less prone to livelocks in recent kernels...
Let's make sure to keep Michael Kerrisk cc'd if anything needs to be clarified in the manpages.
Quoted 45 lines · David Rientjes
Quoted 35 lines · Jan Kara
Quoted 29 lines
Quick summary: We have a vendor who is claiming that it is required for their userspace program to execute
sync(), and I am looking for some sort of authoritative document or person to refer them to that will state that this belief is incorrect and/or that this architecture is not acceptable in a Unix environment.I checked Google and the archives and didn't find anything appropriate. Unfortunately, the word "sync" is very popular. :-)
We have users who have been experiencing 3 to 5 minutes "freezes" for a particular command which often times out and fails. I traced this down from the commercial userspace program (IBM Rational ClearCase / "cleartool mkview") that they are executing to a backend "
view_server" process (also IBM Rational ClearCase) that is runningsync()as a means of synchronizing their database to disk before proceeding, and VMware using a "large" memory mapped file to back it's virtual "RAM". Thesync()for my computer normally completes in 7 to 8 seconds. Thesync()for some of our users is taking 5 minutes or longer. This can be demonstrated simply by typing "time sync" from the command line at intervals. The time itself is relevant because if it finishes before a timeout elapses - the operation works (albeit slowly). If the timeout elapses, the operation fails.The vendor stated that
sync()is integral to their synchronization process to ensure all files reach disk before they are accessed, and that this is not a defect in their product. We have a work around - run "sync" before calling their command, and this generally avoids the failures.Quoted 6 lines
I think the use of
sync()in this regard is a hack. According to POSIX.1 and the Linux man pages, it seems clear to me thatsync()does not guarantee data integrity (bytes guaranteed to have reached disk) - and it also seems clear that forcing all system data to flush out in response to a minor command is over kill. Like cutting down the forest to harvest fruit from a single tree.Quoted 7 lines
Actually the manpage is wrong. Linux waits for all data to be safely on
disk before sync returns. So calling sync is a correct way (although inefficient at times) to achieve data integrity. What kernel version are you using? Different kernel versions are differently efficient when doing sync(2) and quite some effort went to make sync less prone to livelocks in recent kernels...
Quoted 2 lines
Let's make sure to keep Michael Kerrisk cc'd if anything needs to be clarified in the manpages.
also, you may want to check if they are really doing a 'sync' (syncing the entire filesystem) or just a 'fsync' (syncing the file). Depending on the technical depth of the people you are talking to, they may say sync when what is actually happening is a fsync.
there is little dispute that fsync is correct, but not a complete answer to the issue. take a look at the LWN article on the subject at http://lwn.net/Articles/457667
Ext3 has a pathalogical condition where a sync to one file can force a complete journal flush, which isn't as bad as a sync of the entire filesystem, but can still take a long time if there is other ongoing write activity on the system (I knwo I've read about fsyncs taking longer than 30 seconds, and I think I've heard of them taking minutes). As far as I know, Ext3 is the only filesystem to suffer this problem, but unfortunantly it's the default filesystem on most linux distros.
David Lang
Quoted 11 lines · On 10/18/2011 08:03 PM, [email protected] wrote:
Quoted 7 lines · David Rientjes
Quoted 6 lines · Jan Kara
Quoted 6 lines
Quick summary: We have a vendor who is claiming that it is required for their userspace program to execute
sync(), and I am looking for some sort of authoritative document or person to refer them to that will state that this belief is incorrect and/or that this architecture is not acceptable in a Unix environment.also, you may want to check if they are really doing a 'sync' (syncing the entire filesystem) or just a 'fsync' (syncing the file). Depending on the technical depth of the people you are talking to, they may say sync when what is actually happening is a fsync.
there is little dispute that fsync is correct, but not a complete answer to the issue. take a look at the LWN article on the subject at http://lwn.net/Articles/457667
Thanks, Dave. Yes, the vendor is doing a real sync() - and is also doing a few fsync(). We managed to convince them that this is a defect through their sales channel as technical failed. Oh well.
I note that you say "there is little dispute that fsync is correct ..." - which I also would have assumed, but I found very little authoritative documents on this. I think it is just so "self evident" (which the exception that you state in ext3) that people don't tend to do this.
In the mean time, our users are literally doing "sync; cleartool mkview ..." which forces most of the data out to disk before starting the vendor command, so that the IBM Rational "cleartool mkview" is able to complete successfully without timing out and failing after 3 to 5 minutes. *sigh*
The problematic file seems to be a VMware Player mmap()'d file with size of 1 GByte or more. ClearCase "cleartool mkview" vs VMware Player normal operation. *sigh x 2*
Thanks,
mark
Signature · 1 line
Mark Mielke<[email protected]>