Skip to contentSkip to conversation

LWN article: ext4 and data loss

DISCUSSION Michael Monnerie 5 participants 15 messages Mar 12, 2009 – Mar 18, 2009

lore
Share & tools
lore
#0 lore raw

http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/

Very good, maybe similar patches for XFS would help? IANA Coder, but could be a hint.

mfg zmi

Signature · 5 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw

Am Donnerstag 12 März 2009 schrieb Eric Sandeen:

Quoted 10 lines
Quoted 6 lines · Michael Monnerie

http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/

Very good, maybe similar patches for XFS would help? IANA Coder, but could be a hint.

mfg zmi

ext4 is taking its hints from XFS in this regard, not the other way around. XFS dealt with this long ago.

Hmmm, I remember having had similar issues with XFS not to long ago, where at least some KDE configuration were lost or truncated. It seems applications will have to get rid of behavioral assumptions regation filesystem and use safe writing via fsync and whatever else for configuration and other important files.

I am thinking about a bug report for KDE at least.

Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 4 lines · Martin Steigerwald

Am Donnerstag 12 März 2009 schrieb Eric Sandeen:

Quoted 3 lines
Quoted 2 lines · Michael Monnerie

http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Very good, maybe similar patches for XFS would help? IANA Coder, but could be a hint.

mfg zmi

ext4 is taking its hints from XFS in this regard, not the other way around. XFS dealt with this long ago.

Hmmm, I remember having had similar issues with XFS not to long ago, where

depends on what you mean by not too long ago, I think. Yes, kde had this issue on xfs too, and xfs gave up on teaching apps to fsync, and implemented the same sorts of things ext4 has done (or will do) to mitigate this quite some time ago.

Quoted 4 lines

at least some KDE configuration were lost or truncated. It seems applications will have to get rid of behavioral assumptions regation filesystem and use safe writing via fsync and whatever else for configuration and other important files.

It's simple. Want your data safe on disk? fsync. There's not a lot more to it than that. (and if fsync hurts perf too much, re-think how you are storing your data)

Filesystems can hack around some heuristics to try to make unsafe apps safer, but in the end, it's the app's job to make sure a buffered write hits permanent storage when it matters.

-Eric

Quoted 2 lines

I am thinking about a bug report for KDE at least.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw
Quoted 9 lines · Eric Sandeen

It's simple. Want your data safe on disk? fsync. There's not a lot more to it than that. (and if fsync hurts perf too much, re-think how you are storing your data)

Filesystems can hack around some heuristics to try to make unsafe apps safer, but in the end, it's the app's job to make sure a buffered write hits permanent storage when it matters.

Stewart Smith has a highly entertaining presentation on this very topic.

http://www.linux.org.au/conf/2007/talk/278.html

Signature · 8 lines
Greg Banks, P.Engineer, SGI Australian Software Group.
the brightly coloured sporks of revolution.
I don't speak for SGI.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw
Quoted 5 lines · Greg Banks
Quoted 4 lines · Eric Sandeen

It's simple. Want your data safe on disk? fsync. There's not a lot more to it than that. (and if fsync hurts perf too much, re-think how you are storing your data)

Quoted 9 lines
Quoted 4 lines

Filesystems can hack around some heuristics to try to make unsafe apps safer, but in the end, it's the app's job to make sure a buffered write hits permanent storage when it matters.

Stewart Smith has a highly entertaining presentation on this very topic.

http://www.linux.org.au/conf/2007/talk/278.html

and

http://www.flamingspork.com/talks/2007/06/eat_my_data.odp

In hindsight, http://linuxmafia.com/faq/Filesystems/reiserfs.html is entertaining, too ;)

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
Quoted 5 lines · Eric Sandeen

In hindsight, http://linuxmafia.com/faq/Filesystems/reiserfs.html is entertaining, too ;)

There was a fascinating internal thread about that one when somebody noticed it a few years back :-)

Signature · 8 lines
Greg Banks, P.Engineer, SGI Australian Software Group.
the brightly coloured sporks of revolution.
I don't speak for SGI.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw
In reply to Eric Sandeen

Am Donnerstag 12 März 2009 schrieb Eric Sandeen:

Quoted 13 lines
Quoted 12 lines · Martin Steigerwald

Am Donnerstag 12 März 2009 schrieb Eric Sandeen:

Quoted 10 lines
Quoted 6 lines · Michael Monnerie

http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/

Very good, maybe similar patches for XFS would help? IANA Coder, but could be a hint.

mfg zmi

ext4 is taking its hints from XFS in this regard, not the other way around. XFS dealt with this long ago.

Quoted 7 lines
Quoted 2 lines

Hmmm, I remember having had similar issues with XFS not to long ago, where

depends on what you mean by not too long ago, I think. Yes, kde had this issue on xfs too, and xfs gave up on teaching apps to fsync, and implemented the same sorts of things ext4 has done (or will do) to mitigate this quite some time ago.

Well 2.6.28 and 2.6.27.7. See

http://oss.sgi.com/archives/xfs/2008-12/msg00540.html

Quoted 5 lines
Quoted 4 lines

at least some KDE configuration were lost or truncated. It seems applications will have to get rid of behavioral assumptions regation filesystem and use safe writing via fsync and whatever else for configuration and other important files.

Quoted 7 lines

It's simple. Want your data safe on disk? fsync. There's not a lot more to it than that. (and if fsync hurts perf too much, re-think how you are storing your data)

Filesystems can hack around some heuristics to try to make unsafe apps safer, but in the end, it's the app's job to make sure a buffered write hits permanent storage when it matters.

Hmmm, okay. So here is:

http://bugs.kde.org/187172

Feel free to add there. You'd need a bugzilla login tough.

Ciao,

Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw
Quoted 1 line · Martin Steigerwald

Am Donnerstag 12 März 2009 schrieb Eric Sandeen:

...

Quoted 11 lines
Quoted 3 lines

Filesystems can hack around some heuristics to try to make unsafe apps safer, but in the end, it's the app's job to make sure a buffered write hits permanent storage when it matters.

Hmmm, okay. So here is:

http://bugs.kde.org/187172

Feel free to add there. You'd need a bugzilla login tough.

Ciao,

Perhaps you can try this and add info if you like.

There is an environment variable KDE_EXTRA_FSYNC, put in with this changelog:

Do not paranoidly sync every time, it causes I/O performance problems for some users. People who still want it for whatever reason like using XFS can set $KDE_EXTRA_FSYNC to 1."

This is not "extra" - it is necessary if you actually want the data to be on-disk.

See also: http://lists.kde.org/?l=kde-devel&m=120880682813170&w=2

(note however that xfs is _not_ "zeroing just to be sure...")

http://api.kde.org/4.x-api/kdelibs-apidocs/kde3support/html/k3tempfile_8cpp-source.html

bool
K3TempFile::sync()

    int result = 0;

    if (mStream)
    {
       do {
          result = fflush(mStream); // We need to flush first otherwise
fsync may not have our data
       }
       while ((result == -1) && (errno == EINTR));

       if (result)
       {
          kWarning() << "K3TempFile: Error trying to flush " << mTmpName
 " << strerror(errno);
          mError = errno;
       }
    }

    if (mFd >= 0)
    {
       if( qgetenv( "KDE_EXTRA_FSYNC" ) == "1" )
       {
          result = FDATASYNC(mFd);
          if (result)
          {
             kWarning() << "K3TempFile: Error trying to sync " <<
mTmpName << ": " << strerror(errno);
             mError = errno;
          }
       }
    }

    return (mError == 0);

}

So somebody knew it was the right thing to do, but then turned it off, probably because of ext3's painful behavior on fsync (see also: firefox "bug" from a while ago)

what's interesting is that the sync() method isn't automatically called from the other methods, near as I can tell, so it's up to the api user to invoke it; and even then it only does fflush() not fsync() by default, which doesn't actually push things to disk.

I'd suggest turning on this KDE_EXTRA_FSYNC and see how things go from there on.

Although, the API refs say that this is deprecated in favor of KTemporaryFile, and I find no reference whatsoever to "sync" of any kind in ktemporaryfile.cpp.

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#9 lore raw
Quoted 4 lines · Eric Sandeen
Quoted 1 line · Martin Steigerwald

Am Donnerstag 12 März 2009 schrieb Eric Sandeen:

...

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Filesystems can hack around some heuristics to try to make unsafe apps safer, but in the end, it's the app's job to make sure a buffered write hits permanent storage when it matters.

Hmmm, okay. So here is:

http://bugs.kde.org/187172

Feel free to add there. You'd need a bugzilla login tough.

... getting OT sorry but in case anyone else is interested,
http://api.kde.org/4.x-api/kdelibs-apidocs/kdecore/html/classKSaveFile.html
also seems relevant.

I'd like to know just what KDE is doing here, so digging a little.

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw
In reply to Eric Sandeen

Am Samstag 14 März 2009 schrieben Sie:

Quoted 13 lines
Quoted 1 line · Martin Steigerwald

Am Donnerstag 12 März 2009 schrieb Eric Sandeen:

...

Quoted 8 lines
Quoted 3 lines

Filesystems can hack around some heuristics to try to make unsafe apps safer, but in the end, it's the app's job to make sure a buffered write hits permanent storage when it matters.

Hmmm, okay. So here is:

http://bugs.kde.org/187172

Quoted 8 lines
Quoted 3 lines

Feel free to add there. You'd need a bugzilla login tough.

Ciao,

Perhaps you can try this and add info if you like.

There is an environment variable KDE_EXTRA_FSYNC, put in with this changelog:

[...]

Added this and your other post and added

export KDE_EXTRA_FSYNC=1

to my .zshrc. Just logged out and on and do not notice a difference in speed. At least no abysmal performance degradation.

Would need to switch of the machine hardly to see if it helps. Not today anymore tough.

Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#11 lore raw
In reply to Martin Steigerwald

[ ... usual misunderstanding about caching and transactions ... ]

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

ext4 is taking its hints from XFS in this regard, not the other way around. XFS dealt with this long ago.

Hmmm, I remember having had similar issues with XFS not to long ago,

Quoted 20 lines
Quoted 4 lines

depends on what you mean by not too long ago, I think. Yes, kde had this issue on xfs too, and xfs gave up on teaching apps to fsync, and implemented the same sorts of things ext4 has done (or will do) to mitigate this quite some time ago.

Well 2.6.28 and 2.6.27.7. See
http://oss.sgi.com/archives/xfs/2008-12/msg00540.html

Quoted 12 lines
Quoted 4 lines

[ ... ] applications will have to get rid of behavioral assumptions regation filesystem and use safe writing via fsync and whatever else for configuration and other important files.

It's simple. Want your data safe on disk? fsync. There's not a lot more to it than that. (and if fsync hurts perf too much, re-think how you are storing your data)

Filesystems can hack around some heuristics to try to make unsafe apps safer, but in the end, it's the app's job to make sure a buffered write hits permanent storage when it matters.

This discussion is partially misguided, but then how many people study storage system semantics...

The goal is to do atomic transactions: within a transaction there are no guarantees, but at the end of transaction things get stored permanently.

Unfortunately as described 'ext3' has historically done ''rolling'' auto-saving, so many people and application developers have not appreciated the need for transaction semantics (common attitude, for example how many programmers for example check the return code of 'close'?).

Now under Linux and POSIX it is essentially impossible to do atomic, persistent transactions, because:

* 'fsync' does NOT guarantee persistency. Only that *RAM* buffers are flushed; therefore host adapter and disk buffers are not required to be flushed.

* Linux write barriers also only guaranteeq ordering and not persistence, and there is a number of misguided people who think that this is how things should be.

Quoted 2 lines

Hmmm, okay. So here is:
http://bugs.kde.org/187172

In practice, for systems without caching host adapters, and with 'ext3', most of the time informal ''rolling'' transactions every 5s fool most people/work as if they were right, and as asserted this has lulled developers into thinking that transactions don't matter. Too bad this kills performance and/or reliability on anything else.

This is just another example of how much userspace sucks

http://lwn.net/Articles/192214/
http://kernelslacker.livejournal.com/81262.html
http://mirror.linux.org.au/pub/linux.conf.au/2007/video/talks/38.pdf

Note that in a proper design where 'fsync' would guarantee persistence, like in every transactional systems, lots of small transactions have very sharp performance implications. People who earn a living doing transactional systems therefore spend a great deal of money and effort designing them to perform well despite lots of small transactions, with 15k drives, vast parallel RAID, bettery backed logs, etc.

You cannot have all of these:

* Reliable transactions.
* Fast with lots of small transactions.
* With cheap hardware.

In the end one must decided whether to follow the Microsoft strategy (f*ck doing the right thing, cultivate bugs that users are relying on) or the UNIX one (try to do the right thing).

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#12 lore raw
In reply to Michael Monnerie

Am Donnerstag 12 März 2009 schrieb Michael Monnerie:

Quoted 4 lines

http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/

Very good, maybe similar patches for XFS would help? IANA Coder, but could be a hint.

BTW I think you gave away a subcriber only link. The article is not officially available yet:

http://lwn.net/Articles/322823/

LWN might not be too happy with this.

Information about this is available elsewhere like on Heise:

http://www.heise.de/english/newsticker/news/134483

Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#13 lore raw
Quoted 7 lines · On Samstag 14 März 2009 Martin Steigerwald wrote:

Am Donnerstag 12 März 2009 schrieb Michael Monnerie:

Quoted 1 line

http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/

BTW I think you gave away a subcriber only link. The article is not officially available yet:
http://lwn.net/Articles/322823/
LWN might not be too happy with this.

Nope. I am a subscriber, and can generate such Links in order to provider an article to others. If you read that, they offer a "Free trial subscription" for 3 months, and encourage you to pay then. I can recommend lwn.net a lot, so maybe some people subscribe there because of my posting :-)

mfg zmi

Signature · 10 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#14 lore raw

Am Montag 16 März 2009 schrieb Michael Monnerie:

Quoted 9 lines · On Samstag 14 März 2009 Martin Steigerwald wrote:
Quoted 7 lines

Am Donnerstag 12 März 2009 schrieb Michael Monnerie:

Quoted 1 line

http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/

BTW I think you gave away a subcriber only link. The article is not officially available yet:
http://lwn.net/Articles/322823/
LWN might not be too happy with this.

Quoted 5 lines

Nope. I am a subscriber, and can generate such Links in order to provider an article to others. If you read that, they offer a "Free trial subscription" for 3 months, and encourage you to pay then. I can recommend lwn.net a lot, so maybe some people subscribe there because of my posting :-)

Okay, nice to hear.

I like LWN a lot and consider subscribing to it.

Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs