Skip to contentSkip to conversation

Linux 2.6.29

PATCH In review Linus Torvalds 84 participants 653 messages Mar 23, 2009 – May 18, 2009 2 messages not in the archive Page 2 of 3

Subsystems (11)
lore
Share & tools
lore
#255 lore raw
In reply to Jeff Garzik
Quoted 6 lines · Jeff Garzik
Quoted 1 line · Linus Torvalds

Of course, your browsing history database is an excellent example of something you should _not_ care about that much, and where performance is a lot more important than "ooh, if the machine goes down suddenly, I need to be 100% up-to-date". Using fsync on that thing was just stupid, even

If you are doing a ton of web-based work with a bunch of tabs or windows open, you really like the post-crash restoration methods that Firefox now employs. Some users actually do want to checkpoint/restore their web work, regardless of whether it was the browser, the window system or the OS that crashed.

You may not care about that, but others do care about the integrity of the database that stores the active FF state (Web URLs currently open), a database which necessarily changes for each URL visited.

as one of those users with many windows tabs open (a couple hundred normally), even the curent firefox behavior isn't good enough because it doesn't let me _not_ load everything back in when a link I go to triggers a crash in firefox every time it loads.

so what I do is do a git commit in cron every min of the history file. git can do the fsync as needed to get it to disk reasonably without firefox needing to do it _for_every_click_

like laptop mode, you need to be able to define "I'm willing to loose this much activity in the name of performance/power"

ted's suggestion (in his blog) to tweak fsync to 'misbehave' when laptop mode is enabled (only pushing data out to disk when the disk is awake anyway, or the time has hit) would really work well for most users. servers (where you have the data integrity fsync useage) don't use laptop mode. desktops could use 'laptop mode' with a delay of 0.5 or 1 second and get prety close the the guarentee that users want without a huge performance hit.

David Lang

Quoted 16 lines

As an aside, I find it highly ironic that Firefox gained useful session management around the same time that some GNOME jarhead no-op'd GNOME session management[1] in X.

Jeff

[1] http://np237.livejournal.com/22014.html

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#256 lore raw
Quoted 1 line · [email protected] wrote:

ted's suggestion (in his blog) to tweak fsync to 'misbehave' when laptop mode is enabled (only pushing data out to disk when the disk is awake anyway, or the time has hit) would really work well for most users. servers (where you have the data integrity fsync useage) don't use laptop mode. desktops could use 'laptop mode' with a delay of 0.5 or 1 second and get prety close the the guarentee that users want without a huge performance hit.

The existential struggle is overall amusing:

Application writers start using userland transactional databases for crash recovery and consistency, and in response, OS writers work to undercut the consistency guarantees currently provided by the OS.

More seriously, if we get sqlite, db4 and a few others behaving sanely WRT fsync, you cover a wide swath of apps all at once.

I absolutely agree that db4, sqlite and friends need to be smarter in the case of laptop mode or overall power saving.

Jeff

maintainer
#257 lore raw
Quoted 8 lines · Jeff Garzik
Quoted 7 lines

ted's suggestion (in his blog) to tweak fsync to 'misbehave' when laptop mode is enabled (only pushing data out to disk when the disk is awake anyway, or the time has hit) would really work well for most users. servers (where you have the data integrity fsync useage) don't use laptop mode. desktops could use 'laptop mode' with a delay of 0.5 or 1 second and get prety close the the guarentee that users want without a huge performance hit.

Quoted 5 lines

The existential struggle is overall amusing:

Application writers start using userland transactional databases for crash recovery and consistency, and in response, OS writers work to undercut the consistency guarantees currently provided by the OS.

Actually, it makes a lot of sense, if you think about it in this way.

The requirement is this; by default, data which is critical shouldn't be lost. (Whether this should be done by the filesystem performing magic, or the application/database programmer being careful about using fsync --- and whether we should treat all files as critical and to hell with performance, or only those which the application has designated as precious or nonprecious --- there is some dispute.)

However, the system administrator should be able to say, "I want laptop mode functionality", and with the turn of a single dial, be able to say, "In order to save batteries, I'm OK with losing up to X seconds/minutes worth of work." I would envision a control panel GUI where there is one checkbox, "enable laptop mode", and another checkbox, "enable laptop mode only when on battery" (which is greyed out unless the first is checkbox is enabled), and then a slidebar which allows the user to set how many seconds and/or minutes the user is willing to lose if the system crashes.

At that point, it's up to the user. Maybe the defaults should be something like 15 seconds; maybe the defaults should be 5 seconds. Maybe the defaults should be automatically set to different values by different distributions, depending on whether said distro is willing to use badly unstable proprietary bindary video drivers that crash if you look at them funny.

The advantage of such a scheme is that there's a single knob for the user to control, instead one for each application. And fundamentally, it should be OK for a user of the desktop and/or the system administrator to make this tradeoff. That's where the choice belongs; not to the application writer, and not to the filesystem maintainer, or OS programmers in general.

If I have an Lenovo X61s which is rock solid stable, with Intel video drivers, I might be willing to risk lose up to 10 minutes of work, secure in the knowledge it's highly unlikely to happen. If I'm an Ubuntu user with so super-unstable proprietary video driver, maybe I'd be more comfortable with this being 5 or 10 seconds. But if we leave it up to the user, and they have an easy-to-use control panel that controls it, the user can decide for themself where they want to trade off performance, battery life, and potential window for data loss.

So having some mode where we can suspend all writes to the disk for up to a user-defined limit --- and then once the disk wakes up, for reading or for writing, we flush out all dirty data --- makes a lot of sense. Laptop mode does most of this already, except that it doesn't intercept fsync() requests. And as long as the user has given permission to the operating system to defer fsync() requests by up to some user-specified time limit, IMHO that's completely fair game.

- Ted

#258 lore raw
Quoted 7 lines · Theodore Tso

So having some mode where we can suspend all writes to the disk for up to a user-defined limit --- and then once the disk wakes up, for reading or for writing, we flush out all dirty data --- makes a lot of sense. Laptop mode does most of this already, except that it doesn't intercept fsync() requests. And as long as the user has given permission to the operating system to defer fsync() requests by up to some user-specified time limit, IMHO that's completely fair game.

Overall I agree, but I would rewrite that as: it's fair game as long as the OS doesn't undercut the deliberate write ordering performed by the userland application.

When the "laptop mode fsync plug" is uncorked, writes should not be merged across an fsync(2) barrier; otherwise it becomes impossible to build transactional databases with any consistency guarantees at all.

Jeff

#259 lore raw
In reply to Jeff Garzik
Quoted 11 lines · Jeff Garzik
Quoted 5 lines

Of course, your browsing history database is an excellent example of something you should _not_ care about that much, and where performance is a lot more important than "ooh, if the machine goes down suddenly, I need to be 100% up-to-date". Using fsync on that thing was just stupid, even

If you are doing a ton of web-based work with a bunch of tabs or windows open, you really like the post-crash restoration methods that Firefox now employs. Some users actually do want to checkpoint/restore their web work, regardless of whether it was the browser, the window system or the OS that crashed.

This is all about tradeoff.
I guess everybody can afford loosing the last 30 seconds of history (or 5mn ...).
That's not that much of lost work...

#260 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik

What is in Fedora 10 and Debian lenny's iceweasel both definitely sync to disk, as of today, according to my own tests.

Hmm. Go to "about:config" and check your "toolkit.storage.synchronous" setting.

Quoted 7 lines

It _should_ say

default integer 0

and that is what it says for me (yes, on Fedora 10).

The values are: 0 = off, 1 = normal, 2 = full.

Definitely a difference! 1 for both, here. Deb is a fresh OS install and fresh homedir, but my F10 has been through many OS and ff config upgrades over the years.

Jeff

#261 lore raw
Quoted 4 lines · Jeff Garzik

Definitely a difference! 1 for both, here. Deb is a fresh OS install and
fresh homedir, but my F10 has been through many OS and ff config upgrades over the years.

Hmm. I wonder where firefox gets its defaults then.

I can well imagine that Debian has a different firefox build, with different defaults. But if your F10 thing also is set to 1, and still shows as "default", then that's odd, considering that mine shows 0.

I have 'rpm -q firefox': firefox-3.0.7-1.fc10.x86_64.

Is yours a 32-bit one? Maybe it comes with different defaults?

And maybe firefox just has a very odd config setup and I don't understand what "default" means at all. Gene says he doesn't have that toolkit.storage.synchronous thing at all.

Linus

#262 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik

Definitely a difference! 1 for both, here. Deb is a fresh OS install and
fresh homedir, but my F10 has been through many OS and ff config upgrades over the years.

Hmm. I wonder where firefox gets its defaults then.

Quoted 7 lines

I can well imagine that Debian has a different firefox build, with different defaults. But if your F10 thing also is set to 1, and still shows as "default", then that's odd, considering that mine shows 0.

I have 'rpm -q firefox': firefox-3.0.7-1.fc10.x86_64.

Is yours a 32-bit one? Maybe it comes with different defaults?

And maybe firefox just has a very odd config setup and I don't understand what "default" means at all. Gene says he doesn't have that toolkit.storage.synchronous thing at all.

In my case the toolkit.storage.synchronous is present in both, set to 1 in Deb and bolded and set to 1 in F10 (firefox-3.0.7-1.fc10.x86_64).

The latter's bold typeface makes me think my F10 FF toolkit.storage.synchronous setting is NOT set to the F10 default -- although I have never heard of this setting, and have certainly not manually tweaked it. The only FF setting I manually tweak is cache directory.

Jeff

#263 lore raw

2009/3/28 Jeff Garzik <[email protected]>:

Quoted 17 lines
Quoted 16 lines · Linus Torvalds
Quoted 6 lines · Jeff Garzik

Definitely a difference!   1 for both, here.  Deb is a fresh OS install and
fresh homedir, but my F10 has been through many OS and ff config upgrades over
the years.

Hmm. I wonder where firefox gets its defaults then.
I can well imagine that Debian has a different firefox build, with different defaults. But if your F10 thing also is set to 1, and still shows as "default", then that's odd, considering that mine shows 0.

I have 'rpm -q firefox': firefox-3.0.7-1.fc10.x86_64.

Quoted 8 lines
Quoted 5 lines

Is yours a 32-bit one? Maybe it comes with different defaults?

And maybe firefox just has a very odd config setup and I don't understand what "default" means at all. Gene says he doesn't have that toolkit.storage.synchronous thing at all.

In my case the toolkit.storage.synchronous is present in both, set to 1 in Deb and bolded and set to 1 in F10 (firefox-3.0.7-1.fc10.x86_64).

I compiled my own firefox under gentoo, not present. Mozilla Firefox 3.0.7, Copyright (c) 1998 - 2009 mozilla.org

Quoted 14 lines

The latter's bold typeface makes me think my F10 FF toolkit.storage.synchronous setting is NOT set to the F10 default -- although I have never heard of this setting, and have certainly not manually tweaked it.  The only FF setting I manually tweak is cache directory.

       Jeff

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

#264 lore raw
In reply to Jeff Garzik
Quoted 23 lines · Jeff Garzik
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik

Definitely a difference! 1 for both, here. Deb is a fresh OS install
and fresh homedir, but my F10 has been through many OS and ff config upgrades over the years.

Hmm. I wonder where firefox gets its defaults then.

I can well imagine that Debian has a different firefox build, with different defaults. But if your F10 thing also is set to 1, and still shows as "default", then that's odd, considering that mine shows 0.

Quoted 7 lines

I have 'rpm -q firefox': firefox-3.0.7-1.fc10.x86_64.

Is yours a 32-bit one? Maybe it comes with different defaults?

And maybe firefox just has a very odd config setup and I don't understand what "default" means at all. Gene says he doesn't have that toolkit.storage.synchronous thing at all.

In my case the toolkit.storage.synchronous is present in both, set to 1 in Deb and bolded and set to 1 in F10 (firefox-3.0.7-1.fc10.x86_64).

Quoted 7 lines

The latter's bold typeface makes me think my F10 FF toolkit.storage.synchronous setting is NOT set to the F10 default -- although I have never heard of this setting, and have certainly not manually tweaked it. The only FF setting I manually tweak is cache directory.

Jeff

I just let FF update itself to 3.0.8 (from mozilla, not fedora) and there is no 'toolkit' stuff whatsoever in about:config.

Is this perchance some extension I don't have installed?

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 7 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Jacquin's Postulate on Democratic Government:
	No man's life, liberty, or property are safe while the
	legislature is in session.
#265 lore raw
Quoted 3 lines · El Fri, 27 Mar 2009 23:55:06 -0400 Gene Heskett <[email protected]>…

I just let FF update itself to 3.0.8 (from mozilla, not fedora) and there is no 'toolkit' stuff whatsoever in about:config.

I do not have it either FF 3.0.8 Ubuntu 8.10. it does not appear searching with
sync; not toolkit nor storage...

Quoted 4 lines

Is this perchance some extension I don't have installed?

Quoted 1 line
#266 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik

What is in Fedora 10 and Debian lenny's iceweasel both definitely sync to disk, as of today, according to my own tests.

Hmm. Go to "about:config" and check your "toolkit.storage.synchronous" setting.

..

Quoted 1 line

If you don't have that "toolkit.storage.synchronous" entry, that means that you have an older version of firefox-3.

..

Okay, I'll bite. Exactly which version of FF has that variable? Cuz it ain't in the FF 3.0.8 that I'm running here.

Thanks

#267 lore raw
Quoted 3 lines · Mark Lord

Okay, I'll bite. Exactly which version of FF has that variable? Cuz it ain't in the FF 3.0.8 that I'm running here.

I _thought_ it was there since rc2 of FF-3, but clearly there are odd things afoot. You're the second person to report it not there.

I'd suspect that I mistyped it, but I just cut-and-pasted it from my email to make sure. Maybe you did. What happens if you just write "sync" in the
Filter: box? Nothing matches?

Do you see firefox pausing a lot under disk load? If you just add that "toolkit.storage.synchronous" value by hand (right-click in the preference window, do "New" -> "Integer"), and write it in as zero, does it change behavior?

Linus

#268 lore raw
Quoted 7 lines · On 28.03.2009 03:38 Linus Torvalds wrote:
Quoted 3 lines · Mark Lord

Okay, I'll bite. Exactly which version of FF has that variable? Cuz it ain't in the FF 3.0.8 that I'm running here.

I'd suspect that I mistyped it, but I just cut-and-pasted it from my

email

Quoted 4 lines

to make sure. Maybe you did. What happens if you just write "sync" in the
Filter: box? Nothing matches?

No, not with my iceweasel 3.0.7 (Debian/testing).

I couldn't find anything in the Debian patch to the source code, but the source code contains toolkit/components/contentprefs/src/nsContentPrefService.js 733-746:

    // Turn off disk synchronization checking to reduce disk churn and
speed up
    // operations when prefs are changed rapidly (such as when a user
repeatedly
    // changes the value of the browser zoom setting for a site).
    //
    // Note: this could cause database corruption if the OS crashes or
machine
    // loses power before the data gets written to disk, but this is
considered
    // a reasonable risk for the not-so-critical data stored in this
database.
    //
    // If you really don't want to take this risk, however, just set the
    // toolkit.storage.synchronous pref to 1 (NORMAL synchronization) or 2
    // (FULL synchronization), in which case
mozStorageConnection::Initialize
    // will use that value, and we won't override it here.
    if (!this._prefSvc.prefHasUserValue("toolkit.storage.synchronous"))
      dbConnection.executeSimpleSQL("PRAGMA synchronous = OFF");

Probably they preferred the default value "off" so much that they even dropped the entry in standard configuration.

Quoted 1 line

Do you see firefox pausing a lot under disk load?

I see iceweasel pausing/blocking a lot when loading stalling webpages, but that's a different topic.

#269 lore raw
In reply to Matthew Garrett
Quoted 13 lines · Matthew Garrett
Quoted 11 lines · Theodore Tso
Quoted 1 line · Matthew Garrett

(It could be argued that most relevant Unices implemented fsync() even before then, so its status in POSIX was broadly irrelevant. The obvious counterargument is that most relevant Unix filesystems ensure that data is written before a clobbering rename() is carried out, so POSIX is again not especially releant)

Nope, not true. Most relevant Unix file systems sync'ed data blocks on a 30 timer, and metadata on 5 second timers. They did *not* force data to be written before a clobbering rename() was carried you; you're rewriting history when you say that; it's simply not true. Rename was atomic *only* where metadata was concerned, and all the talk about rename being atomic was because back then we didn't have flock() and you built locking primitives open(O_CREAT) and rename(); but that was only metadata, and that was only if the system didn't crash.

Quoted 1 line

No, you're missing my point. The other Unix file systems are irrelevant. The number of people running them and having any real risk of system crash is small, and they're the ones with full system backups anyway.

Are you telling us that the "Linux compatible" really means "Linux compatible, but only on ext3, only on x86, only on Ubuntu, only Gnome or KDE [1]"?
If a program crashes on other setups, is it not a problem of the program but of the environment?

sigh
cate

[1]Yes, I just see a installation script that expect one of the two environment.

#270 lore raw
In reply to Theodore Tso
Quoted 8 lines · Theodore Tso

When I was growing up we were trained to *always* check error returns from *all* system calls, and to *always* fsync() if it was critical that the data survive a crash. That was what competent Unix programmers did. And if you are always checking error returns, the difference in the Lines of Code between doing it right and doing really wasn't that big --- and again, back then fsync() wan't expensive. Making fsync expensive was ext3's data=ordered mode's fault.

This is a fairly narrow view of correct and possible. How can you make "cat" fsync? grep? sort? How do they know they're not dealing with critical data? Apps in general don't know, because "criticality" is a property of the data itself and how its used, not the tools operating on it.

My point isn't that "there should be a way of doing fsync from a shell script" (which is probably true anyway), but that authors can't generally anticipate when their program is going to be dealing with something important. The conservative approach would be to fsync all data on every close, but that's almost certainly the wrong thing for everyone.

If the filesystem has reasonably strong inherent data-preserving properties, then that's much better than scattering fsync everywhere.

fsync obviously makes sense in specific applications; it makes sense to fsync when you're guaranteeing that a database commit hits stable storage, etc. But generic tools can't reasonably perform fsyncs, and its not reasonable to say that "important data is always handled by special important data tools".

J

#271 lore raw
Quoted 4 lines · Jeremy Fitzhardinge <jeremy <at> goop.org> writes:

This is a fairly narrow view of correct and possible. How can you make "cat" fsync? grep? sort? How do they know they're not dealing with critical data? Apps in general don't know, because "criticality" is a property of the data itself and how its used, not the tools operating on it.

Isn't it possible to compile a program that simply calls open()/fsync()/close() on a given file name? If yes, then in your scripts, you can do whatever you want with existing tools on a _scratch_ file, then call your fsync program on that scratch file and then rename it to the real file. No?

In other words, given that you know that your data is critical, you will write processed data to another file, while preserving the original, store the new file safely and then rename it to the original. Just like the apps that know that their files are critical are supposed to do using the API.

--
Bojan

#272 lore raw
Quoted 2 lines · Bojan Smojver <bojan <at> rexursive.com> writes:

Isn't it possible to compile a program that simply calls open()/fsync()/close() on a given file name?

That was stupid. Ignore me.

--
Bojan

#274 lore raw
Quoted 10 lines · Bojan Smojver
Quoted 2 lines · Bojan Smojver <bojan <at> rexursive.com> writes:

That was stupid. Ignore me.

And yet, FreeBSD seems to have a command just like that:

http://www.freebsd.org/cgi/man.cgi?query=fsync&sektion=1&manpath=FreeBSD+7.1-RELEASE

I was thinking something like "munge_important_stuff | fsync > output" - ie, cat which fsyncs on close. In fact, its vaguely surprising that GNU cat doesn't have this already.

J

#275 lore raw
Quoted 2 lines · Jeremy Fitzhardinge

I was thinking something like "munge_important_stuff | fsync > output"
- ie, cat which fsyncs on close.

Yeah, after I wrote my initial comment, I noticed you were saying essentially the same thing in your original post. I know, I should _read_ before posting. Sorry :-(

Quoted 2 lines

In fact, its vaguely surprising that GNU cat doesn't have this already.

I have no idea why we don't have that either. FreeBSD code seems really straightforward.

Signature · 1 line
Bojan
#276 lore raw
Quoted 2 lines · Bojan Smojver

I have no idea why we don't have that either. FreeBSD code seems really straightforward.

I just tried using dd with conv=fsync option and that kinda does what you mentioned. I see this at the end of strace:
---------------------------------
write(1, "<some data...>"..., 512) = 512

read(0, ""..., 512)                = 0
fsync(1)                           = 0
close(0)                           = 0
close(1)                           = 0

---------------------------------

So, maybe GNU folks just don't want to have yet another tool for this.

Signature · 1 line
Bojan
#277 lore raw
Quoted 10 lines · Bojan Smojver
Quoted 3 lines · Bojan Smojver

I have no idea why we don't have that either. FreeBSD code seems really straightforward.

I just tried using dd with conv=fsync option and that kinda does what you mentioned. I see this at the end of strace:
---------------------------------
write(1, "<some data...>"..., 512) = 512

Quoted 8 lines
read(0, ""..., 512)                = 0
fsync(1)                           = 0
close(0)                           = 0
close(1)                           = 0

---------------------------------

So, maybe GNU folks just don't want to have yet another tool for this.

Huh, didn't know dd had grown that. Confusingly similar to the completely different conv=sync, so its a perfect dd addition. Ooh, fdatasync too.

J

#278 lore raw
In reply to Theodore Tso

Hi!

Quoted 9 lines
Quoted 1 line

I'm utterly and screamingly bored of this "Blame userspace" attitude.

I'm not blaming userspace. I'm blaming ourselves, for implementing an attractive nuisance, and not realizing that we had implemented an attractive nuisance; which years later, is also responsible for these latency problems, both with and without fsync() ---- *and* which have also traied people into believing that fsync() is always expensive, and must be avoided at all costs --- which had not previously been true!

Well... fsync is quite expensive. If your disk is down, it costs 3+ and 3J+. If your disk is up, it will only take 20msec+.

OTOH the rename trick on ext3 costs approximately nothing...

Imagine those desktops where they want windows layout preserved. Having 30 second old layout is acceptable, loosing layout altogether is not. If you add fsync to the window manager, user will see those 3seconds+ delays, unless window manager gets multithreaded.

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#279 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

The only times tunables have worked for us is when they auto-tune.

IOW, we don't have "use 35% of memory for buffer cache" tunables, we just dynamically auto-tune memory use. And no, we don't expect user space to run some "tuning program for their load" either.

IOW, what we could reasonably do is something along the lines of:

- start off with some reasonable value for max background dirty (per block device) that defaults to something sane (quite possibly based on simply memory size).

- assume that "foreground dirty" is just always 2* background dirty.

- if we hit the "max foreground dirty" during memory allocation, then we shrink the background dirty value (logic: we never want to have to wait synchronously)

- if we hit some maximum latency on writeback, shrink dirty aggressively and based on how long the latency was (because at that point we have a real _measure_ of how costly it is with that load).

- if we start doing background dirtying, but never hit the foreground dirty even in dirty balancing (ie when a writer is actually _writing_, as opposed to hitting it when allocating memory by a non-writer), then slowly open up the window - we may be limiting too early.

.. add heuristics to taste. The point being, that if we do this based on real loads, and based on hitting the real problems, then we might actually be getting somewhere. In particular, if the filesystem sucks at writeout (ie the limiter is not the _disk_, but the filesystem serialization), then it should automatically also shrink the max dirty state.

The tunable then could become the maximum latency we accept or something like that. Or the hysteresis limits/rules for the soft "grow" or "shrink" events. At that point, maybe we could even find something that works for most people.

Linus

#280 lore raw
Quoted 37 lines · On Thu, 26 Mar 2009 17:51:44 -0700 (PDT) Linus Torvalds…
Quoted 6 lines · Linus Torvalds

The only times tunables have worked for us is when they auto-tune.

IOW, we don't have "use 35% of memory for buffer cache" tunables, we just dynamically auto-tune memory use. And no, we don't expect user space to run some "tuning program for their load" either.

IOW, what we could reasonably do is something along the lines of:

- start off with some reasonable value for max background dirty (per block device) that defaults to something sane (quite possibly based on simply memory size).

- assume that "foreground dirty" is just always 2* background dirty.

- if we hit the "max foreground dirty" during memory allocation, then we shrink the background dirty value (logic: we never want to have to wait synchronously)

- if we hit some maximum latency on writeback, shrink dirty aggressively and based on how long the latency was (because at that point we have a real _measure_ of how costly it is with that load).

- if we start doing background dirtying, but never hit the foreground dirty even in dirty balancing (ie when a writer is actually _writing_, as opposed to hitting it when allocating memory by a non-writer), then slowly open up the window - we may be limiting too early.

.. add heuristics to taste. The point being, that if we do this based on real loads, and based on hitting the real problems, then we might actually be getting somewhere. In particular, if the filesystem sucks at writeout (ie the limiter is not the _disk_, but the filesystem serialization), then it should automatically also shrink the max dirty state.

Quoted 5 lines

The tunable then could become the maximum latency we accept or something like that. Or the hysteresis limits/rules for the soft "grow" or "shrink" events. At that point, maybe we could even find something that works for most people.

hm.

It may not be too hard to account for seekiness. Simplest case: if we dirty a page and that page is file-contiguous to another already dirty page then don't increment the dirty page count by "1": increment it by 0.01.

Another simple case would be to keep track of the _number_ of dirty inodes rather than simply lumping all dirty pages together.

And then there's metadata. The dirty balancing code doesn't account for dirty inodes _at all_ at present.

(Many years ago there was a bug wherein we could have zillions of dirty inodes and exactly zero dirty pages, and the writeback code wouldn't trigger at all - the inodes would just sit there until a page got dirtied - this might still be there).

Then again, perhaps we don't need all those discrete heuristic things. Maybe it can all be done in mark_buffer_dirty(). Do some clever math+data-structure to track the seekiness of our dirtiness. Delayed allocation would mess that up though.

#281 lore raw
In reply to Andrew Morton
Quoted 2 lines

userspace can do it quite easily. Run a self-tuning script after installation and when the disk hardware changes significantly.

Which is "all the time" in some configurations. It really needs to be self tuning internally based on the observed achieved rates (just as you don't use a script to tune your network bandwidth each day)

#282 lore raw
In reply to Theodore Tso
Quoted 14 lines · On Wednesday March 25, [email protected] wrote:
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Theodore Tso

I'm beginning to think that using a "ratio" may be the wrong way to go. We probably need to add an optional dirty_max_megabytes field where we start pushing dirty blocks out when the number of dirty blocks exceeds either the dirty_ratio or the dirty_max_megabytes, which ever comes first.

Quoted 5 lines

We have that. Except it's called "dirty_bytes" and "dirty_background_bytes", and it defaults to zero (off).

The problem being that unlike the ratio, there's no sane default value that you can at least argue is not _entirely_ pointless.

Quoted 9 lines

Well, if the maximum time that someone wants to wait for an fsync() to return is one second, and the RAID array can write 100MB/sec, then setting a value of 100MB makes a certain amount of sense. Yes, this doesn't take seek overheads into account, and it may be that we're not writing things out in an optimal order, as Alan as pointed out. But 100MB is much lower number than 5% of 32GB (1.6GB). It would be better if these numbers were accounted on a per-filesystem instead of a global threshold, but for people who are complaining about huge latencies, it at least a partial workaround that they can use today.

We do a lot of dirty accounting on a per-backing_device basis. This was added to stop slow devices from sucking up too much for the "40% dirty" space. The allowable dirty space is now shared among all devices in rough proportion to how quickly they write data out.

My memory of how it works isn't perfect, but we count write-out completions both globally and per-bdi and maintain a fraction:

      my-writeout-completions
    --------------------------
    total-writeout-completions

That device then gets a share of the available dirty space based on the fraction.

The counts decay some-how so that the fraction represents recent activity.

I shouldn't be too hard to add some concept of total time to this. If we track the number of write-outs per unit time and use that together with a "target time for fsync" to scale the 'dirty_bytes' number, we might be able to auto-tune the amount of dirty space to fit the speeds of the drives.

We would probably start with each device having a very low "max dirty" number which would cause writeouts to start soon. Once the device demonstrates that it can do n-per-second (or whatever) the VM would allow the "max dirty" number to drift upwards. I'm not sure how best to get it to move downwards if the device slows down (or the kernel over-estimated). Maybe it should regularly decay so that the device keeps have to "prove" itself.

We would still leave the "dirty_ratio" as an upper-limit because we don't want all of memory to be dirty (and 40% still sounds about right). But we would not have a time-based value to set a more realistic limit when there is enough memory to keep the devices busy for multiple minutes.

Sorry, no code yet. But I think the idea is sound.

NeilBrown

maintainer
#283 lore raw
Quoted 6 lines · Neil Brown

I shouldn't be too hard to add some concept of total time to this. If we track the number of write-outs per unit time and use that together with a "target time for fsync" to scale the 'dirty_bytes' number, we might be able to auto-tune the amount of dirty space to fit the speeds of the drives.

Quoted 7 lines

We would probably start with each device having a very low "max dirty" number which would cause writeouts to start soon. Once the device demonstrates that it can do n-per-second (or whatever) the VM would allow the "max dirty" number to drift upwards. I'm not sure how best to get it to move downwards if the device slows down (or the kernel over-estimated). Maybe it should regularly decay so that the device keeps have to "prove" itself.

This seems like a really cool idea.

-Ted

#284 lore raw
In reply to David Rees
Quoted 5 lines

posted to the list earlier trying to see if there was anything that could be done to help my specific case. I've got a system where if someone starts writing out a large file, it kills client NFS writes. Makes the system unusable:
http://marc.info/?l=linux-kernel&m=123732127919368&w=2

I have not had this problem since I applied Arjan's (for some reason repeatedly rejected) patch to change the ioprio of the various writeback daemons. Under some loads changing to the noop I/O scheduler also seems to help (as do most of the non default ones)

Quoted 3 lines

Everyone seems to agree that "autotuning" it is the way to go. But no one seems willing to step up and try to do it. Probably because it's hard to get right!

If this is a VM problem why does fixing the I/O priority of the various daemons seem to cure at least some of it ?

Alan

#285 lore raw
Quoted 10 lines · Alan Cox
Quoted 5 lines

posted to the list earlier trying to see if there was anything that could be done to help my specific case. I've got a system where if someone starts writing out a large file, it kills client NFS writes. Makes the system unusable:
http://marc.info/?l=linux-kernel&m=123732127919368&w=2

I have not had this problem since I applied Arjan's (for some reason repeatedly rejected) patch to change the ioprio of the various writeback daemons. Under some loads changing to the noop I/O scheduler also seems to help (as do most of the non default ones)

(link would be useful)

Ingo

#286 lore raw
Quoted 6 lines
Quoted 4 lines

I have not had this problem since I applied Arjan's (for some reason repeatedly rejected) patch to change the ioprio of the various writeback daemons. Under some loads changing to the noop I/O scheduler also seems to help (as do most of the non default ones)

(link would be useful)

"Give kjournald a IOPRIO_CLASS_RT io priority"

October 2007 (yes its that old)

And do the same as per discussion to the writeback tasks.

Which isn't to say there are not also vm problems - look at the I/O patterns with any kernel after about 2.6.18/19 and there seems to be a serious problem with writeback from the mm and fs writes falling over each other and turning the smooth writeout into thrashing back and forth as both try to write out different bits of the same stuff.

<Rant>
Really someone needs to sit down and actually build a proper model of the VM behaviour in a tool like netlogo rather than continually keep adding ever more complex and thus unpredictable hacks to it. That way we might better understand what is occurring and why.
</Rant>

Alan

#287 lore raw
Quoted 11 lines · Alan Cox
Quoted 6 lines
Quoted 4 lines

I have not had this problem since I applied Arjan's (for some reason repeatedly rejected) patch to change the ioprio of the various writeback daemons. Under some loads changing to the noop I/O scheduler also seems to help (as do most of the non default ones)

(link would be useful)

"Give kjournald a IOPRIO_CLASS_RT io priority"

October 2007 (yes its that old)

thx. A more recent submission from Arjan would be:

    http://lkml.org/lkml/2008/10/1/405

Resolution was that Tytso indicated it went into some sort of ext4 patch queue:

| I've ported the patch to the ext4 filesystem, and dropped it into 
| the unstable portion of the ext4 patch queue.
|
|   ext4: akpm's locking hack to fix locking delays

but 6 months down the line and i can find no trace of this upstream anywhere.

<let-me-rant-too>

The thing is ... this is a _bad_ ext3 design bug affecting ext3 users in the last decade or so of ext3 existence. Why is this issue not handled with the utmost high priority and why wasnt it fixed 5 years ago already? :-)

It does not matter whether we have extents or htrees when there are _trivially reproducible_ basic usability problems with ext3.

Ingo

#288 lore raw
Quoted 32 lines · On Tue, 24 Mar 2009 11:31:11 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 11 lines · Alan Cox
Quoted 6 lines
Quoted 4 lines

I have not had this problem since I applied Arjan's (for some reason repeatedly rejected) patch to change the ioprio of the various writeback daemons. Under some loads changing to the noop I/O scheduler also seems to help (as do most of the non default ones)

(link would be useful)

"Give kjournald a IOPRIO_CLASS_RT io priority"

October 2007 (yes its that old)

thx. A more recent submission from Arjan would be:

    http://lkml.org/lkml/2008/10/1/405

Resolution was that Tytso indicated it went into some sort of ext4 patch queue:

| I've ported the patch to the ext4 filesystem, and dropped it into 
| the unstable portion of the ext4 patch queue.
|
|   ext4: akpm's locking hack to fix locking delays

but 6 months down the line and i can find no trace of this upstream anywhere.

<let-me-rant-too>

Quoted 8 lines

The thing is ... this is a _bad_ ext3 design bug affecting ext3 users in the last decade or so of ext3 existence. Why is this issue not handled with the utmost high priority and why wasnt it fixed 5 years ago already? :-)

It does not matter whether we have extents or htrees when there are _trivially reproducible_ basic usability problems with ext3.

It's all there in that Oct 2008 thread.

The proposed tweak to kjournald is a bad fix - partly because it will elevate the priority of vast amounts of IO whose priority we don't _want_ elevated.

But mainly because the problem lies elsewhere - in an area of contention between the committing and running transactions which we knowingly and reluctantly added to fix a bug in

commit 773fc4c63442fbd8237b4805627f6906143204a8

Author:     akpm <akpm>
AuthorDate: Sun May 19 23:23:01 2002 +0000
Commit:     akpm <akpm>
CommitDate: Sun May 19 23:23:01 2002 +0000

[PATCH] fix ext3 buffer-stealing

Patch from sct fixes a long-standing (I did it!) and rather complex problem with ext3.

The problem is to do with buffers which are continually being dirtied by an external agent. I had code in there (for easily-triggerable livelock avoidance) which steals the buffer from checkpoint mode and reattaches it to the running transaction. This violates ext3 ordering requirements - it can permit journal space to be reclaimed before the relevant data has really been written out.

Also, we do have to reliably get a lock on the buffer when moving it between lists and inspecting its internal state. Otherwise a competing read from the underlying block device can trigger an assertion failure, and a competing write to the underlying block device can confuse ext3 journalling state completely.

Now this:

Quoted 2 lines

Resolution was that Tytso indicated it went into some sort of ext4 patch queue:

was not a fix at all. It was a known-buggy hack which I proposed simply to remove that contention point to let us find out if we're on the right track. IIRC Ric was going to ask someone to do some performance testing of that hack, but we never heard back.

The bottom line is that someone needs to do some serious rooting through the very heart of JBD transaction logic and nobody has yet put their hand up. If we do that, and it turns out to be just too hard to fix then yes, perhaps that's the time to start looking at palliative bandaids.

The number of people who can be looked at to do serious ext3/JBD work is pretty small now. Ted, Stephen and I got old and died. Jan does good work but is spread thinly.

#289 lore raw
Quoted 3 lines

The proposed tweak to kjournald is a bad fix - partly because it will elevate the priority of vast amounts of IO whose priority we don't _want_ elevated.

Its a huge improvement in practice because it both fixes the stupid stalls and smooths out the rest of the I/O traffic. I spend a lot of my time looking at what the disk driver is getting fed and its not a good mix. Even more revealing is the noop scheduler and the fact this frequently outperforms all the fancy I/O scheduling we do even on relatively dumb hardware (as well as showing how mixed up our I/O patterns currently are).

Quoted 3 lines

But mainly because the problem lies elsewhere - in an area of contention between the committing and running transactions which we knowingly and reluctantly added to fix a bug in

The problem emerges about 2007 not 2002, so its not that simple.

Quoted 3 lines

The number of people who can be looked at to do serious ext3/JBD work is pretty small now. Ted, Stephen and I got old and died. Jan does good work but is spread thinly.

Which is all the more reason to use a temporary fix in the meantime so the OS is usable. I think its pretty poor that for over a year those in the know who need a good performing system are having to apply out of tree trivial patches rejected on the basis that "eventually like maybe whenever perhaps we'll possibly some day you know consider fixing this, but don't hold your breath"

There is a second reason to do this: If ext4 is the future then it is far better to fix this stuff in ext4 properly and leave ext3 clear of extremely invasive high risk fixes when a quick bandaid will do just fine for the remaining lifetime of fs/jbd

Also not kjournald is only one of the afflicted threads - the same is true of the crypto, and of the vm writeback. Also note the other point about the disk scheduler defaults being terrible for some streaming I/O patterns and the patch for that is also stuck in bugzilla.

If picking "no-op" speeds up my generic x86 box with random onboard SATA we are doing something very non-optimal

maintainer
#290 lore raw
In reply to Andrew Morton
Quoted 3 lines · Andrew Morton

But mainly because the problem lies elsewhere - in an area of contention between the committing and running transactions which we knowingly and reluctantly added to fix a bug in "[PATCH] fix ext3 buffer-stealing"

Well, let's be clear here. The contention between committing and running transaction is an issue, even if we solved this problem, it wouldn't solve the issue of fsync() taking a long time in ext3's data=ordered mode in the case of massive write starvation caused by a read-heavy workload, or a vast number of dirty buffers associated with an inode which is about to be committed, and a process triggers an fsync(). So fixing this issue wouldn't have solved the problem which Ingo complained about (which was an editor calling fsync() leading to long delay when saving a file during or right after a distcc-accelerated kernel compile) or the infamous Firefox 3.0 bug.

Fixing this contention *would* fix the problem where a normal process which is doing normal file I/O could end up getting stalled unnecessarily, but that's not what most people are complaining about --- and shortening the amount of time that it takes do a commit (either with ext4's delayed allocation or ext3's data=writeback mount option) would also address this problem. That doesn't mean that it's not worth it to fix this particular contention, but there are multiple issues going on here.

(Basically we're here:

http://www.kernel.org/pub/linux/kernel/people/paulmck/Confessions/FOSSElephant.html

... in Paul Mckenney's version of parable of the blind men and the elephant:

http://www.kernel.org/pub/linux/kernel/people/paulmck/Confessions/

:-)

Quoted 9 lines

Now this:

Quoted 2 lines

Resolution was that Tytso indicated it went into some sort of ext4 patch queue:

was not a fix at all. It was a known-buggy hack which I proposed simply to remove that contention point to let us find out if we're on the right track. IIRC Ric was going to ask someone to do some performance testing of that hack, but we never heard back.

Ric did do some preliminary performance testing, and it wasn't encouraging. It's still in the unstable portion of the ext4 patch queue, and it's in my "wish I had more time to look at it; I don't get to work on ext3/4 full-time" queue.

Quoted 4 lines

The bottom line is that someone needs to do some serious rooting through the very heart of JBD transaction logic and nobody has yet put their hand up. If we do that, and it turns out to be just too hard to fix then yes, perhaps that's the time to start looking at palliative bandaids.

I disagree that they are _just_ palliative bandaids, because you need these in order to make sure fsync() completes in a reasonable time, so
that people like Ingo don't get cranky. :-) Fixing the contention
between the running and committing transaction is a good thing, and I hope someone puts up their hand or I magically get the time I need to really dive into the jbd layer, but it won't help the Firefox 3.0 problem or Ingo's problem with saving files during a distcc run.

- Ted

reviewer
#291 lore raw
In reply to Andrew Morton
Quoted 38 lines · Andrew Morton
Quoted 8 lines · On Tue, 24 Mar 2009 11:31:11 +0100 Ingo Molnar <[email protected]> wrote:

The thing is ... this is a _bad_ ext3 design bug affecting ext3 users in the last decade or so of ext3 existence. Why is this issue not handled with the utmost high priority and why wasnt it fixed 5 years ago already? :-)

It does not matter whether we have extents or htrees when there are _trivially reproducible_ basic usability problems with ext3.

It's all there in that Oct 2008 thread.

The proposed tweak to kjournald is a bad fix - partly because it will elevate the priority of vast amounts of IO whose priority we don't _want_ elevated.

But mainly because the problem lies elsewhere - in an area of contention between the committing and running transactions which we knowingly and reluctantly added to fix a bug in

commit 773fc4c63442fbd8237b4805627f6906143204a8

Author:     akpm <akpm>
AuthorDate: Sun May 19 23:23:01 2002 +0000
Commit:     akpm <akpm>
CommitDate: Sun May 19 23:23:01 2002 +0000

[PATCH] fix ext3 buffer-stealing

Patch from sct fixes a long-standing (I did it!) and rather complex problem with ext3.

The problem is to do with buffers which are continually being dirtied by an external agent. I had code in there (for easily-triggerable livelock avoidance) which steals the buffer from checkpoint mode and reattaches it to the running transaction. This violates ext3 ordering requirements - it can permit journal space to be reclaimed before the relevant data has really been written out.

Quoted 5 lines

Also, we do have to reliably get a lock on the buffer when moving it between lists and inspecting its internal state. Otherwise a competing read from the underlying block device can trigger an assertion failure, and a competing write to the underlying block device can confuse ext3 journalling state completely.

I've looked at this a bit. I suppose you mean the contention arising from
us taking the buffer lock in do_get_write_access()? But it's not obvious to me why we'd be contending there... We call this function only for metadata buffers (unless in data=journal mode) so there isn't huge amount of these blocks. This buffer should be locked for a longer time only when we do writeout for checkpoint (hmm, maybe you meant this one?). In particular, note that we don't take the buffer lock when committing this block to journal - we lock only the BJ_IO buffer. But in this case we wait when the buffer is on BJ_Shadow list later so there is some contention in this case.
Also when I emailed with a few people about these sync problems, they
wrote that switching to data=writeback mode helps considerably so this would indicate that handling of ordered mode data buffers is causing most of the slowdown...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#292 lore raw
Quoted 11 lines · Jan Kara
Quoted 5 lines

Also, we do have to reliably get a lock on the buffer when moving it between lists and inspecting its internal state. Otherwise a competing read from the underlying block device can trigger an assertion failure, and a competing write to the underlying block device can confuse ext3 journalling state completely.

I've looked at this a bit. I suppose you mean the contention arising from
us taking the buffer lock in do_get_write_access()? But it's not obvious to me why we'd be contending there... We call this function only for metadata buffers (unless in data=journal mode) so there isn't huge amount of these blocks.

There isn't a huge number of those blocks, but if inode #1220 was modified in the previous transaction which is now being committed, and we then need to modify and write out inode #1221 in the current contention, and they share the same inode table block, that would cause the contention. That probably doesn't happen that often in a synchronous code path, but it probably happens more often that you're thinking. I still think the fsync() problem is the much bigger deal, and solving the contention problem isn't going to solve the fsync() latency problem with ext3 data=ordered mode.

Quoted 4 lines

Also when I emailed with a few people about these sync problems, they
wrote that switching to data=writeback mode helps considerably so this would indicate that handling of ordered mode data buffers is causing most of the slowdown...

Yes, but we need to be clear whether this was an fsync() problem or some other random delay problem. If it's the fsync() problem, obviously data=writeback will solve the fsync() latency delay problem. (As will using delayed allocation in ext4 or XFS.)

- Ted

#293 lore raw
Quoted 4 lines · Theodore Tso

I still think the fsync() problem is the much bigger deal, and solving the contention problem isn't going to solve the fsync() latency problem with ext3 data=ordered mode.

The fsync() problem is really annoying, but what is doubly annoying is that sometimes one process doing fsync() (or sync) seems to cause other processes to hickup too.

Now, I personally solved that problem by moving to (good) SSD's on my desktop, and I think that's indeed the long-term solution. But it would be good to try to figure out a solution in the short term for people who don't have new hardware thrown at them from random companies too.

I suspect it's a combination of filesystem transaction locking, together with the VM wanting to write out some unrelated blocks or inodes due to the system just being close to the dirty limits. Which is why the system-wide hickups then happen especially when writing big files.

The VM _tries_ to do writes in the background, but if the writepage() path hits a filesystem-level blocking lock, that background write suddenly becomes largely synchronous.

I suspect there is also some possibility of confusion with inter-file (false) metadata dependencies. If a filesystem were to think that the file size is metadata that should be journaled (in a single journal), and the journaling code then decides that it needs to do those meta-data updates in the correct order (ie the big file write _before_ the file write that wants to be fsync'ed), then the fsync() will be delayed by a totally irrelevant large file having to have its data written out (due to data=ordered or whatever).

I'd like to think that no filesystem designer would ever be that silly, but I'm too scared to try to actually go and check. Because I could well imagine that somebody really thought that "size" is metadata.

Linus

#294 lore raw
Quoted 3 lines

The fsync() problem is really annoying, but what is doubly annoying is that sometimes one process doing fsync() (or sync) seems to cause other processes to hickup too.

Bug #5942 (interaction with anticipatory io scheduler)
Bug #9546 (with reproducer & logs)
Bug #9911 including a rather natty tester (albeit in java) Bug #7372 (some info and figures on certain revs it seemed to get worse) Bug #12309 (more info, including kjournald hack fix using ioprio)

General consensus seems to be 2.6.18 is where the manure intersected with the air impeller

#295 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Theodore Tso

I still think the fsync() problem is the much bigger deal, and solving the contention problem isn't going to solve the fsync() latency problem with ext3 data=ordered mode.

Quoted 8 lines

The fsync() problem is really annoying, but what is doubly annoying is that sometimes one process doing fsync() (or sync) seems to cause other processes to hickup too.

Now, I personally solved that problem by moving to (good) SSD's on my desktop, and I think that's indeed the long-term solution. But it would be good to try to figure out a solution in the short term for people who don't have new hardware thrown at them from random companies too.

Throwing SSDs at it only increases the limit before which it becomes an issue. They hide the underlying issue and are only a workaround. Create enough dirty data and you'll get the same latencies, it's just that that limit is now a lot higher. Your Intel SSD will write streaming data 2-4 times faster than your typical disk - and can be an order of magnitude faster when it comes to small, random writes.

Quoted 9 lines

I suspect it's a combination of filesystem transaction locking, together with the VM wanting to write out some unrelated blocks or inodes due to the system just being close to the dirty limits. Which is why the system-wide hickups then happen especially when writing big files.

The VM _tries_ to do writes in the background, but if the writepage() path hits a filesystem-level blocking lock, that background write suddenly becomes largely synchronous.

Quoted 8 lines

I suspect there is also some possibility of confusion with inter-file (false) metadata dependencies. If a filesystem were to think that the file size is metadata that should be journaled (in a single journal), and the journaling code then decides that it needs to do those meta-data updates in the correct order (ie the big file write _before_ the file write that wants to be fsync'ed), then the fsync() will be delayed by a totally irrelevant large file having to have its data written out (due to data=ordered or whatever).

It certainly "feels" like that is the case from the workloads I have that generate high latencies.

-Dave

#296 lore raw
Quoted 3 lines · David Rees

Your Intel SSD will write streaming data 2-4 times faster than your typical disk

Don't even bother with streaming data. The problem is _never_ streaming data.

Even a suck-ass laptop drive can write streaming data fast enough that people don't care. The problem is invariably that writes from different sources (much of it being metadata) interact and cause seeking.

Quoted 2 lines

and can be an order of magnitude faster when it comes to small, random writes.

Umm. More like two orders of magnitude or more.

Random writes on a disk (even a fast one) tends to be in the hundreds of kilobytes per second. Have you worked with an Intel SSD? It does tens of MB/s on pure random writes.

The problem really is gone with an SSD.

And please realize that the problem for me was never 30-second stalls. For me, a 3-second stall is unacceptable. It's just very annoying.

Linus

#297 lore raw
Quoted 4 lines · Linus Torvalds

Even a suck-ass laptop drive can write streaming data fast enough that people don't care. The problem is invariably that writes from different sources (much of it being metadata) interact and cause seeking.

Actually, not just writes.

The IO priority thing is almost certainly that _reads_ (which get higher priority by default due to being synchronous) get interspersed with the writes, and then even if you _could_ be having streaming writes, what you actually end up with is lots of seeking.

Again, good SSD's don't care. Disks do. It doesn't matter if you have a FC disk array that can eat 300MB/s when streaming - once you start seeking, that 300MB/s goes down like a rock. Battery-protected write caches will help - but not a whole lot when streaming more data than they have RAM. Basic queuing theory.

Linus

#298 lore raw
Quoted 5 lines

Again, good SSD's don't care. Disks do. It doesn't matter if you have a FC disk array that can eat 300MB/s when streaming - once you start seeking, that 300MB/s goes down like a rock. Battery-protected write caches will help - but not a whole lot when streaming more data than they have RAM. Basic queuing theory.

Subtly more complex than that. If your mashed up I/O streams fit into the 2GB or so of cache (minus one stream to disk) you win. You also win because you take a lot of fragmented OS I/O and turn it into bigger chunks of writing better scheduled. The latter win arguably shouldn't happen but it does occur (I guess in part that says we suck) and it occurs big time when you've got multiple accessors to a shared storage system (where the host OS's can't help)

Alan

#299 lore raw
Quoted 11 lines · Alan Cox
Quoted 2 lines

Again, good SSD's don't care. Disks do. It doesn't matter if you have a FC disk array that can eat 300MB/s when streaming - once you start seeking, that 300MB/s goes down like a rock. Battery-protected write caches will help - but not a whole lot when streaming more data than they have RAM. Basic queuing theory.

Subtly more complex than that. If your mashed up I/O streams fit into the 2GB or so of cache (minus one stream to disk) you win. You also win because you take a lot of fragmented OS I/O and turn it into bigger chunks of writing better scheduled. The latter win arguably shouldn't happen but it does occur (I guess in part that says we suck) and it occurs big time when you've got multiple accessors to a shared storage system (where the host OS's can't help)

Quoted 2 lines

Alan

The other thing that can impact random writes on arrays is their internal "track" size - if the random write is of a partial track, it forces a read-modify-write with a back end disk read. Some arrays have large internal tracks, others have smaller ones.

Again, not unlike what you see with some SSD's and their erase block size - give them even multiples of that and they are quite happy.

Ric

#300 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds
Quoted 2 lines · Linus Torvalds

Even a suck-ass laptop drive can write streaming data fast enough that people don't care. The problem is invariably that writes from different sources (much of it being metadata) interact and cause seeking.

Quoted 8 lines

Actually, not just writes.

The IO priority thing is almost certainly that _reads_ (which get higher priority by default due to being synchronous) get interspersed with the writes, and then even if you _could_ be having streaming writes, what you actually end up with is lots of seeking.

Again, good SSD's don't care. Disks do. It doesn't matter if you have a FC disk array that can eat 300MB/s when streaming - once you start seeking, that 300MB/s goes down like a rock. Battery-protected write caches will help - but not a whole lot when streaming more data than they have RAM. Basic queuing theory.

Linus

This is actually not really true - random writes to an enterprise disk array will make your Intel SSD look slow. Effectively, they are extremely large, battery backed banks of DRAM with lots of fibre channel ports. Some of the bigger ones can have several hundred GB of DRAM and dozens of fibre channel ports to feed them.

Of course, if your random writes exceed the cache capacity and you fall back to their internal disks (SSD or traditional), your random write speed will drop.

Ric

maintainer
#301 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds

I suspect there is also some possibility of confusion with inter-file (false) metadata dependencies. If a filesystem were to think that the file size is metadata that should be journaled (in a single journal), and the journaling code then decides that it needs to do those meta-data updates in the correct order (ie the big file write _before_ the file write that wants to be fsync'ed), then the fsync() will be delayed by a totally irrelevant large file having to have its data written out (due to data=ordered or whatever).

It's not just the file size; it's the block allocation decisions. Ext3 doesn't have delayed allocation, so as soon as you issue the write, we have to allocate the block, which means grabbing blocks and making changes to the block bitmap, and then updating the inode with those block allocation decisions. It's a lot more than just i_size. And the problem is that if we do this for the big file write, and the small file write happens to also touch the same inode table block and/or block allocation bitmap, when we fsync() the small file, when we end up pushing out the metadata updates associated with the big file write, and so thus we need to flush out the data blocks associated with the big file write as well.

Now, there are three ways of solving this problem. One is to use delayed allocation, where we don't make the block allocation decisions until the very last minute. This is what ext4 and XFS does. The problem with this is that when we have unrelated filesystem operations that end up causing zero length files before the file write (i.e., replace-via-truncate, where the application does open/truncate/write/ close) or the after the file write (i.e., replace-via-rename, where the application does open/write/close/rename) and the application omits the fsync(). So with ext4 we has workarounds that start pushing out the data blocks in the for replace-via-rename and replace-via-truncate cases, while XFS will do an implied fsync for replace-via-truncate only, and btrfs will do an implied fsync for replace-via-rename only.

The second solution is we could add a huge amount of machinery to try track these logical dependencies, and then be able to "back out" the changes to the inode table or block allocation bitmap for the big file write when we want to fsync out the small file. This is roughly what the BSD Soft Updates mechanisms does, and it works, but at the cost of a *huge* amount of complexity. The amount of accounting data you have to track so that you can partially back out various filesystem operations, and then the state tables that make use of this accounting data is not trivial. One of the downsides of this mechanism is that it makes it extremely difficult to add new features/functionality such as extended attributes or ACL's, since very few people understand the complexities needed to support it. As a result Linux had acl and xattr support long before Kirk McKusick got around to adding those features in UFS2.

The third potential solution we can try doing is to make some tuning adjustments to the VM so that we start pushing out these data blocks much more aggressively out to the disk. If we assume that many applications aren't going to be using fsync, and we need to worry about all sorts of implied dependencies where a small file gets pushed out to disk, but a large file does not, you can have endless amounts of fun in terms of "application level file corruption", which is simply caused by the fact that a small file has been pushed out to disk, and a large file hasn't been pushed out to disk yet. If it's going to be considered fair game that application programmers aren't going to be required to use fsync() when they need to depend on something being on stable storage after a crash, then we need to tune the VM to much more aggressively clean dirty pages. Even if we remove the false dependencies at the filesystem level (i.e., fsck-detectable consistency problems), there is no way for the filesystem to be able to guess about implied dependencies between different files at the application level.

Traditionally, the way applications told us about such dependencies was fsync(). But if application programmers are demanding that fsync() is no longer required for correct operation after a filesystem crash, all we can do is push things out to disk much more aggressively.

- Ted

#302 lore raw
Quoted 5 lines · Theodore Tso

omits the fsync(). So with ext4 we has workarounds that start pushing out the data blocks in the for replace-via-rename and replace-via-truncate cases, while XFS will do an implied fsync for replace-via-truncate only, and btrfs will do an implied fsync for replace-via-rename only.

The XFS one and the ext4 one that I saw only start an _asynchronous_ writeout. Which is not an implied fsync but snake oil to make the most common complaints go away without providing hard guarantees.

IFF we want to go down this route we should better provide strong guranteed semantics and document the propery. And of course implement it consistently on all native filesystems.

Quoted 5 lines

Traditionally, the way applications told us about such dependencies was fsync(). But if application programmers are demanding that fsync() is no longer required for correct operation after a filesystem crash, all we can do is push things out to disk much more aggressively.

Note that the rename for atomic commits trick originated in mail severs which always did the proper fsync. When the word spread into the desktop world it looks like this wisdom got lost.

maintainer
#303 lore raw
Quoted 10 lines · Christoph Hellwig
Quoted 5 lines · Theodore Tso

omits the fsync(). So with ext4 we has workarounds that start pushing out the data blocks in the for replace-via-rename and replace-via-truncate cases, while XFS will do an implied fsync for replace-via-truncate only, and btrfs will do an implied fsync for replace-via-rename only.

The XFS one and the ext4 one that I saw only start an _asynchronous_ writeout. Which is not an implied fsync but snake oil to make the most common complaints go away without providing hard guarantees.

It actually does the right thing for ext4, because once we allocate the blocks, the default data=ordered mode means that we flush the datablocks before we execute the commit. Hence, in the case of open/write/close/rename, the rename will trigger an async writeout, but before the commit block is actually written, we'll have flushed out the data blocks.

I was under the impression that XFS was doing a synchronous fsync before allowing the close() return, but all it is triggering an async writeout, then yes, your concern is correct. The bigger problem from my perspective is that XFS is only doing this for the truncate case, and (from what I've been told) not for the rename case. The truncate is fundamentally racy and application writers that don't do this definitely don't deserve our solicitude, IMHO. But people who do open/write/close/rename, and omit the fsync before the rename, are at least somewhat more deserving for some kind of workaround than the idiots that do open/truncate/write/close.

Quoted 3 lines

IFF we want to go down this route we should better provide strong guranteed semantics and document the propery. And of course implement it consistently on all native filesystems.

That's something we should talk about at LSF. I'm not all that eager (or happy) about doing this, but I think that, given that the application writers massively outnumber us, we are going to be bullied into it.

Quoted 3 lines

Note that the rename for atomic commits trick originated in mail severs which always did the proper fsync. When the word spread into the desktop world it looks like this wisdom got lost.

Yep, agreed.

To be fair, though, one problem which Matthew Garrett has pointed out is that if lots of applications issue fsync(), it will have the tendency to wake up the hard drive a lot, and do a real number on power utilization. I believe the right solution for this is an extension to laptop mode which synchronizes the filesystem at a clean point, and then which suppresses fsync()'s until the hard drive wakes up, at which point it should flush all dirty data to the drive, and then freezes writes to the disk again. Presumably that should be OK, because who are using laptop mode are inherently trading off a certain amount of safety for power savings; but then other people who want to run a mysql server on a laptop get cranky, and then if we start implementing ways that applications can exempt themselves from the fsync() suppression, the complexity level starts rising.

This is a pretty complicated problem.... if people want to mount the filesystem with the sync mount option, sure, but when people want safety, speed, efficiency, power savings, *and* they want to use crappy proprietary device drivers that crash if you look at them funny, *and* be solicitous to application writers that rewrite hundreds of files on desktop startup (even though it's not clear *why* it is useful for KDE or GNOME to rewrite hundreds of files when the user logs in and initializes the desktop), something has got to give.

There's nothing to trade off, other than the sanity of the file system maintainers. (But that's OK, Linus has called us crazy already. :-/)

- Ted

#304 lore raw
Quoted 5 lines · Theodore Tso

To be fair, though, one problem which Matthew Garrett has pointed out is that if lots of applications issue fsync(), it will have the tendency to wake up the hard drive a lot, and do a real number on power utilization. I believe the right solution for this is an extension to laptop mode which synchronizes the filesystem at a clean

Quoted 8 lines

point, and then which suppresses fsync()'s until the hard drive wakes up, at which point it should flush all dirty data to the drive, and then freezes writes to the disk again. Presumably that should be OK, because who are using laptop mode are inherently trading off a certain amount of safety for power savings; but then other people who want to run a mysql server on a laptop get cranky, and then if we start implementing ways that applications can exempt themselves from the fsync() suppression, the complexity level starts rising.

I disagree with this approach. If fsync() means anything other than "Get my data on disk and then return" then we're breaking guarantees to applications. The problem is that you're insisting that the only way applications can ensure that their requests occur in order is to use fsync(), which will achieve that but also provides guarantees above and beyond what the majority of applications want.

I've done some benchmarking now and I'm actually fairly happy with the behaviour of ext4 now - it seems that the real world impact of doing the block allocation at rename time isn't that significant, and if that's the only practical way to ensure ordering guarantees in ext4 then fine. But given that, I don't think there's any reason to try to convince application authors to use fsync() more.

Signature · 1 line
Matthew Garrett | [email protected]
#305 lore raw
Quoted 1 line · Matthew Garrett

I disagree with this approach. If fsync() means anything other than "Get my data on disk and then return" then we're breaking guarantees to applications.

Due to lack of storage dev writeback cache flushing, we are indeed breaking that guarantee in many situations...

Quoted 1 line

The problem is that you're insisting that the only way applications can ensure that their requests occur in order is to use fsync(), which will achieve that but also provides guarantees above and beyond what the majority of applications want.

That remains a true statement... without the *sync* syscalls, you still do not have a _guarantee_ writes occur in a certain order.

Jeff

#306 lore raw
Quoted 8 lines · Jeff Garzik
Quoted 4 lines · Matthew Garrett

The problem is that you're insisting that the only way applications can ensure that their requests occur in order is to use fsync(), which will achieve that but also provides guarantees above and beyond what the majority of applications want.

That remains a true statement... without the *sync* syscalls, you
still do not have a _guarantee_ writes occur in a certain order.

The interesting case is whether data hits disk before metadata when renaming over the top of an existing file, which appears to be guaranteed in the default ext4 configuration now? I'm sure there are filesystems where this isn't the case, but that's mostly just an argument that it's not sensible to use those filesystems if your system's at any risk of crashing.

Signature · 1 line
Matthew Garrett | [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#308 lore raw

Apologies for the HTML email, resent in ASCII below:

Quoted 24 lines · Matthew Garrett
Quoted 8 lines

If fsync() means anything other than "Get
my data on disk and then return" then we're breaking guarantees to applications. The problem is that you're insisting that the only way applications can ensure that their requests occur in order is to use fsync(), which will achieve that but also provides guarantees above and beyond what the majority of applications want.

Quoted 6 lines

I've done some benchmarking now and I'm actually fairly happy with the behaviour of ext4 now - it seems that the real world impact of doing the block allocation at rename time isn't that significant, and if that's the only practical way to ensure ordering guarantees in ext4 then fine. But given that, I don't think there's any reason to try to convince application authors to use fsync() more.

Really, the problem is the filesystem interfaces are incomplete.  There are plenty of ways to specify a "FLUSH CACHE"-type command for an individual file or for the whole filesystem, but there aren't really any ways for programs to specify barriers (either whole-blockdev or per-LBA-range).  An fsync() implies you want to *wait* for the data... there's no way to ask it all to be queued with some ordering constraints. Perhaps we ought to add a couple extra open flags, O_BARRIER_BEFORE and O_BARRIER_AFTER, and rename3(), etc functions that take flags arguments?
Or maybe a new set of syscalls like barrier(file1, file2) and fbarrier(fd1, fd2), which cause all pending changes (perhaps limit to this process?) to the file at fd1 to occur before any successive changes (again limited to this process?) to the file at fd2.
It seems that rename(oldfile, newfile) with an already-existing newfile should automatically imply barrier(oldfile, newfile) before it occurs, simply because so many programs rely on that.
In the cross-filesystem case, the fbarrier() might simply fsync(fd1), since that would provide the equivalent guarantee, albeit with possibly significant performance penalties.  I can't think of any easy way to prevent one filesystem from syncing writes to a particular file until another filesystem has finished an asynchronous fsync() call.  Perhaps a half-way solution would be to asynchronously fsync(fd1) and simply block the next write()/ioctl()/etc on fd2 until the async fsync returns. Are there other ideas for useful barrier()-generating file APIs?
Cheers,
Kyle Moffett

#309 lore raw
Quoted 5 lines · Kyle Moffett
Quoted 5 lines

Really, the problem is the filesystem interfaces are incomplete. There are plenty of ways to specify a "FLUSH CACHE"-type command for an individual file or for the whole filesystem, but there aren't really any ways for programs to specify barriers (either whole-blockdev or per-LBA-range). An fsync() implies you want to *wait* for the data... there's no way to ask it all to be queued with some ordering constraints. Perhaps we ought to add a couple extra open flags, O_BARRIER_BEFORE and O_BARRIER_AFTER, and rename3(), etc functions that take flags arguments?
Or maybe a new set of syscalls like barrier(file1, file2) and fbarrier(fd1, fd2), which cause all pending changes (perhaps limit to this process?) to the file at fd1 to occur before any successive changes (again limited to this process?) to the file at fd2.
It seems that rename(oldfile, newfile) with an already-existing newfile should automatically imply barrier(oldfile, newfile) before it occurs, simply because so many programs rely on that.
In the cross-filesystem case, the fbarrier() might simply fsync(fd1), since that would provide the equivalent guarantee, albeit with possibly significant performance penalties. I can't think of any easy way to prevent one filesystem from syncing writes to a particular file until another filesystem has finished an asynchronous fsync() call. Perhaps a half-way solution would be to asynchronously fsync(fd1) and simply block the next write()/ioctl()/etc on fd2 until the async fsync returns.

Then you have just reinvented the transactional userspace API that people often want to replace POSIX API with. Maybe one day they will succeed.

But "POSIX API replacement" is an area never short of proposals... :)

Jeff

#310 lore raw
Quoted 4 lines · Jeff Garzik

Then you have just reinvented the transactional userspace API that people often want to replace POSIX API with.  Maybe one day they will succeed.

But "POSIX API replacement" is an area never short of proposals... :)

Well, I think the goal is not to *replace* the POSIX API or even provide "transactional" guarantees. The performance penalty for atomic transactions is pretty high, and most programs (like GIT) don't really give a damn, as they provide that on a higher level.

It's like the difference between a modern SMP system that supports memory barriers and write snooping and one of the theoretical "transactional memory" designs that have never caught on.

To be honest I think we could provide much better data consistency guarantees and remove a lot of fsync() calls with just a basic per-filesystem barrier() call.

Cheers,
Kyle Moffett

#311 lore raw
Quoted 5 lines · Kyle Moffett

Well, I think the goal is not to *replace* the POSIX API or even provide "transactional" guarantees. The performance penalty for atomic transactions is pretty high, and most programs (like GIT) don't really give a damn, as they provide that on a higher level.

Speaking with my 'git' hat on, I can tell that

- git was designed to have almost minimal requirements from the filesystem, and to not do anything even half-way clever.

- despite that, we've hit an absolute metric sh*tload of filesystem bugs and misfeatures. Some very much in Linux. And some I bet git was the first to ever notice, exactly because git tries to be really anal, in ways that I can pretty much guarantee no normal program _ever_ is.

For example, the latest one came from git actually checking the error code from 'close()'. Tell me the last time you saw anybody do that in a real program. Hint: it's just not done. EVER. Git does it (and even then, git does it only for the core git object files that we care about so much), and we found a real data-loss CIFS bug thanks to that. Afaik, the bug has been there for a year and half. Don't tell me nobody uses cifs.

Before that, we had cross-directory rename bugs. Or the inexplicable "pread() doesn't work correctly on HP-UX". Or the "readdir() returns the same entry multiple times" bug. And all of this without ever doing anything even _remotely_ odd. No file locking, no rewriting of old files, no lseek()ing in directories, no nothing.

Anybody who wants more complex and subtle filesystem interfaces is just crazy. Not only will they never get used, they'll definitely not be stable.

Quoted 3 lines

To be honest I think we could provide much better data consistency guarantees and remove a lot of fsync() calls with just a basic per-filesystem barrier() call.

The problem is not that we have a lot of fsync() calls. Quite the reverse. fsync() is really really rare. So is being careful in general. The number of applications that do even the _minimal_ safety-net of "create new file, rename it atomically over an old one" is basically zero. Almost everybody ends up rewriting files with something like

	open(name, O_CREAT | O_TRUNC, 0666)
	write();
	close();

where there isn't an fsync in sight, nor any "create temp file", nor likely even any real error checking on the write(), much less the close().

And if we have a Linux-specific magic system call or sync action, it's going to be even more rarely used than fsync(). Do you think anybody really uses the OS X FSYNC_FULL ioctl? Nope. Outside of a few databases, it is almost certainly not going to be used, and fsync() will not be reliable in general.

So rather than come up with new barriers that nobody will use, filesystem people should aim to make "badly written" code "just work" unless people are really really unlucky. Because like it or not, that's what 99% of all code is.

The undeniable FACT that people don't tend to check errors from close() should, for example, mean that delayed allocation must still track disk full conditions, for example. If your filesystem returns ENOSPC at close() rather than at write(), you just lost error coverage for disk full cases from 90% of all apps. It's that simple.

Crying that it's an application bug is like crying over the speed of
light: you should deal with *reality*, not what you wish reality was. Same goes for any complaints that "people should write a temp-file, fsync it, and rename it over the original". You may wish that was what they did, but reality is that "open(filename, O_TRUNC | O_CREAT, 0666)" thing.

Harsh, I know. And in the end, even the _good_ applications will decide that it's not worth the performance penalty of doing an fsync(). In git, for example, where we generally try to be very very very careful, 'fsync()' on the object files is turned off by default.

Why? Because turning it on results in unacceptable behavior on ext3. Now, admittedly, the git design means that a lost new DB file isn't deadly, just potentially very very annoying and confusing - you may have to roll back and re-do your operation by hand, and you have to know enough to be able to do it in the first place.

The point here? Sometimes those filesystem people who say "you must use fsync() to get well-defined semantics" are the same people who SCREWED IT UP SO DAMN BADLY THAT FSYNC ISN'T ACTUALLY REALISTICALLY USEABLE!

Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

Linus

maintainer
#312 lore raw

From: Linus Torvalds <[email protected]>
Date: Wed, 25 Mar 2009 20:40:23 -0700 (PDT)

Quoted 3 lines

For example, the latest one came from git actually checking the error code from 'close()'. Tell me the last time you saw anybody do that in a real program. Hint: it's just not done. EVER.

Emacs does it too, and I know that you consider GNU emacs to be the definition of abnormal :-)

That's how we found some misbehaviors in NFS a while ago, we used to return -EAGAIN or something like that from close() on NFS files. This was like 12 years ago and it gave emacs massive heartburn.

#313 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Kyle Moffett

To be honest I think we could provide much better data consistency guarantees and remove a lot of fsync() calls with just a basic per-filesystem barrier() call.

The problem is not that we have a lot of fsync() calls. Quite the reverse. fsync() is really really rare. So is being careful in general. The number of applications that do even the _minimal_ safety-net of "create new file, rename it atomically over an old one" is basically zero. Almost everybody ends up rewriting files with something like

Quoted 7 lines

       open(name, O_CREAT | O_TRUNC, 0666)
       write();
       close();

where there isn't an fsync in sight, nor any "create temp file", nor likely even any real error checking on the write(), much less the close().

Really, I think virtually all of the database programs would be perfectly happy with an "fsbarrier(fd, flags)" syscall, where if "fd" points to a regular file or directory then it instructs the underlying filesystem to do whatever internal barrier it supports, and if not just fail with -ENOTSUPP (so you can fall back to fdatasync(), etc). Perhaps "flags" would allow a "data" or "metadata" barrier, but if not it's not a big issue.

I've ended up having to write a fair amount of high-performance filesystem library code which almost never ends up using fsync() quite simply because the performance on it sucks so badly. This is one of the big reasons why so many critical database programs use O_DIRECT and reinvent the the wheel^H^H^H^H^H^H pagecache. The only way you can actually use it in high-bandwidth transaction applications is by doing your own IO-thread and buffering system.

You have to have your own buffer ordering dependencies and call fdatasync() or fsync() from individual threads in-between specific ordered IOs. The threading helps you keep other IO in flight while waiting for the flush to finish. For big databases on spinning media (SSDs don't work precisely because they are small and your databases are big) the overhead of a full flush may still be too large. Even with SSDs, with multiple processes vying for IO bandwidth you still want some kind of application-level barrier to avoid introducing bubbles in your IO pipeline.

It all comes down to a trivial calculation: if you can't get (bandwidth * latency-to-stable-storage) bytes of data queued *behind* a flush then your disk is going to sit idle waiting for more data after completing it. If a user-level tool needs to enforce ordering between IOs the only tool right now is is a full flush; when database-oriented tools can use a barrier()-ish call instead, they can issue the op and immediately resume keeping the IO queues full.

Cheers,
Kyle Moffett

#314 lore raw
Quoted 7 lines · Kyle Moffett

Really, I think virtually all of the database programs would be perfectly happy with an "fsbarrier(fd, flags)" syscall, where if "fd" points to a regular file or directory then it instructs the underlying filesystem to do whatever internal barrier it supports, and if not just fail with -ENOTSUPP (so you can fall back to fdatasync(), etc). Perhaps "flags" would allow a "data" or "metadata" barrier, but if not it's not a big issue.

If you want a per-fd barrier call, there is always sync_file_range(2)

Quoted 2 lines

If a user-level tool needs to enforce ordering between IOs the only tool right now is is a full flush

or sync_file_range(2)...

Jeff

#315 lore raw
Quoted 10 lines · Jeff Garzik
Quoted 7 lines · Kyle Moffett

Really, I think virtually all of the database programs would be perfectly happy with an "fsbarrier(fd, flags)" syscall, where if "fd" points to a regular file or directory then it instructs the underlying filesystem to do whatever internal barrier it supports, and if not just fail with -ENOTSUPP (so you can fall back to fdatasync(), etc). Perhaps "flags" would allow a "data" or "metadata" barrier, but if not it's not a big issue.

If you want a per-fd barrier call, there is always sync_file_range(2)

The issue is that sync_file_range doesn't seem to be documented to have any inter-file barrier semantics. Even then, from the manpage it doesn't look like write(fd)+sync_file_range(fd,SYNC_FILE_RANGE_WRITE)+write(fd) would actually prevent the second write from occurring before the first has actually hit disk (assuming both are within the specified range).

Cheers,
Kyle Moffett

#316 lore raw
In reply to a message not in the archive
Quoted 6 lines · Kyle Moffett

Perhaps we ought to add a couple extra open flags, O_BARRIER_BEFORE and O_BARRIER_AFTER, and rename3(), etc functions that take flags arguments? Or maybe a new set of syscalls like barrier(file1, file2) and fbarrier(fd1, fd2), which cause all pending changes (perhaps limit to this process?) to the file at fd1 to occur before any successive changes (again limited to this process?) to the file at fd2.

That's an option, but what would benefit? If rename is expected to preserve ordering (which I think it has to, in order to avoid breaking existing code) then are there any other interesting use cases?

Signature · 1 line
Matthew Garrett | [email protected]
#317 lore raw
Quoted 9 lines · Matthew Garrett
Quoted 6 lines · Kyle Moffett

   Perhaps we ought to add a couple extra open flags, O_BARRIER_BEFORE and O_BARRIER_AFTER, and rename3(), etc functions that take flags arguments? Or maybe a new set of syscalls like barrier(file1, file2) and fbarrier(fd1, fd2), which cause all pending changes (perhaps limit to this process?) to the file at fd1 to occur before any successive changes (again limited to this process?) to the file at fd2.

Quoted 3 lines

That's an option, but what would benefit? If rename is expected to preserve ordering (which I think it has to, in order to avoid breaking existing code) then are there any other interesting use cases?

The use cases would be programs like GIT (or any other kind of database) where you want to ensure that your new pulled packfile has fully hit disk before the ref update does. If that ordering constraint is applied, then we don't really care when we crash, because either we have a partial packfile update (and we have to pull again) or we have the whole thing. The rename() barrier would ensure that we either have the old ref or the new ref, but it would not check to ensure that the whole packfile is on disk yet.

I would imagine that databases like MySQL could also use such support to help speed up their database transaction support, instead of having to run a bunch of threads which fsync() and buffer data internally.

Cheers,
Kyle Moffett

#318 lore raw
In reply to Theodore Tso
Quoted 2 lines · Theodore Tso

Now, there are three ways of solving this problem.

You seem to disregard the "write in the right order" approach. Or is that your:

Quoted 3 lines

The third potential solution we can try doing is to make some tuning adjustments to the VM so that we start pushing out these data blocks much more aggressively out to the disk.

Yes. but at least one problem is, as mentioned, that when the VM calls writepage[s]() to start async writeback, many filesystems do seem to just _block_ on it.

So the VM has a really hard time doing anything sanely early - the filesystems seem to take a perverse pleasure in synchronizing things using blocking semaphores.

Linus

maintainer
#319 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 3 lines

The third potential solution we can try doing is to make some tuning adjustments to the VM so that we start pushing out these data blocks much more aggressively out to the disk.

Yes. but at least one problem is, as mentioned, that when the VM calls writepage[s]() to start async writeback, many filesystems do seem to just _block_ on it.

Um, no, ext3 shouldn't block on writepage(). Since it doesn't do delayed allocation, it should always be able to push out a dirty page to the disk.

- Ted

#320 lore raw
Quoted 4 lines · Theodore Tso

Um, no, ext3 shouldn't block on writepage(). Since it doesn't do delayed allocation, it should always be able to push out a dirty page to the disk.

Umm. Maybe I'm mis-reading something, but they seem to all synchronize with the journal with "ext3_journal_start/stop".

Which will at a minimum wait for 'j_barrier_count == 0' and 't_state != T_LOCKED'. Along with making sure that there are enough transaction buffers.

Do I understand _why_ ext3 does that? Hell no. The code makes no sense to me. But I don't think I'm wrong.

Look at the sane case (data=ordered): it still does

	handle = ext3_journal_start(inode, ext3_writepage_trans_blocks(inode));
	...
	err = ext3_journal_stop(handle);

around all the IO starting. Never mind that the IO shouldn't be needing any journal activity at all afaik in any common case.

Yes, yes, it may need to allocate backing store (a page that was dirtied by mmap), and I'm sure that's the reason for it all, but the point is, most of the time there should be no journal activity at all, yet it looks very much like a simple writepage() will synchronize with a full journal and wait for the journal to get space.

No?

So tell me again how the VM can rely on the filesystem not blocking at random points.

Linus

reviewer
#321 lore raw
Quoted 25 lines · Linus Torvalds
Quoted 4 lines · Theodore Tso

Um, no, ext3 shouldn't block on writepage(). Since it doesn't do delayed allocation, it should always be able to push out a dirty page to the disk.

Umm. Maybe I'm mis-reading something, but they seem to all synchronize with the journal with "ext3_journal_start/stop".

Which will at a minimum wait for 'j_barrier_count == 0' and 't_state != T_LOCKED'. Along with making sure that there are enough transaction buffers.

Do I understand _why_ ext3 does that? Hell no. The code makes no sense to me. But I don't think I'm wrong.

Look at the sane case (data=ordered): it still does

	handle = ext3_journal_start(inode, ext3_writepage_trans_blocks(inode));
	...
	err = ext3_journal_stop(handle);

around all the IO starting. Never mind that the IO shouldn't be needing any journal activity at all afaik in any common case.

Quoted 7 lines

Yes, yes, it may need to allocate backing store (a page that was dirtied by mmap), and I'm sure that's the reason for it all, but the point is, most of the time there should be no journal activity at all, yet it looks very much like a simple writepage() will synchronize with a full journal and wait for the journal to get space.

No?

Yes, you got it right. Furthermore in ordered mode we need to attach
buffers to the running transaction if they aren't there (but for checking whether they are we need to pin the running transaction and we are basically where we started.. damn). But maybe there's a way out of it. We don't have to guarantee data written via mmap are on disk when "the transaction running when somebody decided to call writepage" commits (in case no block allocation happen) and so we could just submit those buffers for IO and don't attach them to the transaction...

Quoted 2 lines

So tell me again how the VM can rely on the filesystem not blocking at random points.

I can write a patch to make writepage() in the non-"mmapped creation"
case non-blocking on journal. But I'll also have to find out whether it really helps something. But it's probably worth trying...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#322 lore raw
Quoted 4 lines · Jan Kara

I can write a patch to make writepage() in the non-"mmapped creation"
case non-blocking on journal. But I'll also have to find out whether it really helps something. But it's probably worth trying...

Actually, it really should be easier to make a patch that just does the journal thing if ->set_page_dirty() is called, and buffers weren't already allocated.

Then ext3_[ordered|writeback]_writepage() _should_ just become something like

	if (test_opt(inode->i_sb, NOBH))
		return nobh_writepage(page, ext3_get_block, wbc);

	return block_write_full_page(page, ext3_get_block, wbc);

and that's it. The code would be simpler to understand to boot.

Linus

#323 lore raw

ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Jan Kara
Quoted 7 lines · Jan Kara
Quoted 2 lines

So tell me again how the VM can rely on the filesystem not blocking at random points.

I can write a patch to make writepage() in the non-"mmapped
creation" case non-blocking on journal. But I'll also have to find out whether it really helps something. But it's probably worth trying...

_all_ the problems i ever had with ext3 were 'collateral damage' type of things: simple writes (sometimes even reads) getting serialized on some large [but reasonable] dirtying activity elsewhere - even if the system was still well within its hard-dirty-limit threshold.

So it sure sounds like an area worth improving, and it's not that hard to reproduce either. Take a system with enough RAM but only a single disk, and do this in a kernel tree:

sync
echo 3 > /proc/sys/vm/drop_caches

while :; do
date
make mrproper 2>/dev/null >/dev/null
make defconfig 2>/dev/null >/dev/null
make -j32 bzImage 2>/dev/null >/dev/null
done &

Plain old kernel build, no distcc and no icecream. Wait a few minutes for the system to reach equilibrium. There's no tweaking anywhere, kernel, distro and filesystem defaults used everywhere:

aldebaran:/home/mingo/linux/linux> ./compile-test Thu Mar 26 10:33:03 CET 2009
Thu Mar 26 10:35:24 CET 2009
Thu Mar 26 10:36:48 CET 2009
Thu Mar 26 10:38:54 CET 2009
Thu Mar 26 10:41:22 CET 2009
Thu Mar 26 10:43:41 CET 2009
Thu Mar 26 10:46:02 CET 2009
Thu Mar 26 10:48:28 CET 2009

And try to use the system while this workload is going on. Use Vim to edit files in this kernel tree. Use plain _cat_ - and i hit delays all the time - and it's not the CPU scheduler but all IO related.

I have such an ext3 based system where i can do such tests and where i dont mind crashes and data corruption either, so if you send me experimental patches against latet -git i can try them immediately. The system has 16 CPUs, 12GB of RAM and a single disk.

Btw., i had this test going on that box while i wrote some simple scripts in Vim - and it was a horrible experience. The worst wait was well above one minute - Vim just hung there indefinitely. Not even Ctrl-Z was possible. I captured one such wait, it was hanging right here:

aldebaran:~/linux/linux> cat /proc/3742/stack

 [<ffffffff8034790a>] log_wait_commit+0xbd/0x110
 [<ffffffff803430b2>] journal_stop+0x1df/0x20d
 [<ffffffff8034421f>] journal_force_commit+0x28/0x2d
 [<ffffffff80331c69>] ext3_force_commit+0x2b/0x2d
 [<ffffffff80328b56>] ext3_write_inode+0x3e/0x44
 [<ffffffff802ebb9d>] __sync_single_inode+0xc1/0x2ad
 [<ffffffff802ebed6>] __writeback_single_inode+0x14d/0x15a
 [<ffffffff802ebf0c>] sync_inode+0x29/0x34
 [<ffffffff80327453>] ext3_sync_file+0xa7/0xb4
 [<ffffffff802ef17d>] vfs_fsync+0x78/0xaf
 [<ffffffff802ef1eb>] do_fsync+0x37/0x4d
 [<ffffffff802ef228>] sys_fsync+0x10/0x14
 [<ffffffff8020bd1b>] system_call_fastpath+0x16/0x1b
 [<ffffffffffffffff>] 0xffffffffffffffff

It took about 120 seconds for it to recover.

And it's not just sys_fsync(). The script i wrote tests file read latencies. I have created 1000 files with the same size (all copies of kernel/sched.c ;-), and tested their cache-cold plain-cat performance via:

for ((i=0;i<1000;i++)); do
printf "file #%4d, plain reading it took: " $i /usr/bin/time -f "%e seconds." cat $i >/dev/null
done

I.e. plain, supposedly high-prio reads. The result is very common hickups in read latencies:

file # 579 (253560 bytes), reading it took: 0.08 seconds. file # 580 (253560 bytes), reading it took: 0.05 seconds. file # 581 (253560 bytes), reading it took: 0.01 seconds. file # 582 (253560 bytes), reading it took: 0.01 seconds. file # 583 (253560 bytes), reading it took: 4.61 seconds. file # 584 (253560 bytes), reading it took: 1.29 seconds. file # 585 (253560 bytes), reading it took: 3.01 seconds. file # 586 (253560 bytes), reading it took: 7.74 seconds. file # 587 (253560 bytes), reading it took: 3.22 seconds. file # 588 (253560 bytes), reading it took: 0.05 seconds. file # 589 (253560 bytes), reading it took: 0.36 seconds. file # 590 (253560 bytes), reading it took: 7.39 seconds. file # 591 (253560 bytes), reading it took: 7.58 seconds. file # 592 (253560 bytes), reading it took: 7.90 seconds. file # 593 (253560 bytes), reading it took: 8.78 seconds. file # 594 (253560 bytes), reading it took: 8.01 seconds. file # 595 (253560 bytes), reading it took: 7.47 seconds. file # 596 (253560 bytes), reading it took: 11.52 seconds. file # 597 (253560 bytes), reading it took: 10.33 seconds. file # 598 (253560 bytes), reading it took: 8.56 seconds. file # 599 (253560 bytes), reading it took: 7.58 seconds.

The system's RAM is ridiculously under-utilized, 96.1% is free, only 3.9% is utilized:

total used free shared buffers cached

 Mem:      12318192     476732   11841460          0      48324     142936
 -/+ buffers/cache:     285472   12032720

Swap: 4096564 0 4096564

Dirty data in /proc/meminfo fluctuates between 0.4% and 1.6% of total RAM. (the script removes the freshly build kernel object files, so the workload is pretty steady.)

The peak of 1.6% looks like this:

Dirty:            118376 kB
Dirty:            143784 kB
Dirty:            161756 kB
Dirty:            185084 kB
Dirty:            210524 kB
Dirty:            213348 kB
Dirty:            200124 kB
Dirty:            122152 kB
Dirty:            121508 kB
Dirty:            121512 kB

(1 second snapshots)

So the problems are all around the place and they are absolutely, trivially reproducible. And this is how a default ext3 based distro and the default upstream kernel will present itself to new Linux users and developers. It's not a pretty experience.

Oh, and while at it - also a job control complaint. I tried to Ctrl-C the above script:

file # 858 (253560 bytes), reading it took: 0.06 seconds. file # 859 (253560 bytes), reading it took: 0.02 seconds. file # 860 (253560 bytes), reading it took: 5.53 seconds. file # 861 (253560 bytes), reading it took: 3.70 seconds. file # 862 (253560 bytes), reading it took: 0.88 seconds. file # 863 (253560 bytes), reading it took: 0.04 seconds. file # 864 (253560 bytes), reading it took: ^C0.69 seconds. file # 865 (253560 bytes), reading it took: ^C0.49 seconds. file # 866 (253560 bytes), reading it took: ^C0.01 seconds. file # 867 (253560 bytes), reading it took: ^C0.02 seconds. file # 868 (253560 bytes), reading it took: ^C^C0.01 seconds. file # 869 (253560 bytes), reading it took: ^C^C0.04 seconds. file # 870 (253560 bytes), reading it took: ^C^C^C0.03 seconds. file # 871 (253560 bytes), reading it took: ^C0.02 seconds. file # 872 (253560 bytes), reading it took: ^C^C0.02 seconds. file # 873 (253560 bytes), reading it took:
^C^C^C^Caldebaran:~/linux/linux/test-files/src>

I had to hit Ctrl-C numerous times before Bash would honor it. This to is a very common thing on large SMP systems. I'm willing to test patches until all these problems are fixed. Any takers?

Ingo

#324 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 21 lines · Ingo Molnar

Oh, and while at it - also a job control complaint. I tried to Ctrl-C the above script:

file # 858 (253560 bytes), reading it took: 0.06 seconds. file # 859 (253560 bytes), reading it took: 0.02 seconds. file # 860 (253560 bytes), reading it took: 5.53 seconds. file # 861 (253560 bytes), reading it took: 3.70 seconds. file # 862 (253560 bytes), reading it took: 0.88 seconds. file # 863 (253560 bytes), reading it took: 0.04 seconds. file # 864 (253560 bytes), reading it took: ^C0.69 seconds. file # 865 (253560 bytes), reading it took: ^C0.49 seconds. file # 866 (253560 bytes), reading it took: ^C0.01 seconds. file # 867 (253560 bytes), reading it took: ^C0.02 seconds. file # 868 (253560 bytes), reading it took: ^C^C0.01 seconds. file # 869 (253560 bytes), reading it took: ^C^C0.04 seconds. file # 870 (253560 bytes), reading it took: ^C^C^C0.03 seconds. file # 871 (253560 bytes), reading it took: ^C0.02 seconds. file # 872 (253560 bytes), reading it took: ^C^C0.02 seconds. file # 873 (253560 bytes), reading it took:
^C^C^C^Caldebaran:~/linux/linux/test-files/src>

Quoted 2 lines

I had to hit Ctrl-C numerous times before Bash would honor it. This to is a very common thing on large SMP systems. [...]

It happened when i tried to Ctrl-C the compile job as well:

Thu Mar 26 11:04:05 CET 2009
Thu Mar 26 11:06:30 CET 2009
^CThu Mar 26 11:07:55 CET 2009 ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^Caldebaran:/home/mingo/linux/linux>

a single Ctrl-C is rarely enough to stop busy Bash shell scripts on Linux. Why?

Ingo

#325 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Ingo Molnar
Quoted 26 lines · Ingo Molnar

And it's not just sys_fsync(). The script i wrote tests file read latencies. I have created 1000 files with the same size (all copies of kernel/sched.c ;-), and tested their cache-cold plain-cat performance via:

for ((i=0;i<1000;i++)); do
printf "file #%4d, plain reading it took: " $i /usr/bin/time -f "%e seconds." cat $i >/dev/null
done

I.e. plain, supposedly high-prio reads. The result is very common hickups in read latencies:

file # 579 (253560 bytes), reading it took: 0.08 seconds. file # 580 (253560 bytes), reading it took: 0.05 seconds. file # 581 (253560 bytes), reading it took: 0.01 seconds. file # 582 (253560 bytes), reading it took: 0.01 seconds. file # 583 (253560 bytes), reading it took: 4.61 seconds. file # 584 (253560 bytes), reading it took: 1.29 seconds. file # 585 (253560 bytes), reading it took: 3.01 seconds. file # 586 (253560 bytes), reading it took: 7.74 seconds. file # 587 (253560 bytes), reading it took: 3.22 seconds. file # 588 (253560 bytes), reading it took: 0.05 seconds. file # 589 (253560 bytes), reading it took: 0.36 seconds. file # 590 (253560 bytes), reading it took: 7.39 seconds. file # 591 (253560 bytes), reading it took: 7.58 seconds.

Quoted 8 lines

file # 592 (253560 bytes), reading it took: 7.90 seconds. file # 593 (253560 bytes), reading it took: 8.78 seconds. file # 594 (253560 bytes), reading it took: 8.01 seconds. file # 595 (253560 bytes), reading it took: 7.47 seconds. file # 596 (253560 bytes), reading it took: 11.52 seconds. file # 597 (253560 bytes), reading it took: 10.33 seconds. file # 598 (253560 bytes), reading it took: 8.56 seconds. file # 599 (253560 bytes), reading it took: 7.58 seconds.

Did you capture the trace of the long delays in the read test case? It can be two things, at least. One is that each little read takes much longer than it should, the other is that we get stuck waiting on a dirty page and hence that slows down the reads a lot.

Signature · 1 line
Jens Axboe
#326 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 6 lines · On Thu, 26 Mar 2009 12:08:15 +0100 Jens Axboe <[email protected]> wrote:

Did you capture the trace of the long delays in the read test case? It can be two things, at least. One is that each little read takes much longer than it should, the other is that we get stuck waiting on a dirty page and hence that slows down the reads a lot.

would be interesting to run latencytop during such runs...

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#327 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 10 lines · Arjan van de Ven
Quoted 6 lines · On Thu, 26 Mar 2009 12:08:15 +0100 Jens Axboe <[email protected]> wrote:

Did you capture the trace of the long delays in the read test case? It can be two things, at least. One is that each little read takes much longer than it should, the other is that we get stuck waiting on a dirty page and hence that slows down the reads a lot.

would be interesting to run latencytop during such runs...

Things just drown in the noise in there sometimes, is my experience... And disappear. Perhaps I'm just not very good at reading the output, but it was never very useful for me. I know, not a very useful complaint, I'll try and use it again and come up with something more productive :-)

But in this case, I bet it's also the atime updates. If Ingo turned those off, the read results would likely be a lot better (and consistent).

Signature · 1 line
Jens Axboe
#328 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 19 lines · On Thu, 26 Mar 2009 15:36:18 +0100 Jens Axboe <[email protected]> wrote:
Quoted 11 lines · Arjan van de Ven
Quoted 7 lines · On Thu, 26 Mar 2009 12:08:15 +0100 Jens Axboe <[email protected]> wrote:

Did you capture the trace of the long delays in the read test case? It can be two things, at least. One is that each little read takes much longer than it should, the other is that we get stuck waiting on a dirty page and hence that slows down the reads a lot.

would be interesting to run latencytop during such runs...

Things just drown in the noise in there sometimes, is my experience... And disappear. Perhaps I'm just not very good at reading the output, but it was never very useful for me. I know, not a very useful complaint, I'll try and use it again and come up with something more productive :-)

Quoted 3 lines

But in this case, I bet it's also the atime updates. If Ingo turned those off, the read results would likely be a lot better (and consistent).

latencytop does identify those for sure.

Quoted 1 line
Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
maintainer
#329 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Ingo Molnar

Ingo,

Interesting. I wonder if the problem is the journal is cycling fast enough that it is checkpointing all the time. If so, it could be that a bigger-sized journal might help. Can you try this as an experiment? Mount the filesystem using ext4, with the mount option nodelalloc. With an filesystem formatted as ext3, and with delayed allocation disabled, it should behave mostly the same as ext3; try and make sure you're still seeing the same problems.

Then could you grab /proc/fs/jbd2/<dev>:8/history and /proc/fs/jbd2/<dev>:8/info while running your test workload?

Also, can you send me the output of "dumpe2fs -h /dev/sdXX | grep Journal"?

Quoted 5 lines

Oh, and while at it - also a job control complaint. I tried to Ctrl-C the above script:

I had to hit Ctrl-C numerous times before Bash would honor it. This to is a very common thing on large SMP systems.

Well, the script you sent runs the compile in the background. It did:

Quoted 6 lines

while :; do
date
make mrproper 2>/dev/null >/dev/null
make defconfig 2>/dev/null >/dev/null
make -j32 bzImage 2>/dev/null >/dev/null
done &

         ^^

So there would have been nothing to ^C; I assume you were running this with a variant that didn't have the ampersand, which would have run the whole shell pipeline in a detached background process?

In any case, the workaround for this is to ^Z the script, and then "kill %" it.

I'm pretty sure this is actually a bash problem. When you send a Ctrl-C, it sends a SIGINT to all of the members of the tty's foreground process group. Under some circumstances, bash sets the signal handler for SIGINT to be SIGIGN. I haven't looked at this super closely (it would require diving into the bash sources), but you can see it if you attach an strace to the bash shell driving a script such as

#!/bin/bash

while /bin/true; do
date
sleep 60
done &

If you do a "ps axo pid,ppid,pgrp,args", you'll see that the bash and the sleep 60 have the same process group. If you emulate hitting ^C by sending a SIGINT to pid of the shell, you'll see that it ignores it. Sleep also seems to be ignoring the SIGINT when run in the background; but it does honor SIGINT in the foreground --- I didn't have time to dig into that.

In any case, bash appears to SIGIGN the INT signal if there is a child process running, and only takes the ^C if bash itself is actually "running" the shell script. For example, if you run the command "date;sleep 10;date;sleep 10;date", the ^C only interrupts the sleep command. It doesn't stop the series of commands which bash is running.

- Ted

#330 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 6 lines · Theodore Tso
Quoted 6 lines

while :; do
date
make mrproper 2>/dev/null >/dev/null
make defconfig 2>/dev/null >/dev/null
make -j32 bzImage 2>/dev/null >/dev/null
done &

Quoted 6 lines
         ^^

So there would have been nothing to ^C; I assume you were running this with a variant that didn't have the ampersand, which would have run the whole shell pipeline in a detached background process?

That was just the example - the real script did not go into the background so it was Ctrl-C-able. I've attached it.

Quoted 25 lines

In any case, the workaround for this is to ^Z the script, and then "kill %" it.

I'm pretty sure this is actually a bash problem. When you send a Ctrl-C, it sends a SIGINT to all of the members of the tty's foreground process group. Under some circumstances, bash sets the signal handler for SIGINT to be SIGIGN. I haven't looked at this super closely (it would require diving into the bash sources), but you can see it if you attach an strace to the bash shell driving a script such as

#!/bin/bash

while /bin/true; do
date
sleep 60
done &

If you do a "ps axo pid,ppid,pgrp,args", you'll see that the bash and the sleep 60 have the same process group. If you emulate hitting ^C by sending a SIGINT to pid of the shell, you'll see that it ignores it. Sleep also seems to be ignoring the SIGINT when run in the background; but it does honor SIGINT in the foreground --- I didn't have time to dig into that.

Quoted 6 lines

In any case, bash appears to SIGIGN the INT signal if there is a child process running, and only takes the ^C if bash itself is actually "running" the shell script. For example, if you run the command "date;sleep 10;date;sleep 10;date", the ^C only interrupts the sleep command. It doesn't stop the series of commands which bash is running.

It happens all the time - and it does look like a Bash bug. I reported it to the Bash maintainer one or two years ago. He said he does not see it as he's using MacOS X. Can dig into archives if needed.

Ingo

#
# Cool down caches as much as possible:
#
sync
echo 3 > /proc/sys/vm/drop_caches

#
# Kernel developer builds his kernel workload:
#
while :; do
date
make mrproper 2>/dev/null >/dev/null
make defconfig 2>/dev/null >/dev/null
make -j32 bzImage 2>/dev/null >/dev/null
done

#331 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Theodore Tso
Quoted 1 line · Theodore Tso

Also, can you send me the output of "dumpe2fs -h /dev/sdXX | grep Journal"?

Sure:

[root@aldebaran ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on

 /dev/sda1             40313964  11595520  26670560  31% /
 /dev/sda2            403160732  35165460 347515212  10% /home

tmpfs 6159096 48 6159048 1% /dev/shm

[root@aldebaran ~]# dumpe2fs -h /dev/sda2 | grep Journal dumpe2fs 1.40.8 (13-Mar-2008)

 Journal inode:            8
 Journal backup:           inode blocks
 Journal size:             128M

Stock Fedora 9 release/install, updated, and booted to 2.6.29.

Ingo

#332 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Theodore Tso
Quoted 10 lines · Theodore Tso

Ingo,

Interesting. I wonder if the problem is the journal is cycling fast enough that it is checkpointing all the time. If so, it could be that a bigger-sized journal might help. Can you try this as an experiment? Mount the filesystem using ext4, with the mount option nodelalloc. With an filesystem formatted as ext3, and with delayed allocation disabled, it should behave mostly the same as ext3; try and make sure you're still seeing the same problems.

Quoted 2 lines

Then could you grab /proc/fs/jbd2/<dev>:8/history and /proc/fs/jbd2/<dev>:8/info while running your test workload?

i tried it:

/dev/sda2 on /home type ext4 (rw,nodelalloc)

I still see similarly bad latencies in Vim:

aldebaran:~> cat /proc/10227/stack

[<ffffffff80370cad>] jbd2_log_wait_commit+0xbd/0x110
[<ffffffff8036bc70>] jbd2_journal_stop+0x1f3/0x221
[<ffffffff8036ccb0>] jbd2_journal_force_commit+0x28/0x2c
[<ffffffff80352660>] ext4_force_commit+0x2e/0x34
[<ffffffff80346682>] ext4_write_inode+0x3e/0x44
[<ffffffff802eb941>] __sync_single_inode+0xc1/0x2ad
[<ffffffff802ebc7a>] __writeback_single_inode+0x14d/0x15a
[<ffffffff802ebcb0>] sync_inode+0x29/0x34
[<ffffffff80343e16>] ext4_sync_file+0xf6/0x138
[<ffffffff802eef21>] vfs_fsync+0x78/0xaf
[<ffffffff802eef8f>] do_fsync+0x37/0x4d
[<ffffffff802eefcc>] sys_fsync+0x10/0x14
[<ffffffff8020bd1b>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Vim is still almost unusable during this workload - even if i dont write out the source file just use it interactively to edit it.

The read-test is somewhat better. There are occasional blips of 4-5 seconds:

file # 928 (253560 bytes), reading it took: 0.76 seconds. file # 929 (253560 bytes), reading it took: 3.98 seconds. file # 930 (253560 bytes), reading it took: 3.45 seconds. file # 931 (253560 bytes), reading it took: 0.04 seconds.

I have also written a 'vim open' test which does vim -c q, i.e. it just opens a source file and closes it without writing the file. That too takes a lot of time:

file #   0 (253560 bytes), Vim-opening it took: 2.04 seconds.
file #   1 (253560 bytes), Vim-opening it took: 2.39 seconds.
file #   2 (253560 bytes), Vim-opening it took: 2.03 seconds.
file #   3 (253560 bytes), Vim-opening it took: 2.81 seconds.
file #   4 (253560 bytes), Vim-opening it took: 2.11 seconds.
file #   5 (253560 bytes), Vim-opening it took: 2.44 seconds.
file #   6 (253560 bytes), Vim-opening it took: 2.04 seconds.
file #   7 (253560 bytes), Vim-opening it took: 3.59 seconds.
file #   8 (253560 bytes), Vim-opening it took: 2.06 seconds.
file #   9 (253560 bytes), Vim-opening it took: 3.26 seconds.
file #  10 (253560 bytes), Vim-opening it took: 2.04 seconds.
file #  11 (253560 bytes), Vim-opening it took: 2.38 seconds.
file #  12 (253560 bytes), Vim-opening it took: 2.04 seconds.
file #  13 (253560 bytes), Vim-opening it took: 3.05 seconds.

Here's a few snapshots of Vim waiting spots:

aldebaran:~> cat /proc/$(ps aux | grep -m 1 'vim -c' | cut -d' ' -f5)/stack

[<ffffffff8036c1ae>] do_get_write_access+0x22b/0x452
[<ffffffff8036c3fc>] jbd2_journal_get_write_access+0x27/0x38
[<ffffffff8035aa8c>] __ext4_journal_get_write_access+0x51/0x59
[<ffffffff80346f30>] ext4_reserve_inode_write+0x3d/0x79
[<ffffffff80346f9f>] ext4_mark_inode_dirty+0x33/0x187
[<ffffffff8034724e>] ext4_dirty_inode+0x6a/0x9f
[<ffffffff802ec4eb>] __mark_inode_dirty+0x38/0x199
[<ffffffff802e27f5>] touch_atime+0xf6/0x101
[<ffffffff8029cc83>] do_generic_file_read+0x37c/0x3c7
[<ffffffff8029d770>] generic_file_aio_read+0x15b/0x197
[<ffffffff802d0816>] do_sync_read+0xec/0x132
[<ffffffff802d11de>] vfs_read+0xb0/0x139
[<ffffffff802d1335>] sys_read+0x4c/0x74
[<ffffffff8020bd1b>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

aldebaran:~> cat /proc/$(ps aux | grep -m 1 'vim -c' | cut -d' ' -f5)/stack

[<ffffffff8029c0ed>] sync_page+0x41/0x45
[<ffffffff8029c274>] wait_on_page_bit+0x73/0x7a
[<ffffffff802a5a76>] truncate_inode_pages_range+0x2f6/0x37b
[<ffffffff802a5b0d>] truncate_inode_pages+0x12/0x15
[<ffffffff8034b97b>] ext4_delete_inode+0x6a/0x25f
[<ffffffff802e378e>] generic_delete_inode+0xe7/0x174
[<ffffffff802e382f>] generic_drop_inode+0x14/0x1d
[<ffffffff802e2866>] iput+0x66/0x6a
[<ffffffff802db889>] do_unlinkat+0x107/0x15d
[<ffffffff802db8f5>] sys_unlink+0x16/0x18
[<ffffffff8020bd1b>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

aldebaran:~> cat /proc/$(ps aux | grep -m 1 'vim -c' | cut -d' ' -f5)/stack

[<ffffffff8036c1ae>] do_get_write_access+0x22b/0x452
[<ffffffff8036c3fc>] jbd2_journal_get_write_access+0x27/0x38
[<ffffffff8035aa8c>] __ext4_journal_get_write_access+0x51/0x59
[<ffffffff80346f30>] ext4_reserve_inode_write+0x3d/0x79
[<ffffffff80346f9f>] ext4_mark_inode_dirty+0x33/0x187
[<ffffffff8034724e>] ext4_dirty_inode+0x6a/0x9f
[<ffffffff802ec4eb>] __mark_inode_dirty+0x38/0x199
[<ffffffff802e27f5>] touch_atime+0xf6/0x101
[<ffffffff8029cc83>] do_generic_file_read+0x37c/0x3c7
[<ffffffff8029d770>] generic_file_aio_read+0x15b/0x197
[<ffffffff802d0816>] do_sync_read+0xec/0x132
[<ffffffff802d11de>] vfs_read+0xb0/0x139
[<ffffffff802d1335>] sys_read+0x4c/0x74
[<ffffffff8020bd1b>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

aldebaran:~> cat /proc/$(ps aux | grep -m 1 'vim -c' | cut -d' ' -f5)/stack

[<ffffffff8036c1ae>] do_get_write_access+0x22b/0x452
[<ffffffff8036c3fc>] jbd2_journal_get_write_access+0x27/0x38
[<ffffffff8035aa8c>] __ext4_journal_get_write_access+0x51/0x59
[<ffffffff80346f30>] ext4_reserve_inode_write+0x3d/0x79
[<ffffffff80346f9f>] ext4_mark_inode_dirty+0x33/0x187
[<ffffffff8034724e>] ext4_dirty_inode+0x6a/0x9f
[<ffffffff802ec4eb>] __mark_inode_dirty+0x38/0x199
[<ffffffff802e27f5>] touch_atime+0xf6/0x101
[<ffffffff8029cc83>] do_generic_file_read+0x37c/0x3c7
[<ffffffff8029d770>] generic_file_aio_read+0x15b/0x197
[<ffffffff802d0816>] do_sync_read+0xec/0x132
[<ffffffff802d11de>] vfs_read+0xb0/0x139
[<ffffffff802d1335>] sys_read+0x4c/0x74
[<ffffffff8020bd1b>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

aldebaran:~> cat /proc/$(ps aux | grep -m 1 'vim -c' | cut -d' ' -f5)/stack

[<ffffffff8036c1ae>] do_get_write_access+0x22b/0x452
[<ffffffff8036c3fc>] jbd2_journal_get_write_access+0x27/0x38
[<ffffffff8035aa8c>] __ext4_journal_get_write_access+0x51/0x59
[<ffffffff80346f30>] ext4_reserve_inode_write+0x3d/0x79
[<ffffffff80346f9f>] ext4_mark_inode_dirty+0x33/0x187
[<ffffffff8034724e>] ext4_dirty_inode+0x6a/0x9f
[<ffffffff802ec4eb>] __mark_inode_dirty+0x38/0x199
[<ffffffff802e27f5>] touch_atime+0xf6/0x101
[<ffffffff8029cc83>] do_generic_file_read+0x37c/0x3c7
[<ffffffff8029d770>] generic_file_aio_read+0x15b/0x197
[<ffffffff802d0816>] do_sync_read+0xec/0x132
[<ffffffff802d11de>] vfs_read+0xb0/0x139
[<ffffffff802d1335>] sys_read+0x4c/0x74
[<ffffffff8020bd1b>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

That's in good deal atime update latencies. We still appear to default to atime enabled in ext4.

That's stupid - only around 0.01% of all Linux systems relies on atime - and even those who rely on it would be well served by relatime. Why arent the relatime patches upstream? Why isnt it the default? They have been submitted several times.

Atime in its current mandatory do-a-write-for-every-read form is a stupid relic and we have been paying the fool's tax for it in the past 10 years.

Ingo

#333 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 3 lines
Quoted 3 lines

Then could you grab /proc/fs/jbd2/<dev>:8/history and /proc/fs/jbd2/<dev>:8/info while running your test workload?

grabbed them after the tests. (not much else ran before and after the tests) Here they are:

R/C  tid   wait  run   lock  flush log   hndls  block inlog ctime write drop  close
R    642267 0     5000  0     84    3288  76104  560   562  
R    642268 0     5000  956   2024  5760  68891  491   493  
R    642269 956   7788  8     5104  6696  182270 667   669  
R    642270 8     11800 216   7000  6816  186159 834   837  
R    642271 60    13816 0     0     492   45115  2162  2169 
R    642272 0     5000  0     0     2144  44278  1266  1270 
R    642273 0     5000  0     80    3144  73604  444   446  
R    642274 0     5000  0     276   3120  71741  488   490  
R    642275 0     5000  0     288   3608  87334  526   528  
R    642276 0     5000  0     112   2992  83061  512   514  
R    642277 0     5000  0     84    5892  75029  468   470  
R    642278 0     5976  0     848   8564  71693  483   485  
R    642279 0     9412  340   5432  7104  167415 664   666  
R    642280 340   12536 0     8764  3820  270409 906   909  
R    642281 0     12584 0     0     576   38603  2175  2182 
R    642282 0     5000  0     16    2620  51638  1275  1279 
R    642283 0     5000  0     16    2364  58962  376   378  
R    642284 0     5000  0     56    2812  66644  442   444  
R    642285 0     5000  0     64    2744  61323  479   481  
R    642286 0     5000  0     16    2328  61109  439   441  
R    642287 0     5000  0     40    2752  69227  471   473  
R    642288 0     5000  0     20    2536  60836  454   456  
R    642289 0     5000  0     16    2612  63580  440   442  
R    642290 0     5000  0     48    2528  72629  463   465  
R    642291 0     5000  0     68    2848  75262  498   500  
R    642292 0     5000  0     60    2688  77164  468   470  
R    642293 0     5000  0     0     2188  60922  458   460  
R    642294 0     5000  0     348   3124  79928  528   530  
R    642295 0     5100  0     1896  3128  62695  672   674  
R    642296 0     5024  0     8     4840  17110  90    91   
R    642297 0     5000  0     0     816   14232  1534  1539 
R    642298 0     5000  0     0     2636  52309  2070  2077 
R    642299 0     5000  0     60    2936  67005  375   377  
R    642300 0     5000  0     0     2064  52821  374   376  
R    642301 0     5000  0     228   3344  85321  539   541  
R    642302 0     5000  0     24    5360  54421  458   460  
R    642303 0     5384  52    1388  6548  66593  452   454  
R    642304 52    7936  876   7316  3832  192877 640   642  
R    642305 876   11152 0     5620  3656  262638 821   824  
R    642306 0     9276  0     6172  3768  125379 806   809  
R    642307 0     9920  0     0     6256  11091  702   705  
R    642308 0     5636  0     0     4976  266    119   120  
R    642309 0     5004  0     0     432   11035  1894  1900 
R    642310 0     5000  0     0     2608  43443  1270  1274 
R    642311 0     5000  0     36    2204  55368  333   334  
R    642312 0     5000  0     224   3180  77649  476   478  
R    642313 0     5000  0     20    2496  65989  431   433  
R    642314 0     5000  0     36    2512  71443  466   468  
R    642315 0     5000  956   1796  3916  62650  460   462  
R    642316 956   5712  0     1608  3676  136304 581   583  
R    642317 0     5284  0     1732  3704  123922 572   574  
R    642318 0     5436  0     2032  3652  133710 610   612  
R    642319 0     5684  0     1496  3652  129017 622   624  
R    642320 0     5148  0     640   3304  36529  602   604  
R    642321 0     5004  0     580   5404  23212  216   217  
R    642322 0     5980  0     0     272   4878   149   150  
R    642323 0     5000  0     0     1408  32432  2803  2812 
R    642324 0     5000  0     1184  3416  94751  420   422  
R    642325 0     5000  0     592   3260  94208  502   504  
R    642326 0     5000  0     92    3648  88419  486   488  
R    642327 0     5000  0     80    3052  76222  489   491  
R    642328 0     5000  0     0     4460  60005  429   431  
R    642329 0     5000  1108  2408  6260  77172  426   428  
R    642330 1108  8668  4     4992  3356  166166 622   624  
R    642331 0     8340  0     0     120   75     62    63   
R    642332 0     6984  0     0     68    51     14    15   
R    642333 0     6072  0     0     72    35     14    15   
R    642334 0     6072  0     0     80    31     12    13   
R    642335 0     5732  0     0     84    49     15    16   
R    642336 0     5684  0     0     68    31     12    13   
R    642337 0     5612  0     0     72    36     14    15   
R    642338 0     5636  0     0     68    35     15    16   
R    642339 0     5564  0     0     48    8      3     4    
R    642340 0     6224  0     0     4136  7      8     9    
R    642341 0     5596  0     0     100   11     13    14   
R    642342 0     5660  0     0     156   31     11    12   
R    642343 0     5752  0     0     76    24     12    13   
R    642344 0     5088  0     0     60    12     4     5    
R    642345 0     5032  0     0     40    8      3     4    
R    642346 0     5440  0     0     36    16     4     5    
R    642347 0     8616  0     0     44    13     4     5    
R    642348 0     5284  0     0     44    7      4     5    
R    642349 0     5444  0     0     56    8      4     5    
R    642350 0     5292  0     0     36    8      4     5    
R    642351 0     5140  0     0     44    4      3     4    
R    642352 0     5452  0     0     80    26     7     8    
R    642353 0     11144 0     0     108   5      3     4    
R    642354 0     5692  0     0     36    5      3     4    
R    642355 0     5520  0     0     40    4      3     4    
R    642356 0     5376  0     0     40    5      3     4    
R    642357 0     5228  0     0     44    4      3     4    
R    642358 0     5080  0     0     44    5      3     4    
R    642359 0     17148 0     0     48    6      3     4    
R    642360 0     18148 0     0     44    4      3     4    
R    642361 0     5064  0     0     64    12     4     5    
R    642362 0     5560  0     0     64    33     13    14   
R    642363 0     5228  0     0     36    7      4     5    
R    642364 0     5096  0     0     40    3      3     4    
R    642365 0     8136  0     0     64    19     8     9    
R    642366 0     9140  0     0     64    9      4     5    

312 transaction, each upto 8192 blocks
average:
52ms waiting for transaction
5692ms running transaction
52ms transaction was being locked
728ms flushing data (in ordered mode)
2748ms logging transaction
51724us average transaction commit time
61570 handles per transaction
565 blocks per transaction
567 logged blocks per transaction

(Hopefully i captured them the right way - should i have done this while the test was going on?)

Ingo

#334 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Ingo Molnar
Quoted 1 line · On Thu, 26 Mar 2009 15:03:12 +0100 Ingo Molnar <[email protected]> wrote:

Why arent the relatime patches upstream?

They have been for ages.

Quoted 1 line

Why isnt it the default?

That would have been a non-backward-compatible change.

#335 lore raw

relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 4 lines · Andrew Morton
Quoted 1 line · On Thu, 26 Mar 2009 15:03:12 +0100 Ingo Molnar <[email protected]> wrote:

Why arent the relatime patches upstream?

They have been for ages.

I assume Ingo means the patches to make relatime update atime at least once per day to ensure better compatibility with apps that do use or rely on access times.
These patches are already being included by several distros and, FWIW, Debian would like to see them upstream as well because we feel .

They were last submitted by Matthew Garrett:
http://lkml.org/lkml/2008/11/27/234
http://lkml.org/lkml/2008/11/27/235

Loads of people seem to want this, but even though it's been submitted at least twice and discussed even more often, it never gets anywhere.

Cheers,
FJP

#336 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 12 lines · On Thu, 26 Mar 2009 16:32:38 +0100 Frans Pop <[email protected]> wrote:
Quoted 4 lines · Andrew Morton
Quoted 1 line · On Thu, 26 Mar 2009 15:03:12 +0100 Ingo Molnar <[email protected]> wrote:

Why arent the relatime patches upstream?

They have been for ages.

I assume Ingo means the patches to make relatime update atime at least once per day to ensure better compatibility with apps that do use or rely on access times.
These patches are already being included by several distros and, FWIW, Debian would like to see them upstream as well because we feel .

Quoted 7 lines

They were last submitted by Matthew Garrett:
http://lkml.org/lkml/2008/11/27/234
http://lkml.org/lkml/2008/11/27/235

Loads of people seem to want this, but even though it's been submitted at least twice and discussed even more often, it never gets anywhere.

Hard-wiring a 24-hour interval into the core VFS for all mounted filesystems is dumb.

I (and others) pointed out that it would be better to implement this as a mount option. That suggestion was met with varying sillinesses and that is where things stand.

#337 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 3 lines · Andrew Morton

Hard-wiring a 24-hour interval into the core VFS for all mounted filesystems is dumb.

Umm.

I generally agree witht he "leave policy to user space" people, but this is an area where (a) user space has shown itself to not get it right (ie people don't do even the existing relatime because distros don't) and (b) what's the alternative?

Quoted 3 lines

I (and others) pointed out that it would be better to implement this as a mount option. That suggestion was met with varying sillinesses and that is where things stand.

I'd suggest first just doing the 24 hour thing, and then, IF user space actually ever gets its act together, and people care, and they _ask_ for a mount option, that's when it's worth doing.

Linus

maintainer
#338 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 4 lines · Linus Torvalds

I generally agree witht he "leave policy to user space" people, but this is an area where (a) user space has shown itself to not get it right (ie people don't do even the existing relatime because distros don't) and (b) what's the alternative?

I thought at least some distro's were adding relatime by default; I could be wrong, but I thought Ubuntu was doing this. Personally, I actually think that if we're going to give up on POSIX, I'll go all the way to noatime since it helps even more.

I've always thought the right approach would be to have a "atime dirty" flag, and update atime, but never flush it out to disk unless
(a) we're about to unmount the disk, or (b) we need to update some other inode in the same inode table block, or (c) we have memory pressure and we're trying to evict the inode from the inode cache. That way we get full POSIX compliance, without taking the I/O hit of atime updates. The atime updates get lost if we crash, but that's allowed by POSIX, and most people don't care about losing atime updates after a crash.

Since it's fully backwards (and POSIX) compatible, there would no question about enabling it by default.

- Ted

#339 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Words by Theodore Tso [Thu, Mar 26, 2009 at 12:30:26PM -0400]:

Quoted 8 lines · Linus Torvalds
Quoted 4 lines

I generally agree witht he "leave policy to user space" people, but this is an area where (a) user space has shown itself to not get it right (ie people don't do even the existing relatime because distros don't) and (b) what's the alternative?

I thought at least some distro's were adding relatime by default; I could be wrong, but I thought Ubuntu was doing this.

Yes.

Signature · 3 lines
Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc
----------------------------------------------------------------
"One man’s theology is another man’s belly laugh." -- Robert A. Heinlein
#340 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 11 lines · Jose Celestino

Words by Theodore Tso [Thu, Mar 26, 2009 at 12:30:26PM -0400]:

Quoted 8 lines · Linus Torvalds
Quoted 4 lines

I generally agree witht he "leave policy to user space" people, but this is an area where (a) user space has shown itself to not get it right (ie people don't do even the existing relatime because distros don't) and (b) what's the alternative?

I thought at least some distro's were adding relatime by default; I could be wrong, but I thought Ubuntu was doing this.

Yes.

No, not the "pure" relatime that's in the upstream kernel. And that's the whole point here. See my direct reply to Ted.

Cheers,
FJP

#341 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

In reply to Theodore Tso
Quoted 6 lines · Theodore Tso

I've always thought the right approach would be to have a "atime dirty" flag, and update atime, but never flush it out to disk unless
(a) we're about to unmount the disk, or (b) we need to update some other inode in the same inode table block, or (c) we have memory pressure and we're trying to evict the inode from the inode cache.

I tried to do that a few years ago (ok, probably more than a few by now).

It was surprisingly hard.

Some of it is absolutely trivial: we already have multiple "dirty" flags for the inode (I_DIRTY_SYNC vs I_DIRTY_DATASYNC vs I_DIRTY_PAGES). Adding a I_DIRTY_ATIME bit for unimportant data was trivial.

But at least back then, "sync_inode()" (or whatever) was called without the reason for doing the sync, so it was really hard to decide whether to write things out or not.

That may actually have changed these days. We now have that "writeback_control" thing that we pass around for all the IO.

Heh. I just looked back in the history. That writeback_control thing was added back in 2002, so it's a _really_ long time since I tried to do that whole atime thing.

Maybe it's really easy these days.

Linus

#342 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

In reply to Theodore Tso
Quoted 4 lines · Theodore Tso

I thought at least some distro's were adding relatime by default; I could be wrong, but I thought Ubuntu was doing this. Personally, I actually think that if we're going to give up on POSIX, I'll go all the way to noatime since it helps even more.

They indeed do have relatime by default, but *only because* they have the additional patch with the 24 hour limit in their kernel [1]. The same is true for Fedora IIUC.

Debian would like to activate it by default as well for new installations, but has so far been blocked from doing that because our kernel team has a policy of not including patches that are not upstream (or at least, in the process of being included upstream). And the Debian Installer team has so far felt that it would be irresponsible of activating it by default without this safeguard.

Cheers,
FJP

[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/199427
linux (2.6.24-12.18) hardy; urgency=low
[...]
* build/configs: Enable relatime config option for all flavors

#343 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

In reply to Linus Torvalds
Quoted 14 lines · On Thu, 26 Mar 2009 09:14:28 -0700 (PDT) Linus Torvalds…
Quoted 3 lines · Andrew Morton

Hard-wiring a 24-hour interval into the core VFS for all mounted filesystems is dumb.

Umm.

I generally agree witht he "leave policy to user space" people, but this is an area where (a) user space has shown itself to not get it right (ie people don't do even the existing relatime because distros don't) and (b) what's the alternative?

Quoted 8 lines
Quoted 3 lines

I (and others) pointed out that it would be better to implement this as a mount option. That suggestion was met with varying sillinesses and that is where things stand.

I'd suggest first just doing the 24 hour thing, and then, IF user space actually ever gets its act together, and people care, and they _ask_ for a mount option, that's when it's worth doing.

We wouldn't normally just enable the new feature by default because it changes kernel behaviour. Userspace needs to be changed in some manner to opt-in. One way it's `mount -o remount', the other way it's a poke in /proc.

#344 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 1 line · Andrew Morton

On Thu, 26 Mar 2009 09:14:28 -0700 (PDT) Linus Torvalds

Quoted 9 lines · <[email protected]> wrote:
Quoted 9 lines

I generally agree witht he "leave policy to user space" people, but this is an area where (a) user space has shown itself to not get it right (ie people don't do even the existing relatime because distros don't) and (b) what's the alternative?

Quoted 3 lines

I (and others) pointed out that it would be better to implement this as a mount option. That suggestion was met with varying sillinesses and that is where things stand.

Quoted 8 lines
Quoted 3 lines

I'd suggest first just doing the 24 hour thing, and then, IF user space actually ever gets its act together, and people care, and they _ask_ for a mount option, that's when it's worth doing.

We wouldn't normally just enable the new feature by default because it changes kernel behaviour. Userspace needs to be changed in some manner to opt-in. One way it's `mount -o remount', the other way it's a poke in /proc.

What change are you talking about here exactly? The "change relatime to have a 24 hour safeguard" of Matthes's first patch or the "enable relatime by default" options in the second patch?

For the first I don't think it's that big a deal as it is a change that makes the behavior of relatime safer and not riskier. Also, it's something people have argued should have been part of the initial functionality of relatime (it was part of the discussion back then), and finally for a lot of users it's already current functionality as major distros already do include the patch.

For the second, I can see your point and can understand reservations to make enabling relatime a kernel config option.

Speaking exclusively for myself, I would be happy enough if only the first of Matthew's patches would get accepted.

Cheers,
FJP

#345 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 32 lines · On Thu, 26 Mar 2009 18:12:04 +0100 Frans Pop <[email protected]> wrote:
Quoted 1 line · Andrew Morton

On Thu, 26 Mar 2009 09:14:28 -0700 (PDT) Linus Torvalds

Quoted 9 lines · <[email protected]> wrote:
Quoted 9 lines

I generally agree witht he "leave policy to user space" people, but this is an area where (a) user space has shown itself to not get it right (ie people don't do even the existing relatime because distros don't) and (b) what's the alternative?

Quoted 3 lines

I (and others) pointed out that it would be better to implement this as a mount option. That suggestion was met with varying sillinesses and that is where things stand.

Quoted 8 lines
Quoted 3 lines

I'd suggest first just doing the 24 hour thing, and then, IF user space actually ever gets its act together, and people care, and they _ask_ for a mount option, that's when it's worth doing.

We wouldn't normally just enable the new feature by default because it changes kernel behaviour. Userspace needs to be changed in some manner to opt-in. One way it's `mount -o remount', the other way it's a poke in /proc.

What change are you talking about here exactly? The "change relatime to have a 24 hour safeguard" of Matthes's first patch or the "enable relatime by default" options in the second patch?

For the first I don't think it's that big a deal as it is a change that makes the behavior of relatime safer and not riskier. Also, it's something people have argued should have been part of the initial functionality of relatime (it was part of the discussion back then), and finally for a lot of users it's already current functionality as major distros already do include the patch.

Quoted 6 lines

For the second, I can see your point and can understand reservations to make enabling relatime a kernel config option.

Speaking exclusively for myself, I would be happy enough if only the first of Matthew's patches would get accepted.

Oh, the feature itself is desirable. But the interface isn't.

- It's a magic number. Maybe someone runs tmpwatch twice per day, or weekly, or...

- That's fixable by making "24" tunable, but it's still a global thing. Better to make it per-fs.

- mount(8) is the standard way of tuning fs behaviour. There's no need to deviate from that here.

Note that none of this involves the default setting. With a per-mount tunable we can still make the default for each fs be "on, 24 hours" if we so decide.

#346 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 4 lines · Andrew Morton

Oh, the feature itself is desirable. But the interface isn't.

- It's a magic number. Maybe someone runs tmpwatch twice per day, or weekly, or...

There has to be a default. 24 hours is a sensible one.

Quoted 2 lines

- That's fixable by making "24" tunable, but it's still a global thing. Better to make it per-fs.

Patches welcome.

Quoted 2 lines

- mount(8) is the standard way of tuning fs behaviour. There's no need to deviate from that here.

Patches welcome.

When did we adopt a mindset that led to code having to satisfy every single user requirement before being accepted, rather than being happy with code that provides an incremental improvement over what exists already? If there are actually users who want to be able to tune this per filesystem then I'm sure someone (possibly even me) will write code to support them, but right now it just sounds like features for the sake of some sense of aesthetic correctness.

Signature · 1 line
Matthew Garrett | [email protected]
#347 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 8 lines · On Thu, 26 Mar 2009 18:49:00 +0000 Matthew Garrett <[email protected]> wrote:
Quoted 4 lines · Andrew Morton

Oh, the feature itself is desirable. But the interface isn't.

- It's a magic number. Maybe someone runs tmpwatch twice per day, or weekly, or...

There has to be a default. 24 hours is a sensible one.

"default" implies that it can be altered by some means.

Quoted 4 lines

When did we adopt a mindset that led to code having to satisfy every single user requirement before being accepted, rather than being happy with code that provides an incremental improvement over what exists already?

Shortcomings have been identified. Weaselly verbiage is not a suitable way of addressing shortcomings!

Yes, we could (and do) merge things as a halfway step. But when the features are visible to userspace we just can't do that - we have to get the interface right on day one, because interfaces are for ever.

(Bear in mind that kernel-does-X-once-per-day _is_ an interface)

Quoted 4 lines

If there are actually users who want to be able to tune this per filesystem then I'm sure someone (possibly even me) will write code to support them, but right now it just sounds like features for the sake of some sense of aesthetic correctness.

A hard-wired global 24-hours constant is in no way superior to a per-mount tunable. If we're going to do this we should do it in the best way we know, and we certainly should not lock ourselves into the inferior implementation for all time by exposing it to userspace.

#348 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

Quoted 5 lines · Andrew Morton
Quoted 2 lines · On Thu, 26 Mar 2009 18:49:00 +0000 Matthew Garrett <[email protected]> wrote:

On Thu, Mar 26, 2009 at 10:48:43AM -0700, Andrew Morton wrote:
There has to be a default. 24 hours is a sensible one.

"default" implies that it can be altered by some means.

You need a default even in the tunable case.

Quoted 7 lines
Quoted 4 lines

When did we adopt a mindset that led to code having to satisfy every single user requirement before being accepted, rather than being happy with code that provides an incremental improvement over what exists already?

Shortcomings have been identified. Weaselly verbiage is not a suitable way of addressing shortcomings!

What shortcomings? So far we have a hypothetical complaint that some users will want to choose a different value. Right now they have the choice of continuing to not use relatime. Things are no worse for them than they were previously.

Quoted 9 lines
Quoted 4 lines

If there are actually users who want to be able to tune this per filesystem then I'm sure someone (possibly even me) will write code to support them, but right now it just sounds like features for the sake of some sense of aesthetic correctness.

A hard-wired global 24-hours constant is in no way superior to a per-mount tunable. If we're going to do this we should do it in the best way we know, and we certainly should not lock ourselves into the inferior implementation for all time by exposing it to userspace.

I don't claim that it's superior, merely that it deals with all the use cases I've had to worry about and so is good enough. If it turns out that there are people in the real world who need the better version then I can write that code, but I'm not going to while it's a hypothetical.

Signature · 1 line
Matthew Garrett | [email protected]
#349 lore raw

Re: relatime: update once per day patches (was: ext3 IO latency measurements)

In reply to Andrew Morton

It seems to me that, rather than having the kernel maintain a timer (or multiple timers, one per mount) itself, it would make sense to have entries in /sys which, when written to, cause the file system layer to flush all atime data to the mounted volume.

Something like
/sys
/sys/atime
/sys/atime/all
/sys/atime/<mountpoint id>/flush

where <mountpoint id> would be the name of the file system (e.g. /sys/atime/usr/flush).

The only sticky part would be how to describe "/" in such a system.

(Better still would be a /sys/ system for each file system with the various parameters (e.g. uid, journal) as entries + an entry for flushing atime, but that is beyond the scope of this discussion.)

That would truly let userspace set policy, while the kernel provides mechanism. Thus, a script that depends upon atime being accurate could simply tickle the sysfs entries as needed before running.

#350 lore raw

PATCH 1/1 Allow relatime to update atime once a day

In reply to Linus Torvalds

Allow atime to be updated once per day even with relatime. This lets utilities like tmpreaper (which delete files based on last access time) continue working, making relatime a plausible default for distributions.

---

fs/inode.c +38 -9
index 0487ddb..057c92b 100644
@@ -1179,6 +1179,40 @@ sector_t bmap(struct inode * inode, sector_t block)
 }
 EXPORT_SYMBOL(bmap);
 
+/*
+ * With relative atime, only update atime if the previous atime is
+ * earlier than either the ctime or mtime or if at least a day has
+ * passed since the last atime update.
+ */
+static int relatime_need_update(struct vfsmount *mnt, struct inode *inode,
+			     struct timespec now)
+{
+
+	if (!(mnt->mnt_flags & MNT_RELATIME))
+		return 1;
+	/*
+	 * Is mtime younger than atime? If yes, update atime:
+	 */
+	if (timespec_compare(&inode->i_mtime, &inode->i_atime) >= 0)
+		return 1;
+	/*
+	 * Is ctime younger than atime? If yes, update atime:
+	 */
+	if (timespec_compare(&inode->i_ctime, &inode->i_atime) >= 0)
+		return 1;
+
+	/*
+	 * Is the previous atime value older than a day? If yes,
+	 * update atime:
+	 */
+	if ((long)(now.tv_sec - inode->i_atime.tv_sec) >= 24*60*60)
+		return 1;
+	/*
+	 * Good, we can skip the atime update:
+	 */
+	return 0;
+}
+
 /**
  *	touch_atime	-	update the access time
  *	@mnt: mount the inode is accessed on
@@ -1206,17 +1240,12 @@ void touch_atime(struct vfsmount *mnt, struct dentry *dentry)
 		goto out;
 	if ((mnt->mnt_flags & MNT_NODIRATIME) && S_ISDIR(inode->i_mode))
 		goto out;
-	if (mnt->mnt_flags & MNT_RELATIME) {
-		/*
-		 * With relative atime, only update atime if the previous
-		 * atime is earlier than either the ctime or mtime.
-		 */
-		if (timespec_compare(&inode->i_mtime, &inode->i_atime) < 0 &&
-		    timespec_compare(&inode->i_ctime, &inode->i_atime) < 0)
-			goto out;
-	}
 
 	now = current_fs_time(inode->i_sb);
+
+	if (!relatime_need_update(mnt, inode, now))
+		goto out;
+
 	if (timespec_equal(&inode->i_atime, &now))
 		goto out;
 
Signature · 1 line
Matthew Garrett | [email protected]
#351 lore raw

Re: [PATCH] Allow relatime to update atime once a day

Quoted 27 lines · Matthew Garrett

Allow atime to be updated once per day even with relatime. This lets utilities like tmpreaper (which delete files based on last access time) continue working, making relatime a plausible default for distributions.

fs/inode.c +23 -0
+/*
+ * With relative atime, only update atime if the previous atime is
+ * earlier than either the ctime or mtime or if at least a day has
+ * passed since the last atime update.
+ */
+static int relatime_need_update(struct vfsmount *mnt, struct inode *inode,
+			     struct timespec now)
+{
+
+	if (!(mnt->mnt_flags & MNT_RELATIME))
+		return 1;
+	/*
+	 * Is mtime younger than atime? If yes, update atime:
+	 */
+	if (timespec_compare(&inode->i_mtime, &inode->i_atime) >= 0)
+		return 1;
+	/*
+	 * Is ctime younger than atime? If yes, update atime:
+	 */
+	if (timespec_compare(&inode->i_ctime, &inode->i_atime) >= 0)
+		return 1;
+
+	/*

fs/inode.c sector_t bmap(struct inode * inode, sector_t block) · line 1214

Quoted 10 lines
fs/inode.c +10 -0
+	 * Is the previous atime value older than a day? If yes,
+	 * update atime:
+	 */
+	if ((long)(now.tv_sec - inode->i_atime.tv_sec) >= 24*60*60)
+		return 1;
+	/*
+	 * Good, we can skip the atime update:
+	 */
+	return 0;
+}

Good example of overcommented code.

#352 lore raw

Re: [PATCH] Allow relatime to update atime once a day

In reply to Matthew Garrett
Quoted 3 lines · On Thu, 26 Mar 2009 17:32:14 +0000 Matthew Garrett <[email protected]> wrote:

Allow atime to be updated once per day even with relatime. This lets utilities like tmpreaper (which delete files based on last access time) continue working, making relatime a plausible default for distributions.

And while I think forcing relatime on is a really dumb dangerous idea, providing it so you can enable it (or distro new releases can for new installs etc) is a *very good* one

maintainer
#353 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar

That's in good deal atime update latencies. We still appear to default to atime enabled in ext4.

That's stupid - only around 0.01% of all Linux systems relies on atime - and even those who rely on it would be well served by relatime. Why arent the relatime patches upstream? Why isnt it the default? They have been submitted several times.

The relatime patches are upstream. Both noatime and relatime are handled at the VFS layer, not at the per-filesystem level. The reason why it sin't the default is because of a desire for POSIX compliance, I suspect. Most distributions are putting relatime into /etc/fstab by default, but we haven't changed the mount option. It wouldn't be hard to add an "atime" option to turn on atime updates, and make either "noatime" or "relatime" the default. This is a simple patch to fs/namespace.c

Quoted 3 lines

Atime in its current mandatory do-a-write-for-every-read form is a stupid relic and we have been paying the fool's tax for it in the past 10 years.

No argument here. I use noatime, myself. It actually saves a lot more than relatime, and unless you are using mutt with local Maildir delivery, relatime isn't really that helpful, and the benefit of noatime is roughly double that of relatime vs normal atime update, in my measurements:

http://thunk.org/tytso/blog/2009/03/01/ssds-journaling-and-noatimerelatime/

- Ted

#354 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 3 lines · Theodore Tso

Most distributions are putting relatime into /etc/fstab by default, but we haven't changed the mount option.

I don't think this is true. Fedora certainly does not. Not in F10, not in F11.

And quite frankly, even if you then _manually_ put 'relatime' in /etc/fstab, the default Fedora install will totally ignore it. Why? Because it mounts the root partition while using initrd, and totally ignores /etc/fstab.

In other words, not only do distributions not do it, but you can't even do it by hand afterwards the sane way in the most common distro!

There really is reason for the kernel to just say "user space has sh*t for brains, and we'd better change the default - and if some distro really _thinks_ about it, and decides that they really want old-fashioned atime, let them do that".

Because right now, I do not believe for a moment that any distro that defaults to "atime" has spent lots of effort thinking about it. Quite the reverse. They probably default to "atime" because they spent no time AT ALL thinking about it.

Quoted 3 lines

It wouldn't be hard to add an "atime" option to turn on atime updates, and make either "noatime" or "relatime" the default. This is a simple patch to fs/namespace.c

Yes. I think we have to.

Quoted 7 lines

No argument here. I use noatime, myself. It actually saves a lot more than relatime, and unless you are using mutt with local Maildir delivery, relatime isn't really that helpful, and the benefit of noatime is roughly double that of relatime vs normal atime update, in my measurements:

http://thunk.org/tytso/blog/2009/03/01/ssds-journaling-and-noatimerelatime/

I do agree that "noatime" is better, but with "relatime" you at least are likely to not break anything. A program has to be _really_ odd to care about the "relatime" vs "atime" behavior.

Linus

#355 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 4 lines · Linus Torvalds

And quite frankly, even if you then _manually_ put 'relatime' in /etc/fstab, the default Fedora install will totally ignore it. Why? Because it mounts the root partition while using initrd, and totally ignores /etc/fstab.

That works here in openSUSE 11.1. The initrd remounts the rootfs with any options it founds in /etc/fstab.

Andreas.

Signature · 3 lines
Andreas Schwab, [email protected]
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
maintainer
#356 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Theodore Tso

Most distributions are putting relatime into /etc/fstab by default, but we haven't changed the mount option.

I don't think this is true. Fedora certainly does not. Not in F10, not in F11.

Ubuntu does. I thought Fedora had, but I stand corrected.

Quoted 4 lines

And quite frankly, even if you then _manually_ put 'relatime' in /etc/fstab, the default Fedora install will totally ignore it. Why? Because it mounts the root partition while using initrd, and totally ignores /etc/fstab.

You can, actually, but it requires hacking /boot/grub/menu.list. The boot command option "rootflags=noatime" should do it, if their initrd scripts are at all sane (and they honor rootfstype, so they probably do also honor rootflags).

The question is whether we can make Fedora 11 and OpenSUSE do the right thing now that this has become a highly visible discussion. I'm actually fairly optimistic on this front. (Maybe some distro folks will care to chime in on whether upcoming releases of F11 and OpenSuSE can be changed to DTRT?)

Actually, given where F11 is on its release schedule, I suspect it would be *easier* for them to make a change to default boot options in grub's menu.conf than it would be backport a kernel patch, since they will be releasing their beta release within the week, and their final development freeze is in less than two weeks.

- Ted

#357 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 5 lines · Theodore Tso

You can, actually, but it requires hacking /boot/grub/menu.list. The boot command option "rootflags=noatime" should do it, if their initrd scripts are at all sane (and they honor rootfstype, so they probably do also honor rootflags).

Not when I tried it. It just causes the initrd to be mounted noatime, and then the real root filesystem gets mounted atime again.

Maybe I screwed up. But I don't think so.

Quoted 5 lines

The question is whether we can make Fedora 11 and OpenSUSE do the right thing now that this has become a highly visible discussion. I'm actually fairly optimistic on this front. (Maybe some distro folks will care to chime in on whether upcoming releases of F11 and OpenSuSE can be changed to DTRT?)

And what's the argument for not doing it in the kernel?

The fact is, "atime" by default is just wrong.

Linus

#358 lore raw

PATCH 1/2 Add a strictatime mount option

Add support for explicitly requesting full atime updates. This makes it possible for kernels to default to relatime but still allow userspace to override it.

---
 fs/namespace.c        |    6 +++++-
 include/linux/fs.h    |    1 +
 include/linux/mount.h |    1 +
 3 files changed, 7 insertions(+), 1 deletions(-)
fs/namespace.c +5 -1
index 06f8e63..d0659ec 100644
@@ -780,6 +780,7 @@ static void show_mnt_opts(struct seq_file *m, struct vfsmount *mnt)
 		{ MNT_NOATIME, ",noatime" },
 		{ MNT_NODIRATIME, ",nodiratime" },
 		{ MNT_RELATIME, ",relatime" },
+		{ MNT_STRICTATIME, ",strictatime" },
 		{ 0, NULL }
 	};
 	const struct proc_fs_info *fs_infop;
@@ -1932,11 +1933,14 @@ long do_mount(char *dev_name, char *dir_name, char *type_page,
 		mnt_flags |= MNT_NODIRATIME;
 	if (flags & MS_RELATIME)
 		mnt_flags |= MNT_RELATIME;
+	if (flags & MS_STRICTATIME)
+		mnt_flags &= ~(MNT_RELATIME | MNT_NOATIME);
 	if (flags & MS_RDONLY)
 		mnt_flags |= MNT_READONLY;
 
 	flags &= ~(MS_NOSUID | MS_NOEXEC | MS_NODEV | MS_ACTIVE |
-		   MS_NOATIME | MS_NODIRATIME | MS_RELATIME| MS_KERNMOUNT);
+		   MS_NOATIME | MS_NODIRATIME | MS_RELATIME| MS_KERNMOUNT |
+		   MS_STRICTATIME);
 
 	/* ... and get the mountpoint */
 	retval = kern_path(dir_name, LOOKUP_FOLLOW, &path);
include/linux/fs.h +1 -0
index 92734c0..5bc81c4 100644
@@ -141,6 +141,7 @@ struct inodes_stat_t {
 #define MS_RELATIME	(1<<21)	/* Update atime relative to mtime/ctime. */
 #define MS_KERNMOUNT	(1<<22) /* this is a kern_mount call */
 #define MS_I_VERSION	(1<<23) /* Update inode I_version field */
+#define MS_STRICTATIME	(1<<24) /* Always perform atime updates */
 #define MS_ACTIVE	(1<<30)
 #define MS_NOUSER	(1<<31)
 
include/linux/mount.h +1 -0
index cab2a85..51f55f9 100644
@@ -27,6 +27,7 @@ struct mnt_namespace;
 #define MNT_NODIRATIME	0x10
 #define MNT_RELATIME	0x20
 #define MNT_READONLY	0x40	/* does the user want this to be r/o? */
+#define MNT_STRICTATIME 0x80
 
 #define MNT_SHRINKABLE	0x100
 #define MNT_IMBALANCED_WRITE_COUNT	0x200 /* just for debugging */
Signature · 1 line
Matthew Garrett | [email protected]
#359 lore raw

PATCH 2/2 Make relatime default

Change the default behaviour of the kernel to use relatime for all filesystems. This can be overridden with the "strictatime" mount option.

---
 fs/namespace.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)
fs/namespace.c +3 -2
index d0659ec..f0e7530 100644
@@ -1920,6 +1920,9 @@ long do_mount(char *dev_name, char *dir_name, char *type_page,
 	if (data_page)
 		((char *)data_page)[PAGE_SIZE - 1] = 0;
 
+	/* Default to relatime */
+	mnt_flags |= MNT_RELATIME;
+
 	/* Separate the per-mountpoint flags */
 	if (flags & MS_NOSUID)
 		mnt_flags |= MNT_NOSUID;
@@ -1931,8 +1934,6 @@ long do_mount(char *dev_name, char *dir_name, char *type_page,
 		mnt_flags |= MNT_NOATIME;
 	if (flags & MS_NODIRATIME)
 		mnt_flags |= MNT_NODIRATIME;
-	if (flags & MS_RELATIME)
-		mnt_flags |= MNT_RELATIME;
 	if (flags & MS_STRICTATIME)
 		mnt_flags &= ~(MNT_RELATIME | MNT_NOATIME);
 	if (flags & MS_RDONLY)
Signature · 1 line
Matthew Garrett | [email protected]
#360 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 3 lines · On Thu, 26 Mar 2009 17:53:14 +0000 Matthew Garrett <[email protected]> wrote:

Change the default behaviour of the kernel to use relatime for all filesystems. This can be overridden with the "strictatime" mount option.

NAK this again

There is an expected behaviour pattern that is standards compliant and suddenly breaking that on people when they upgrade could cause serious problems in some server environments.

What you propose is basically a bogus ABI change. Fix it in user space (in fact all the distros *are* so this patch is silly and pointless)

#361 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 2 lines · Alan Cox

NAK this again

And I don't care.

If the distro's had done this right in the year+ that this has been in, I m ight consider your NAK to have some weight.

As it is, we know that didn't happen, and we've had three different people from different distributions say that they wanted to use relatime anyway, so it's now the default in my git tree.

If you want to live in some dark ages, you can do so with the "strictatime" thing.

Linus

#362 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 9 lines · Linus Torvalds

If the distro's had done this right in the year+ that this has been in, I might consider your NAK to have some weight.

As it is, we know that didn't happen, and we've had three different people from different distributions say that they wanted to use relatime anyway, so it's now the default in my git tree.

If you want to live in some dark ages, you can do so with the "strictatime" thing.

I obviously welcome the inclusion of the first patch and I'm neutral about the second one, but I'm not at all sure that making relatime the default (yet) is the right thing, especially as util-linux' mount command does not yet even support that "strictatime" thing. Shouldn't that at least happen first (and have been supported in distro's stable releases for some time)?

I guess users and distros can still elect not to set it as default, but it still seems a bit like going from one extreme to another.

#363 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 3 lines · Frans Pop

I guess users and distros can still elect not to set it as default, but it still seems a bit like going from one extreme to another.

Why? RELATIME has been around since 2006 now. Nothing has happened. People who think "we should leave it up to user land" lost their credibility long ago.

Linus

#364 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 1 line

Why? RELATIME has been around since 2006 now.

The workable fixes to relatime (the always update once per 24 hours) you only just comitted - and did come from a vendor.

It also looks btw that we don't want to have a "relatime" option and a "strictatime" option and a "relatimebutdoitevery24hrs" option.

All three of these are the same thing so it should (regardless of default choice) be

relatime=n

n = 0 ('update if its more than 0 seconds out of date') = strictatime
n = MAXINT (basically equals relatime)
n = 24hrs (the new 'fixed' relatime but not too relative)

n = anything else - user tuned

Alan

#365 lore raw

Re: [PATCH 2/2] Make relatime default

In reply to Linus Torvalds
Quoted 7 lines · you
Quoted 2 lines · Frans Pop

I guess users and distros can still elect not to set it as default, but it still seems a bit like going from one extreme to another.

Why? RELATIME has been around since 2006 now. Nothing has happened. People who think "we should leave it up to user land" lost their credibility long ago.

As I think Andrew already noted, the discussion today is largely a rehash of one in 2007, summarized by lwn [1] and kerneltrap [2]. That's also when Ingo first submitted the patch (based on a suggestion from you). But it has been blocked by others twice, and for exactly the same reasons.

relatime *without* the 24-hour safeguard has unanimously been deemed unsuitable as a default by distros.
So the real problem is that nobody ever did the work needed to make Ingo's original patch acceptable to the fs devs and the resulting stalemate for the last 1 3/4 years. IMO that's mainly a kernel community failure and not a user land failure. You've now at least broken that stalemate.

Your statement is also not quite true. At least Ubuntu has had relatime enabled by default for new installations for a couple of releases. And AFAICT they now even have it enabled by default now in their kernel config, but I'm not entirely sure.

For Debian Lenny (current stable release), relatime is a mount option that can be activated during new installs (admittedly only if you look hard enough). All that would have been needed for Debian to enable relatime by default for new installs was to have something like the *first* patch of the three you've now committed to have been included in 2.6.26.

Cheers,
FJP

[1] http://lwn.net/Articles/244829/
[2] http://kerneltrap.org/node/14148

#366 lore raw

Re: [PATCH 2/2] Make relatime default

In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Alan Cox

NAK this again

And I don't care.

If the distro's had done this right in the year+ that this has been in, I m ight consider your NAK to have some weight.

'No changes of ABI in stable series?' If you tweaked defaults for ext4, before it was widely used... that would be acceptable I guess. By breaking old setups with kernel change is bad.

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#367 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 2 lines · Pavel Machek

'No changes of ABI in stable series?'

(a) we don't have a stable series any more
and
(b) this isn't an abi change, it's a system management change.

If you don't think we can make those, then I assume that you also claim that we can't do thigns like commit 1b5e62b42, which doubled the writeback dirty thresholds, or any of the things that changed how dirty accounting was done in the first place?

I would _love_ for distros to do the sane thing, but they don't. That's a fact.

Linus

#368 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 9 lines · Linus Torvalds
Quoted 2 lines · Pavel Machek

'No changes of ABI in stable series?'

(a) we don't have a stable series any more
and
(b) this isn't an abi change, it's a system management change.

Well, stat() syscall no longer returns sane value in st_atime, while all the userland stayed the same; only kernel changed. I believe that is ABI change.

Quoted 4 lines

If you don't think we can make those, then I assume that you also claim that we can't do thigns like commit 1b5e62b42, which doubled the writeback dirty thresholds, or any of the things that changed how dirty accounting was done in the first place?

Writeback dirty thresholds will only change timing, that was not part of ABI. st_atime field is.

Quoted 2 lines

I would _love_ for distros to do the sane thing, but they don't. That's a fact.

But is this a way to do it? Are there maybe better ways?

a) Publicly call those distros broken?

b) Add nasty printk() to mount to force their attention?

	bb) Add nasty printk() and mdelay(1000) to really force their
		attention? :-)

c) Modify mount command to do the dirty work instead of changing default in kernel?

[as mountflags are not passed as a string by sys_mount(), you are creating pretty nasty situation for users; users with old distro but new kernel will not be even able to get old behaviour back w/o updating /sbin/mount. This would prevent it].
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#369 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 4 lines · Pavel Machek

Well, stat() syscall no longer returns sane value in st_atime, while all the userland stayed the same; only kernel changed. I believe that is ABI change.

No.

If you want the old abi, use "strictatime" in your /etc/fstime.

No ABI changed. Just the default mount options changed to be what most people (especially non-specialists) would likely want.

Deal with it.

Linus

#370 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Pavel Machek

Well, stat() syscall no longer returns sane value in st_atime, while all the userland stayed the same; only kernel changed. I believe that is ABI change.

No.

If you want the old abi, use "strictatime" in your /etc/fstime.

~~~~~~~~~~~~~~~~~~~~~~~~

So you agree it is an ABI change :-).

Quoted 2 lines

No ABI changed. Just the default mount options changed to be what most people (especially non-specialists) would likely want.

Maybe.

But lets see what awaits me with 2.6.30-rc1 update:

root@amd:~# mount /data -oremount,noatime
root@amd:~# mount /data -oremount,strictatime
mount: /data not mounted already, or bad option

...oh no, my mount is too old. My mount seems to be up-to-date with debian testing. Should I have to install mount from sources just to keep the compatible system settings?

There has to be a better way.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#371 lore raw

Re: [PATCH 2/2] Make relatime default

In reply to Alan Cox
Quoted 8 lines · Alan Cox
Quoted 3 lines · On Thu, 26 Mar 2009 17:53:14 +0000 Matthew Garrett <[email protected]> wrote:

Change the default behaviour of the kernel to use relatime for all filesystems. This can be overridden with the "strictatime" mount option.

NAK this again

NAK but because if we change the default it is better to change it to the real thing: noatime.

I think this can be solved in userland but perhaps changing this in kernel would be a stronger message that atime is officially obsoleted. (and nothing will break, not even mutt users will notice, and if they really do it won't be anything more than aesthetical)

About the open(destination, O_TRUNC); write; close, I think it's not worth changing the kernel or the VM in any way to hide buggy programming like that, to the contrary it's great it was found early on (instead of being filed as some obscure not reproducible bug lost in some bugzilla and hitting once in a while with an unlucky power-loss during boot). But solving this bug with fsync so it works for writeback mode too, would make me prefer to gamble and run the the buggy version ;). Not sure if it worth providing any ordering guarantee more than 'ordered' mode in the long term or some proper barrier, but at least ordered mode already allows for renaming the tempfile to be enough and that is clearly the best tradeoff. fsync really should be used only to avoid total loss of information (like when we need to avoid losing the delivery of an email after the smpt client is told the email was already received by the smtp server).

Using fsync to tell the kernel in what order to write is dirty pagecache data to disk, is as inefficient as driving a car to travel a 10 meters distance, so rightfully people isn't using it for this even if it's the only way it could work for writeback and ext2 too.

#372 lore raw

Re: [PATCH 2/2] Make relatime default

Quoted 10 lines · Andrea Arcangeli
Quoted 8 lines · Alan Cox
Quoted 3 lines · On Thu, 26 Mar 2009 17:53:14 +0000 Matthew Garrett <[email protected]> wrote:

Change the default behaviour of the kernel to use relatime for all filesystems. This can be overridden with the "strictatime" mount option.

NAK this again

Quoted 7 lines

NAK but because if we change the default it is better to change it to the real thing: noatime.

I think this can be solved in userland but perhaps changing this in kernel would be a stronger message that atime is officially obsoleted. (and nothing will break, not even mutt users will notice, and if they really do it won't be anything more than aesthetical)

Actually I liked the previous relatime (without the 24h hack) pretty much. It would have preserved the atime functionality without making something like slocate dirty huge parts of the fs daily.

I vote for relatime-without-24h-hack !

Xav

#373 lore raw

Re: [PATCH 2/2] Make relatime default

In reply to Andrea Arcangeli
Quoted 9 lines · Andrea Arcangeli
Quoted 7 lines · Alan Cox
Quoted 3 lines · On Thu, 26 Mar 2009 17:53:14 +0000 Matthew Garrett <[email protected]> wrote:

Change the default behaviour of the kernel to use relatime for all filesystems. This can be overridden with the "strictatime" mount option.

NAK this again

Quoted 8 lines

NAK but because if we change the default it is better to change it to the real thing: noatime.

I think this can be solved in userland but perhaps changing this in kernel would be a stronger message that atime is officially obsoleted. (and nothing will break, not even mutt users will notice, and if they really do it won't be anything more than aesthetical)

This makes the assumption that atime is not used, which may be true on your system but isn't on others. I regularly move data between faster and slower storage based on atime, and promote reactivated projects to something faster, while retiring inactive project data elsewhere. Other admins use it to identify unused files which are candidates for backup to offline media or the bit bucket.

Let people who want that behavior specify it for existing filesystems, if you want to remove functionality from ext4 or btrfs or some thing place where people have no existing expectations, I still think it's wrong, but I couldn't say I think it might break anything.

I did a patch a few years ago which only updated atime on open and write, and that worked about as well as relatime, the inode update on open is cheap, the head is already there, and it was only slightly slower than noatime. The were no programs which kept files open for days and just read them. The the only storage hierarchy was "slow and cheap." ;-)

Signature · 3 lines
Bill Davidsen <[email protected]>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot
#374 lore raw

Re: [PATCH 1/2] Add a strictatime mount option

In reply to Matthew Garrett
Quoted 5 lines · On Thu, 26 Mar 2009 17:49:56 +0000 Matthew Garrett <[email protected]> wrote:

Add support for explicitly requesting full atime updates. This makes it possible for kernels to default to relatime but still allow userspace to override it.

NAK this is unneccessary complication from a broken ABI change that isn't safe to make anyway.

#375 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Linus Torvalds
Quoted 3 lines

And what's the argument for not doing it in the kernel?

The fact is, "atime" by default is just wrong.

It probably was a wrong default - twenty years ago. Actually it may well have been a wrong default in Unix v6 8)

However
- atime behaviour is SuS required
- there are users with systems out there using atime and dependant on proper atime

So we can't change the ABI on them any more than we can decide that next week write() should return short values on writes to disk interrupted by signals...

Letting distros flip to relatime means new installs and gradual migration occurs and nobody gets spectacularly blown up when their archiving system, their usage profiling and disk balancing tools and the like go wrong.

#376 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 9 lines · Alan Cox
Quoted 3 lines

And what's the argument for not doing it in the kernel?

The fact is, "atime" by default is just wrong.

It probably was a wrong default - twenty years ago. Actually it may well have been a wrong default in Unix v6 8)

However
- atime behaviour is SuS required

SuS says "An implementation may update fields that are marked for update immediately, or it may update such fields periodically. At an update point in time, any marked fields shall be set to the current time and the update marks shall be cleared" but doesn't appear to specify any kind of time limit. A conforming implementation could wait a century before performing the update. So while relatime doesn't conform, the practical difference is meaningless. You can't depend on atime being updated in a timely manner.

Signature · 1 line
Matthew Garrett | [email protected]
#377 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 3 lines

before performing the update. So while relatime doesn't conform, the practical difference is meaningless. You can't depend on atime being updated in a timely manner.

POSIX says a disk write interrupted by a signal can be a short write. If you do this in practice all hell breaks loose.

A conforming implementation needs to conform with expectations not just play lawyer games with users systems.

Alan

#378 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 9 lines · Alan Cox
Quoted 3 lines

before performing the update. So while relatime doesn't conform, the practical difference is meaningless. You can't depend on atime being updated in a timely manner.

POSIX says a disk write interrupted by a signal can be a short write. If you do this in practice all hell breaks loose.

A conforming implementation needs to conform with expectations not just play lawyer games with users systems.

I agree, but arguing for something on the basis of a spec isn't terribly convincing if the spec allows effectively identical behaviour. SuS isn't a relevant consideration when it comes to deciding default atime policy.

Signature · 1 line
Matthew Garrett | [email protected]
#379 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 3 lines

I agree, but arguing for something on the basis of a spec isn't terribly convincing if the spec allows effectively identical behaviour. SuS isn't a relevant consideration when it comes to deciding default atime policy.

I'd says its or minor relevance. The default expected behaviour we had last week is however of major relevance and that is my big concern.

#380 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Alan Cox
Quoted 9 lines · Alan Cox
Quoted 3 lines

before performing the update. So while relatime doesn't conform, the practical difference is meaningless. You can't depend on atime being updated in a timely manner.

POSIX says a disk write interrupted by a signal can be a short write. If you do this in practice all hell breaks loose.

A conforming implementation needs to conform with expectations not just play lawyer games with users systems.

Is this the same Alan Cox who thought a couple of months ago that having an insanely low default maximum number epoll instances was a reasonable answer to a theoretical DoS risk, despite it breaking pretty much every reasonable user of the epoll interface?

Bron ( what stable interface? )

#381 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 4 lines

Is this the same Alan Cox who thought a couple of months ago that having an insanely low default maximum number epoll instances was a reasonable answer to a theoretical DoS risk, despite it breaking pretty much every reasonable user of the epoll interface?

In the short term yes - because security has to be a very high priority. Lesser of two evils.

Alan

#382 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 7 lines · Alan Cox
Quoted 4 lines

Is this the same Alan Cox who thought a couple of months ago that having an insanely low default maximum number epoll instances was a reasonable answer to a theoretical DoS risk, despite it breaking pretty much every reasonable user of the epoll interface?

In the short term yes - because security has to be a very high priority. Lesser of two evils.

So turn the machine off.

It seems to me that having atime turned on is a DoS risk. Any punk can cause lots of disk IO that will make everyone else's fsync's turn into molasses simply by reading lots of files. ZOMG (as the kiddies of today would say) - we'd better fix this DoS risk by disabling or rate limiting this dangeous vector (eleventyone!)

Bron ( ok, I'm getting a bit silly here - but if we blocked every
potential DoS by making sure a single user could only use a small percentage of the machine's total capacity at maximum... )

#383 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 1 line

Bron ( ok, I'm getting a bit silly here

Yes you are - completely.

#384 lore raw

Re: ext3 IO latency measurements

In reply to Alan Cox
Quoted 9 lines · Alan Cox
Quoted 3 lines

And what's the argument for not doing it in the kernel?

The fact is, "atime" by default is just wrong.

It probably was a wrong default - twenty years ago. Actually it may well have been a wrong default in Unix v6 8)

However
- atime behaviour is SuS required

so I propose an other mount option along to strictatime:
nowatime: it give the actual time as atime:
it is totally useless, but fast *and* POSIX compatible:
- no disk writes on accesses
- POSIX doesn't mandate the behaviour of other processes, so we simulate that fs are scanned at every fs-tick.
- IMHO more programs break, but in this case only the POSIX incompatible programs.

Quoted 2 lines

- there are users with systems out there using atime and dependant on proper atime

This is the real problem.

ciao
cate

#385 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

And quite frankly, even if you then _manually_ put 'relatime' in /etc/fstab, the default Fedora install will totally ignore it. Why? Because it mounts the root partition while using initrd, and totally ignores /etc/fstab.

That's a difference between Fedora's initrd and Debian/Ubuntu's initramfs-tools then. We do respect the mount options in fstab for the root partition when root is mounted from the initrd:

$ cat /etc/fstab | grep " / "
/dev/mapper/main-root /      ext3    relatime,errors=remount-ro 0       1
$ mount | grep " / "
/dev/mapper/main-root on / type ext3 (rw,relatime,errors=remount-ro)
$ cat /proc/cmdline
root=/dev/mapper/main-root ro vga=791 quiet

Cheers,
FJP

#386 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

And quite frankly, even if you then _manually_ put 'relatime' in /etc/fstab, the default Fedora install will totally ignore it. Why? Because it mounts the root partition while using initrd, and totally ignores /etc/fstab.

It should honor /etc/fstab changes, if the initramfs is rebuilt after the change is made. If it doesn't, that's a bug.

Bill

#387 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 8 lines · Bill Nottingham
Quoted 4 lines · Linus Torvalds

And quite frankly, even if you then _manually_ put 'relatime' in /etc/fstab, the default Fedora install will totally ignore it. Why? Because it mounts the root partition while using initrd, and totally ignores /etc/fstab.

It should honor /etc/fstab changes, if the initramfs is rebuilt after the change is made. If it doesn't, that's a bug.

Why the hell should I rebuild initramfs?

Anyway, I fixed it. I don't use initramfs any more, after all the idiocies it has done. I had to make everything primary partitions in order to do that, but hey, that solved a lot of other problems too, so that was no loss.

Linus

#388 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 10 lines · Linus Torvalds
Quoted 8 lines
Quoted 4 lines · Linus Torvalds

And quite frankly, even if you then _manually_ put 'relatime' in /etc/fstab, the default Fedora install will totally ignore it. Why? Because it mounts the root partition while using initrd, and totally ignores /etc/fstab.

It should honor /etc/fstab changes, if the initramfs is rebuilt after the change is made. If it doesn't, that's a bug.

Why the hell should I rebuild initramfs?

Well, it's got to find the root fs options somewhere. Pulling them from the modified /etc/fstab in the root fs before you mount it, well...

As for why fstab options aren't applied with remount once the root fs has been mounted, 1) historical reasons 2) someone specifies 'data=writeback' or similar can't-be-applied-with-remount flag in /etc/fstab, and then mount refuses to remount it at all, and the system refuses to boot. Arguably pilot error, of course.

Bill

#389 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 3 lines · Bill Nottingham

Well, it's got to find the root fs options somewhere. Pulling them from the modified /etc/fstab in the root fs before you mount it, well...

Umm.

The _only_ sane thng to do is to mount the root read-only from initramfs, and then re-mount it with the options in the /etc/fstab later when you re-mount it read-write _anyway_ (which may possibly be immediately, of course).

Anybody who thinks you should re-write initramfs for something like this really hasn't spent a single second thinking about it.

Linus

#390 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 9 lines · Linus Torvalds
Quoted 2 lines

Well, it's got to find the root fs options somewhere. Pulling them from the modified /etc/fstab in the root fs before you mount it, well...

Umm.

The _only_ sane thng to do is to mount the root read-only from initramfs, and then re-mount it with the options in the /etc/fstab later when you re-mount it read-write _anyway_ (which may possibly be immediately, of course).

Sure, and as said, as soon as you try to specify journal options (and possibly others), this immediately fails. You can apply the options one at a time, and decide some aren't fatal, or you can actually have your later remount have code to drop specific options, requiring implementation knowledge of any filesystem to be used. Or you say people who specify journal options in fstab don't get to boot.

But if you blindly attempt to apply fstab options later in the remount, some options will break.

Bill

#391 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Bill Nottingham
Quoted 8 lines · On Thu, 26 Mar 2009 13:32:38 -0400 Bill Nottingham <[email protected]> wrote:
Quoted 4 lines · Linus Torvalds

And quite frankly, even if you then _manually_ put 'relatime' in /etc/fstab, the default Fedora install will totally ignore it. Why? Because it mounts the root partition while using initrd, and totally ignores /etc/fstab.

It should honor /etc/fstab changes, if the initramfs is rebuilt after the change is made. If it doesn't, that's a bug.

Surely it should also look at the real /etc/fstab after mounting root r/o and then flip the options needed so you don't have to.

maintainer
#392 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

I still see similarly bad latencies in Vim:

When you say "similarly bad", how many seconds were you seeing? I understand that from the user's perspective, the 120 seconds you saw with ext3 isn't going to be that different from 15 seconds (which seems to be the maximum commit time in the jbd2 history file you sent me), but I'm curious if what you saw was just as bad with ext4, or was it somewhat better (i.e., 120 seconds vs 15 or so). Or were you also seeing a net time to save the file using vim of around 120 seconds with ext4?

Ext4 in nodelalloc mode is mostly similar to ext3, but it does have some improvements, such as a slightly elevated I/O priority for kjournald, and the ext4's writepage doesn't take the journal handle as it does in ext3. (That's why I was confused about Linus's assertion about ext3 waiting on the journal; ext4 doesn't any more, and I had ext4 on the brain.) Unfortunately, we don't have the /proc/fs/jbd/<dev>/history for ext3, so it would be interesting to compare whether the vim save latencies were improved or not with ext4. If they are, then it might be worth Jan's time to fix up ext3's writepage to not try request journal access if it's not needed. It might also be worth backporting ext4's slightly raised I/O priority patch.

Another thing that's worth trying. Suppose you use ionice to raise the priority of kjournald to a real-time I/O priority (which is what Arjan's patch does). How much does that help? Is it more or less compared to what we're seeing with ext4's slightly reaised I/O priority.

And if we mount the filesystem noatime, does that change the results significantly as far as Vim write latencies and the jbd2 history file?

Quoted 2 lines

The read-test is somewhat better. There are occasional blips of 4-5 seconds:

Presumably these go away once we mount the filesystem noatime, right?

- Ted

#393 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 5 lines · Theodore Tso
Quoted 3 lines · Ingo Molnar

I still see similarly bad latencies in Vim:

Quoted 8 lines

When you say "similarly bad", how many seconds were you seeing? I understand that from the user's perspective, the 120 seconds you saw with ext3 isn't going to be that different from 15 seconds (which seems to be the maximum commit time in the jbd2 history file you sent me), but I'm curious if what you saw was just as bad with ext4, or was it somewhat better (i.e., 120 seconds vs 15 or so). Or were you also seeing a net time to save the file using vim of around 120 seconds with ext4?

It was in the minute range, iirc. It was totally unusable interactively. I wrote the vim-test script during that workload and i'm still getting annoyed thinking back at the experience. Is that enough to consider it bad? :-)

This isnt me streaming gigs of data in and out of the system dirtying 90% of all RAM. This is a trivial workload barely scratching the RAM and CPU capabilities of the system.

Do you have a non-tweaked default Fedora install somewhere? These kinds of delays in Vim were easily reproducible in the last 5 years and i saw it reported frequently on various lists.

Have you tried to reproduce it? Have you tried CONFIG_LATENCYTOP? We implemented that kernel feature specifically to make it easy for developers to instrument their kernel and keep system latencies down. This isnt some oddball workload or oddball system. These latencies are reproducible on just about any Linux development system i ever tried with ext3.

And the thing is, to 99.9% of the people it doesnt matter how scalable we are to 16000 CPUs or whether a directory with 1 million files in it takes 10 or 200 msecs to parse.

But it gives a permanent impression how much delay basic everyday operations on the system have. So latency optimizations (and i use the term losely here) have to be the primary development metric in Linux IMHO. ( If i were doing filesystem development i'd sure already have my low-latency filesystem patchset ;-)

Ingo

maintainer
#394 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 3 lines · Ingo Molnar

This isnt me streaming gigs of data in and out of the system dirtying 90% of all RAM. This is a trivial workload barely scratching the RAM and CPU capabilities of the system.

Have you tried with maxcpus set to say, 2? My guess is you won't see the problems in that case. So I'm not sure saying "barely scratching the CPU capabilities of the system" is completely fair. I can probably get be able to get temporary access to a 16 CPU system, but that's not the kind of system that I normally get to use for my kernel devleopment.

Quoted 6 lines

Have you tried to reproduce it? Have you tried CONFIG_LATENCYTOP? We implemented that kernel feature specifically to make it easy for developers to instrument their kernel and keep system latencies down. This isnt some oddball workload or oddball system. These latencies are reproducible on just about any Linux development system i ever tried with ext3.

My normal development is not all that different from yours (make -j<numcpus*2>) and I do edit and save files while the compile is going. I use emacs, but it calls fsync() when saving files, just like vim does. The big difference is that for me, numcpus is normally 2. And my machine has 4 gigs of memory, not 12 gigs. So I don't see these problems. I agree that what you have isn't an "oddball workload"; as far as whether it is an "oddball system", it is certainly a system I would lust after. And I acknowledge the world is a bit different from when Linus declared that 99% of the world was 1 or 2 CPU's. I suspect the percentage of machines with 16 CPU's is still somewhat small, though.

So I'll try to reproduce it on a 16 CPU system, when I have a chance --- but it's something that I'm going to have to borrow and try to get remote access to play with such a system. Clearly your employer is way more generous with equipment than mine is, at least for personal development machines. :-)

In the meantime, if you could run some of the tests and vary some of the variables I requested, I'd appreciate it, and thank you for your help. Otherwise, I'll try to run them when I get remote access to such a machine where I'm allowed to replace kernels and mount random test filesystems.

- Ted

P.S. Another interesting test would be to plot the vim save latencies versus the number of CPU's enabled when running the kernel build workload.

P.P.S. I assume there's no way you could give me remote ssh access to your nice 16-way machine? :-)

#395 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 7 lines · Theodore Tso
Quoted 3 lines · Ingo Molnar

This isnt me streaming gigs of data in and out of the system dirtying 90% of all RAM. This is a trivial workload barely scratching the RAM and CPU capabilities of the system.

Have you tried with maxcpus set to say, 2? My guess is you won't see the problems in that case.

Note, my previous devel box was a single socket quad and it had such delays all the time as well. Havent tried it on a dual-core. (i dont have dual-core systems with enough RAM to be able to build a kernel purely in RAM)

Ingo

#396 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Theodore Tso
Quoted 11 lines · On Thu, 26 Mar 2009 19:59:36 -0400 Theodore Tso <[email protected]> wrote:
Quoted 3 lines · Ingo Molnar

This isnt me streaming gigs of data in and out of the system dirtying 90% of all RAM. This is a trivial workload barely scratching the RAM and CPU capabilities of the system.

Have you tried with maxcpus set to say, 2? My guess is you won't see the problems in that case. So I'm not sure saying "barely scratching the CPU capabilities of the system" is completely fair. I can probably get be able to get temporary access to a 16 CPU system, but that's not the kind of system that I normally get to use for my kernel devleopment.

Nope, I saw this with my dual CPU machine too (before I upgraded to quad core)... Just doing kernel builds and/or icecream and/or VMware. It didn't take much. I have 8G of memory now but I used to have less (3G iirc) and saw it then too.

Quoted 11 lines

My normal development is not all that different from yours (make -j<numcpus*2>) and I do edit and save files while the compile is going. I use emacs, but it calls fsync() when saving files, just like vim does. The big difference is that for me, numcpus is normally 2. And my machine has 4 gigs of memory, not 12 gigs. So I don't see these problems. I agree that what you have isn't an "oddball workload"; as far as whether it is an "oddball system", it is certainly a system I would lust after. And I acknowledge the world is a bit different from when Linus declared that 99% of the world was 1 or 2 CPU's. I suspect the percentage of machines with 16 CPU's is still somewhat small, though.

I'm surprised you haven't seen this then... Maybe your journal is bigger? Or some other config difference...

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#397 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Theodore Tso
Quoted 1 line · On Thursday March 26, [email protected] wrote:

Ingo,

.....

Quoted 46 lines
Quoted 6 lines

Oh, and while at it - also a job control complaint. I tried to Ctrl-C the above script:

I had to hit Ctrl-C numerous times before Bash would honor it. This to is a very common thing on large SMP systems.

Well, the script you sent runs the compile in the background. It did:

Quoted 6 lines

while :; do
date
make mrproper 2>/dev/null >/dev/null
make defconfig 2>/dev/null >/dev/null
make -j32 bzImage 2>/dev/null >/dev/null
done &

         ^^

So there would have been nothing to ^C; I assume you were running this with a variant that didn't have the ampersand, which would have run the whole shell pipeline in a detached background process?

In any case, the workaround for this is to ^Z the script, and then "kill %" it.

I'm pretty sure this is actually a bash problem. When you send a Ctrl-C, it sends a SIGINT to all of the members of the tty's foreground process group. Under some circumstances, bash sets the signal handler for SIGINT to be SIGIGN. I haven't looked at this super closely (it would require diving into the bash sources), but you can see it if you attach an strace to the bash shell driving a script such as

#!/bin/bash

while /bin/true; do
date
sleep 60
done &

If you do a "ps axo pid,ppid,pgrp,args", you'll see that the bash and the sleep 60 have the same process group. If you emulate hitting ^C by sending a SIGINT to pid of the shell, you'll see that it ignores it. Sleep also seems to be ignoring the SIGINT when run in the background; but it does honor SIGINT in the foreground --- I didn't have time to dig into that.

Quoted 6 lines

In any case, bash appears to SIGIGN the INT signal if there is a child process running, and only takes the ^C if bash itself is actually "running" the shell script. For example, if you run the command "date;sleep 10;date;sleep 10;date", the ^C only interrupts the sleep command. It doesn't stop the series of commands which bash is running.

This is something that is really hard to get right.

If the shell is running a program when SIGINT arrives, it needs to wait until the program exits, and then try to decide if the program died because of the signal, or actually caught the signal (from the user's perspective), did something useful, and then chose to exit.

If the program's exit status shows that it died due to SIGINT, it is easy to know what to do. But lots of non-trivial programs, probably including 'make' catch SIGINT, do some quick cleanup and then exit. In that case the shell has a hard time deciding what to do.

I wrote a job-controlling shell many years ago and I think the heuristic I came up with was that if the process exited with the SIGINT status, or with a non-zero error status in less that 3 seconds after the signal actually arrived, then react to the signal and abort any script. However it the process takes longer to exit or returns a zero exit status, assume that it was interactive and handled the interrupt to the user's satisfaction, and continue with any script.

I don't know what bash does, and it is possible that it could do a better job. But it is a problem for which there is no straight forward solution (a bit like filesystem data safety it would seem :-)

NeilBrown

#398 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Ingo Molnar

Hi Ingo,

Quoted 5 lines · Ingo Molnar

Btw., i had this test going on that box while i wrote some simple scripts in Vim - and it was a horrible experience. The worst wait was well above one minute - Vim just hung there indefinitely. Not even Ctrl-Z was possible. I captured one such wait, it was hanging right here:

Just a data point: I've seen this exact same time for a long time (1-2 years) too even with stock distribution kernels. Never bothered to investigate it though.

Pekka

#399 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 5 lines · Ingo Molnar
Quoted 5 lines

Btw., i had this test going on that box while i wrote some simple scripts in Vim - and it was a horrible experience. The worst wait was well above one minute - Vim just hung there indefinitely. Not even Ctrl-Z was possible. I captured one such wait, it was hanging right here:

Quoted 1 line · Pekka Enberg

Just a data point: I've seen this exact same time for a long time (1-2

s/same time/same thing/

Quoted 2 lines

years) too even with stock distribution kernels. Never bothered to investigate it though.

#400 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Ingo Molnar
Quoted 26 lines · On Thu, 26 Mar 2009 10:06:30 +0100 Ingo Molnar <[email protected]> wrote:

And it's not just sys_fsync(). The script i wrote tests file read latencies. I have created 1000 files with the same size (all copies of kernel/sched.c ;-), and tested their cache-cold plain-cat performance via:

for ((i=0;i<1000;i++)); do
printf "file #%4d, plain reading it took: " $i /usr/bin/time -f "%e seconds." cat $i >/dev/null
done

I.e. plain, supposedly high-prio reads. The result is very common hickups in read latencies:

file # 579 (253560 bytes), reading it took: 0.08 seconds. file # 580 (253560 bytes), reading it took: 0.05 seconds. file # 581 (253560 bytes), reading it took: 0.01 seconds. file # 582 (253560 bytes), reading it took: 0.01 seconds. file # 583 (253560 bytes), reading it took: 4.61 seconds. file # 584 (253560 bytes), reading it took: 1.29 seconds. file # 585 (253560 bytes), reading it took: 3.01 seconds. file # 586 (253560 bytes), reading it took: 7.74 seconds. file # 587 (253560 bytes), reading it took: 3.22 seconds. file # 588 (253560 bytes), reading it took: 0.05 seconds. file # 589 (253560 bytes), reading it took: 0.36 seconds. file # 590 (253560 bytes), reading it took: 7.39 seconds. file # 591 (253560 bytes), reading it took: 7.58 seconds.

Quoted 8 lines

file # 592 (253560 bytes), reading it took: 7.90 seconds. file # 593 (253560 bytes), reading it took: 8.78 seconds. file # 594 (253560 bytes), reading it took: 8.01 seconds. file # 595 (253560 bytes), reading it took: 7.47 seconds. file # 596 (253560 bytes), reading it took: 11.52 seconds. file # 597 (253560 bytes), reading it took: 10.33 seconds. file # 598 (253560 bytes), reading it took: 8.56 seconds. file # 599 (253560 bytes), reading it took: 7.58 seconds.

(gets deja-vu feelings)

http://lkml.org/lkml/2003/2/21/10

Maybe you should be running a 2.5.61 kernel.

reviewer
#401 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Ingo Molnar
Quoted 24 lines · Ingo Molnar
Quoted 7 lines · Jan Kara
Quoted 2 lines

So tell me again how the VM can rely on the filesystem not blocking at random points.

I can write a patch to make writepage() in the non-"mmapped
creation" case non-blocking on journal. But I'll also have to find out whether it really helps something. But it's probably worth trying...

_all_ the problems i ever had with ext3 were 'collateral damage' type of things: simple writes (sometimes even reads) getting serialized on some large [but reasonable] dirtying activity elsewhere - even if the system was still well within its hard-dirty-limit threshold.

So it sure sounds like an area worth improving, and it's not that hard to reproduce either. Take a system with enough RAM but only a single disk, and do this in a kernel tree:

sync
echo 3 > /proc/sys/vm/drop_caches

Quoted 6 lines

while :; do
date
make mrproper 2>/dev/null >/dev/null
make defconfig 2>/dev/null >/dev/null
make -j32 bzImage 2>/dev/null >/dev/null
done &

I've played with it a bit. I don't have a fast enough machine so that a
compile would feed my SATA drive fast enough (and I also have just 2 GB of memory) but copying kernel tree there and back seemed to load it reasonably.
I've tried a kernel with and without attached patch which makes writepage
not start a transaction when not needed.

Quoted 14 lines

Plain old kernel build, no distcc and no icecream. Wait a few minutes for the system to reach equilibrium. There's no tweaking anywhere, kernel, distro and filesystem defaults used everywhere:

aldebaran:/home/mingo/linux/linux> ./compile-test Thu Mar 26 10:33:03 CET 2009
Thu Mar 26 10:35:24 CET 2009
Thu Mar 26 10:36:48 CET 2009
Thu Mar 26 10:38:54 CET 2009
Thu Mar 26 10:41:22 CET 2009
Thu Mar 26 10:43:41 CET 2009
Thu Mar 26 10:46:02 CET 2009
Thu Mar 26 10:48:28 CET 2009

Quoted 4 lines

And try to use the system while this workload is going on. Use Vim to edit files in this kernel tree. Use plain _cat_ - and i hit delays all the time - and it's not the CPU scheduler but all IO related.

So I observed long delays when VIM was saving a file but in all cases it
was hanging in fsync() which was committing a large transaction (this was both with and without patch) - not a big surprise. Working on the machine seemed a bit better when the patch was applied - in the kernel with the patch VIM at least didn't hang when just writing into the file.
Reads are measurably better with the patch - the test with cat you
describe below took ~0.5s per file without the patch and always less than 0.02s with the patch. So it seems to help something. Can you check on your machine whether you see some improvements? Thanks.

Quoted 55 lines

I have such an ext3 based system where i can do such tests and where i dont mind crashes and data corruption either, so if you send me experimental patches against latet -git i can try them immediately. The system has 16 CPUs, 12GB of RAM and a single disk.

Btw., i had this test going on that box while i wrote some simple scripts in Vim - and it was a horrible experience. The worst wait was well above one minute - Vim just hung there indefinitely. Not even Ctrl-Z was possible. I captured one such wait, it was hanging right here:

aldebaran:~/linux/linux> cat /proc/3742/stack

 [<ffffffff8034790a>] log_wait_commit+0xbd/0x110
 [<ffffffff803430b2>] journal_stop+0x1df/0x20d
 [<ffffffff8034421f>] journal_force_commit+0x28/0x2d
 [<ffffffff80331c69>] ext3_force_commit+0x2b/0x2d
 [<ffffffff80328b56>] ext3_write_inode+0x3e/0x44
 [<ffffffff802ebb9d>] __sync_single_inode+0xc1/0x2ad
 [<ffffffff802ebed6>] __writeback_single_inode+0x14d/0x15a
 [<ffffffff802ebf0c>] sync_inode+0x29/0x34
 [<ffffffff80327453>] ext3_sync_file+0xa7/0xb4
 [<ffffffff802ef17d>] vfs_fsync+0x78/0xaf
 [<ffffffff802ef1eb>] do_fsync+0x37/0x4d
 [<ffffffff802ef228>] sys_fsync+0x10/0x14
 [<ffffffff8020bd1b>] system_call_fastpath+0x16/0x1b
 [<ffffffffffffffff>] 0xffffffffffffffff

It took about 120 seconds for it to recover.

And it's not just sys_fsync(). The script i wrote tests file read latencies. I have created 1000 files with the same size (all copies of kernel/sched.c ;-), and tested their cache-cold plain-cat performance via:

for ((i=0;i<1000;i++)); do
printf "file #%4d, plain reading it took: " $i /usr/bin/time -f "%e seconds." cat $i >/dev/null
done

I.e. plain, supposedly high-prio reads. The result is very common hickups in read latencies:

file # 579 (253560 bytes), reading it took: 0.08 seconds. file # 580 (253560 bytes), reading it took: 0.05 seconds. file # 581 (253560 bytes), reading it took: 0.01 seconds. file # 582 (253560 bytes), reading it took: 0.01 seconds. file # 583 (253560 bytes), reading it took: 4.61 seconds. file # 584 (253560 bytes), reading it took: 1.29 seconds. file # 585 (253560 bytes), reading it took: 3.01 seconds. file # 586 (253560 bytes), reading it took: 7.74 seconds. file # 587 (253560 bytes), reading it took: 3.22 seconds. file # 588 (253560 bytes), reading it took: 0.05 seconds. file # 589 (253560 bytes), reading it took: 0.36 seconds. file # 590 (253560 bytes), reading it took: 7.39 seconds. file # 591 (253560 bytes), reading it took: 7.58 seconds.

Quoted 8 lines

file # 592 (253560 bytes), reading it took: 7.90 seconds. file # 593 (253560 bytes), reading it took: 8.78 seconds. file # 594 (253560 bytes), reading it took: 8.01 seconds. file # 595 (253560 bytes), reading it took: 7.47 seconds. file # 596 (253560 bytes), reading it took: 11.52 seconds. file # 597 (253560 bytes), reading it took: 10.33 seconds. file # 598 (253560 bytes), reading it took: 8.56 seconds. file # 599 (253560 bytes), reading it took: 7.58 seconds.

<snip>

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Quoted 1 line

From 1bf84d0f6162196b4c0d83e9db1ee11507a8f91f Mon Sep 17 00:00:00 2001

From: Jan Kara <[email protected]>
Date: Thu, 26 Mar 2009 13:08:04 +0100
Subject: [PATCH] ext3: Avoid starting a transaction in writepage when not necessary

We don't have to start a transaction in writepage() when all the blocks are a properly allocated. Even in ordered mode either the data has been written via write() and they are thus already added to transaction's list or the data was written via mmap and then it's random in which transaction they get written anyway.

This should help VM to pageout dirty memory without blocking on transaction commits.

---
 fs/ext3/inode.c |   19 ++++++++++++++-----
 1 files changed, 14 insertions(+), 5 deletions(-)
fs/ext3/inode.c +14 -5
index e230f7a..61bce1a 100644
@@ -1420,6 +1420,10 @@ static int bput_one(handle_t *handle, struct buffer_head *bh)
 	return 0;
 }
 
+static int buffer_unmapped(handle_t *handle, struct buffer_head *bh)
+{
+	return !buffer_mapped(bh);
+}
 /*
  * Note that we always start a transaction even if we're not journalling
  * data.  This is to preserve ordering: any hole instantiation within
@@ -1490,6 +1494,16 @@ static int ext3_ordered_writepage(struct page *page,
 	if (ext3_journal_current_handle())
 		goto out_fail;
 
+	if (!page_has_buffers(page)) {
+		create_empty_buffers(page, inode->i_sb->s_blocksize,
+				(1 << BH_Dirty)|(1 << BH_Uptodate));
+	} else if (!walk_page_buffers(NULL, page_buffers(page), 0, PAGE_CACHE_SIZE, NULL, buffer_unmapped)) {
+		/* Provide NULL instead of get_block so that we catch bugs if buffers weren't really mapped */
+		return block_write_full_page(page, NULL, wbc);
+	}
+	page_bufs = page_buffers(page);
+
+
 	handle = ext3_journal_start(inode, ext3_writepage_trans_blocks(inode));
 
 	if (IS_ERR(handle)) {
@@ -1497,11 +1511,6 @@ static int ext3_ordered_writepage(struct page *page,
 		goto out_fail;
 	}
 
-	if (!page_has_buffers(page)) {
-		create_empty_buffers(page, inode->i_sb->s_blocksize,
-				(1 << BH_Dirty)|(1 << BH_Uptodate));
-	}
-	page_bufs = page_buffers(page);
 	walk_page_buffers(handle, page_bufs, 0,
 			PAGE_CACHE_SIZE, NULL, bget_one);
 
Signature · 1 line
1.6.0.2
#402 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

The patch looks OK to me.

fs/ext3/inode.c static int ext3_ordered_writepage(struct page *page, · line 1499

Quoted 7 lines · On Thu, 26 Mar 2009 19:11:06 +0100 Jan Kara <[email protected]> wrote:
fs/ext3/inode.c +3 -0
@@ -1490,6 +1494,16 @@ static int ext3_ordered_writepage(struct page *page,
 	if (ext3_journal_current_handle())
 		goto out_fail;
 
+	if (!page_has_buffers(page)) {
+		create_empty_buffers(page, inode->i_sb->s_blocksize,
+				(1 << BH_Dirty)|(1 << BH_Uptodate));

This will attach dirty buffers to a clean page, which is an invalid state (but OK if we immediately fix it up).

fs/ext3/inode.c static int ext3_ordered_writepage(struct page *page, · line 1509

Quoted 10 lines
fs/ext3/inode.c +7 -0
+	} else if (!walk_page_buffers(NULL, page_buffers(page), 0, PAGE_CACHE_SIZE, NULL, buffer_unmapped)) {
+		/* Provide NULL instead of get_block so that we catch bugs if buffers weren't really mapped */
+		return block_write_full_page(page, NULL, wbc);
+	}
+	page_bufs = page_buffers(page);
+
+
 	handle = ext3_journal_start(inode, ext3_writepage_trans_blocks(inode));
 
 	if (IS_ERR(handle)) {

And if this error happens we'll go on to run redirty_page_for_writepage() which will do the right thing.

However if PageMappedToDisk() is working right, we should be able to avoid that newly-added buffer walk. Possibly SetPageMappedToDisk() isn't being run in all the right places though, dunno.

reviewer
#403 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 9 lines · Andrew Morton

The patch looks OK to me.

Quoted 4 lines · On Thu, 26 Mar 2009 19:11:06 +0100 Jan Kara <[email protected]> wrote:
fs/ext3/inode.c +0 -0
@@ -1490,6 +1494,16 @@ static int ext3_ordered_writepage(struct page *page,
 	if (ext3_journal_current_handle())
 		goto out_fail;
 
Quoted 6 lines

fs/ext3/inode.c static int ext3_ordered_writepage(struct page *page, · line 1499

Quoted 3 lines
fs/ext3/inode.c +3 -0
+	if (!page_has_buffers(page)) {
+		create_empty_buffers(page, inode->i_sb->s_blocksize,
+				(1 << BH_Dirty)|(1 << BH_Uptodate));

This will attach dirty buffers to a clean page, which is an invalid state (but OK if we immediately fix it up).

Yes - actually the page has been dirty just the moment before when we run
clear_page_dirty_for_io() - and at this function could have also created the clean page with dirty buffers...

Quoted 9 lines
Quoted 9 lines
fs/ext3/inode.c +7 -0
+	} else if (!walk_page_buffers(NULL, page_buffers(page), 0, PAGE_CACHE_SIZE, NULL, buffer_unmapped)) {
+		/* Provide NULL instead of get_block so that we catch bugs if buffers weren't really mapped */
+		return block_write_full_page(page, NULL, wbc);
+	}
+	page_bufs = page_buffers(page);
+
+
 	handle = ext3_journal_start(inode, ext3_writepage_trans_blocks(inode));
 
Quoted 8 lines

fs/ext3/inode.c static int ext3_ordered_writepage(struct page *page, · line 1509

Quoted 1 line
fs/ext3/inode.c +0 -0
 	if (IS_ERR(handle)) {

And if this error happens we'll go on to run redirty_page_for_writepage() which will do the right thing.

However if PageMappedToDisk() is working right, we should be able to avoid that newly-added buffer walk. Possibly SetPageMappedToDisk() isn't being run in all the right places though, dunno.

Yes, SetPageMappedToDisk is set only by block_read_full_page(),
mpage_readpage() and nobh_write_begin(). Obviously not enough... It would be nice to improve that but that's another story...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#404 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

In reply to Jan Kara
Quoted 4 lines · Jan Kara

Reads are measurably better with the patch - the test with cat you
describe below took ~0.5s per file without the patch and always less than 0.02s with the patch. So it seems to help something.

That would seem to be a _huge_ improvement.

Reads are the biggest issue for starting a new process (eg starting firefox while under load), and if cat'ing that small file improved by that much, then I bet there's a huge practical implication for a lot of desktop uses.

The fundamental fsync() latency problem we sadly can't help much with, the way ext3 seems to work. But I do suspect that the whole "don't synchronize with the journal for normal write-outs" may end up helping even fsync just a bit, if only because I suspect it will improve writeout throughput too and thus avoid one particular bottleneck.

Linus

#405 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 7 lines · On Thu, 26 Mar 2009 15:39:53 -0700 (PDT) Linus Torvalds…
Quoted 4 lines · Jan Kara

Reads are measurably better with the patch - the test with cat you
describe below took ~0.5s per file without the patch and always less than 0.02s with the patch. So it seems to help something.

That would seem to be a _huge_ improvement.

It's strange that we still don't have an ext3_writepages(). Open a transaction, do a large pile of writes, close the transaction again. We don't even have a data=writeback writepages() implementation, which should be fairly simple.

Bizarre.

Mingming had a shot at it a few years ago and I think Badari did as well, but I guess it didn't work out.

Falling back to generic_writepages() on our main local fs is a bit lame.

reviewer
#406 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Thu, 26 Mar 2009 15:39:53 -0700 (PDT) Linus Torvalds…
Quoted 4 lines · Jan Kara

Reads are measurably better with the patch - the test with cat you
describe below took ~0.5s per file without the patch and always less than 0.02s with the patch. So it seems to help something.

Quoted 6 lines
Quoted 1 line

That would seem to be a _huge_ improvement.

It's strange that we still don't have an ext3_writepages(). Open a transaction, do a large pile of writes, close the transaction again. We don't even have a data=writeback writepages() implementation, which should be fairly simple.

Doable but not fairly simple ;) Firstly you have to restart a transaction
when you've used up all the credits you originally started with (easy), secondly ext3 uses lock order PageLock -> "transaction start" which is unusable for the scheme you suggest. So we'd have to revert that - which needs larger audit of our locking scheme and that's probably the reason why noone has done it yet.

Quoted 6 lines

Bizarre.

Mingming had a shot at it a few years ago and I think Badari did as well, but I guess it didn't work out.

Falling back to generic_writepages() on our main local fs is a bit lame.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#407 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 7 lines · Jan Kara

Doable but not fairly simple ;) Firstly you have to restart a transaction
when you've used up all the credits you originally started with (easy), secondly ext3 uses lock order PageLock -> "transaction start" which is unusable for the scheme you suggest. So we'd have to revert that - which needs larger audit of our locking scheme and that's probably the reason why noone has done it yet.

It's also not clear that ext3 can really do much better than the regular generic_writepages() logic. I mean, seriously, what's there to improve on? The transaction code is all normally totally pointless, and I merged the patch that avoids it when not necessary.

It might be different if more people used "data=journal", but I don't doubt that is very common. For data=writeback and data=ordered, I bet generic_writepages() is as good as anything ext3-specific could be.

Linus

#408 lore raw

Re: ext3 IO latency measurements (was: Linux 2.6.29)

Quoted 11 lines · On Fri, 27 Mar 2009 15:10:56 -0700 (PDT) Linus Torvalds…
Quoted 7 lines · Jan Kara

Doable but not fairly simple ;) Firstly you have to restart a transaction
when you've used up all the credits you originally started with (easy), secondly ext3 uses lock order PageLock -> "transaction start" which is unusable for the scheme you suggest. So we'd have to revert that - which needs larger audit of our locking scheme and that's probably the reason why noone has done it yet.

It's also not clear that ext3 can really do much better than the regular generic_writepages() logic. I mean, seriously, what's there to improve on?

- opening a single transaction for many pages in the cases when a transaction _is_ needed.

- single large BIO versus zillions of single-page BIOs.

Relatively minor benefits, but it's a bit odd that we never got around to doing it.

It just got quite a bit harder to do, so I expect we won't be doing it.

#409 lore raw

updated: ext3 IO latency measurements on v2.6.30-rc1

In reply to Ingo Molnar

This is an update to the ext3+CFQ latency measurements i did early in the merge window - originally with a v2.6.29 based kernel.

Today i've repeated my measurements under v2.6.30-rc1, using the exact same system and the exact same workload.

The quick executive summary:

_what a difference a week of upstream development makes_!

:-)

Here are the specific details, as a reply to my earlier mail:

Quoted 44 lines · Ingo Molnar
Quoted 7 lines · Jan Kara
Quoted 2 lines

So tell me again how the VM can rely on the filesystem not blocking at random points.

I can write a patch to make writepage() in the non-"mmapped
creation" case non-blocking on journal. But I'll also have to find out whether it really helps something. But it's probably worth trying...

_all_ the problems i ever had with ext3 were 'collateral damage' type of things: simple writes (sometimes even reads) getting serialized on some large [but reasonable] dirtying activity elsewhere - even if the system was still well within its hard-dirty-limit threshold.

So it sure sounds like an area worth improving, and it's not that hard to reproduce either. Take a system with enough RAM but only a single disk, and do this in a kernel tree:

sync
echo 3 > /proc/sys/vm/drop_caches

while :; do
date
make mrproper 2>/dev/null >/dev/null
make defconfig 2>/dev/null >/dev/null
make -j32 bzImage 2>/dev/null >/dev/null
done &

Plain old kernel build, no distcc and no icecream. Wait a few minutes for the system to reach equilibrium. There's no tweaking anywhere, kernel, distro and filesystem defaults used everywhere:

aldebaran:/home/mingo/linux/linux> ./compile-test Thu Mar 26 10:33:03 CET 2009
Thu Mar 26 10:35:24 CET 2009
Thu Mar 26 10:36:48 CET 2009
Thu Mar 26 10:38:54 CET 2009
Thu Mar 26 10:41:22 CET 2009
Thu Mar 26 10:43:41 CET 2009
Thu Mar 26 10:46:02 CET 2009
Thu Mar 26 10:48:28 CET 2009

Quoted 4 lines

And try to use the system while this workload is going on. Use Vim to edit files in this kernel tree. Use plain _cat_ - and i hit delays all the time - and it's not the CPU scheduler but all IO related.

Under .30-rc1 i couldnt hit a single (!) annoying delay during half an hour of trying. The "Vim experience" is _totally_ smooth with a load average of 40+.

And this is with default, untweaked ext3 - not even ext4. I'm impressed.

Quoted 28 lines

I have such an ext3 based system where i can do such tests and where i dont mind crashes and data corruption either, so if you send me experimental patches against latet -git i can try them immediately. The system has 16 CPUs, 12GB of RAM and a single disk.

Btw., i had this test going on that box while i wrote some simple scripts in Vim - and it was a horrible experience. The worst wait was well above one minute - Vim just hung there indefinitely. Not even Ctrl-Z was possible. I captured one such wait, it was hanging right here:

aldebaran:~/linux/linux> cat /proc/3742/stack

 [<ffffffff8034790a>] log_wait_commit+0xbd/0x110
 [<ffffffff803430b2>] journal_stop+0x1df/0x20d
 [<ffffffff8034421f>] journal_force_commit+0x28/0x2d
 [<ffffffff80331c69>] ext3_force_commit+0x2b/0x2d
 [<ffffffff80328b56>] ext3_write_inode+0x3e/0x44
 [<ffffffff802ebb9d>] __sync_single_inode+0xc1/0x2ad
 [<ffffffff802ebed6>] __writeback_single_inode+0x14d/0x15a
 [<ffffffff802ebf0c>] sync_inode+0x29/0x34
 [<ffffffff80327453>] ext3_sync_file+0xa7/0xb4
 [<ffffffff802ef17d>] vfs_fsync+0x78/0xaf
 [<ffffffff802ef1eb>] do_fsync+0x37/0x4d
 [<ffffffff802ef228>] sys_fsync+0x10/0x14
 [<ffffffff8020bd1b>] system_call_fastpath+0x16/0x1b
 [<ffffffffffffffff>] 0xffffffffffffffff
Quoted 1 line

It took about 120 seconds for it to recover.

These delays are definitely below 300 msecs now. (100 msecs is roughly the lag i can still notice in typing)

Quoted 26 lines

And it's not just sys_fsync(). The script i wrote tests file read latencies. I have created 1000 files with the same size (all copies of kernel/sched.c ;-), and tested their cache-cold plain-cat performance via:

for ((i=0;i<1000;i++)); do
printf "file #%4d, plain reading it took: " $i /usr/bin/time -f "%e seconds." cat $i >/dev/null
done

I.e. plain, supposedly high-prio reads. The result is very common hickups in read latencies:

file # 579 (253560 bytes), reading it took: 0.08 seconds. file # 580 (253560 bytes), reading it took: 0.05 seconds. file # 581 (253560 bytes), reading it took: 0.01 seconds. file # 582 (253560 bytes), reading it took: 0.01 seconds. file # 583 (253560 bytes), reading it took: 4.61 seconds. file # 584 (253560 bytes), reading it took: 1.29 seconds. file # 585 (253560 bytes), reading it took: 3.01 seconds. file # 586 (253560 bytes), reading it took: 7.74 seconds. file # 587 (253560 bytes), reading it took: 3.22 seconds. file # 588 (253560 bytes), reading it took: 0.05 seconds. file # 589 (253560 bytes), reading it took: 0.36 seconds. file # 590 (253560 bytes), reading it took: 7.39 seconds. file # 591 (253560 bytes), reading it took: 7.58 seconds.

Quoted 8 lines

file # 592 (253560 bytes), reading it took: 7.90 seconds. file # 593 (253560 bytes), reading it took: 8.78 seconds. file # 594 (253560 bytes), reading it took: 8.01 seconds. file # 595 (253560 bytes), reading it took: 7.47 seconds. file # 596 (253560 bytes), reading it took: 11.52 seconds. file # 597 (253560 bytes), reading it took: 10.33 seconds. file # 598 (253560 bytes), reading it took: 8.56 seconds. file # 599 (253560 bytes), reading it took: 7.58 seconds.

This test is totally smooth now:

file #   6 (253560 bytes), reading it took: 0.05 seconds.
file #   7 (253560 bytes), reading it took: 0.11 seconds.
file #   8 (253560 bytes), reading it took: 0.12 seconds.
file #   9 (253560 bytes), reading it took: 0.06 seconds.
file #  10 (253560 bytes), reading it took: 0.05 seconds.
file #  11 (253560 bytes), reading it took: 0.11 seconds.
file #  12 (253560 bytes), reading it took: 0.09 seconds.
file #  13 (253560 bytes), reading it took: 0.09 seconds.
file #  14 (253560 bytes), reading it took: 0.03 seconds.
file #  15 (253560 bytes), reading it took: 0.08 seconds.
file #  16 (253560 bytes), reading it took: 0.15 seconds.
file #  17 (253560 bytes), reading it took: 0.06 seconds.
file #  18 (253560 bytes), reading it took: 0.13 seconds.
file #  19 (253560 bytes), reading it took: 0.16 seconds.
file #  20 (253560 bytes), reading it took: 0.29 seconds.
file #  21 (253560 bytes), reading it took: 0.18 seconds.
file #  22 (253560 bytes), reading it took: 0.28 seconds.
file #  23 (253560 bytes), reading it took: 0.04 seconds.

290 msecs was the worst in thes series above.

The vim read+write test takes longer:

aldebaran:~/linux/linux/test-files/src> ./vim-test

file #   0 (253560 bytes), Vim-opening it took: 2.35 seconds.
file #   1 (253560 bytes), Vim-opening it took: 2.09 seconds.
file #   2 (253560 bytes), Vim-opening it took: 2.20 seconds.
file #   3 (253560 bytes), Vim-opening it took: 2.14 seconds.
file #   4 (253560 bytes), Vim-opening it took: 2.15 seconds.
file #   5 (253560 bytes), Vim-opening it took: 2.13 seconds.
file #   6 (253560 bytes), Vim-opening it took: 2.11 seconds.
file #   7 (253560 bytes), Vim-opening it took: 2.09 seconds.
file #   8 (253560 bytes), Vim-opening it took: 2.03 seconds.
file #   9 (253560 bytes), Vim-opening it took: 2.03 seconds.
file #  10 (253560 bytes), Vim-opening it took: 2.06 seconds.
file #  11 (253560 bytes), Vim-opening it took: 2.19 seconds.
file #  12 (253560 bytes), Vim-opening it took: 2.07 seconds.
file #  13 (253560 bytes), Vim-opening it took: 2.02 seconds.

I suspect that is to be expected? The test does:

vim -c "q" $i 2>/dev/null >/dev/null

2 seconds is OK-ish - when close+writing a file i mentally expect some short delay anyway. I think vim fsyncs the swap file in that case as well.

I havent actually experienced any such delays while editing files.

Quoted 53 lines

The system's RAM is ridiculously under-utilized, 96.1% is free, only 3.9% is utilized:

total used free shared buffers cached

 Mem:      12318192     476732   11841460          0      48324     142936
 -/+ buffers/cache:     285472   12032720

Swap: 4096564 0 4096564

Dirty data in /proc/meminfo fluctuates between 0.4% and 1.6% of total RAM. (the script removes the freshly build kernel object files, so the workload is pretty steady.)

The peak of 1.6% looks like this:

Dirty:            118376 kB
Dirty:            143784 kB
Dirty:            161756 kB
Dirty:            185084 kB
Dirty:            210524 kB
Dirty:            213348 kB
Dirty:            200124 kB
Dirty:            122152 kB
Dirty:            121508 kB
Dirty:            121512 kB

(1 second snapshots)

So the problems are all around the place and they are absolutely, trivially reproducible. And this is how a default ext3 based distro and the default upstream kernel will present itself to new Linux users and developers. It's not a pretty experience.

Oh, and while at it - also a job control complaint. I tried to Ctrl-C the above script:

file # 858 (253560 bytes), reading it took: 0.06 seconds. file # 859 (253560 bytes), reading it took: 0.02 seconds. file # 860 (253560 bytes), reading it took: 5.53 seconds. file # 861 (253560 bytes), reading it took: 3.70 seconds. file # 862 (253560 bytes), reading it took: 0.88 seconds. file # 863 (253560 bytes), reading it took: 0.04 seconds. file # 864 (253560 bytes), reading it took: ^C0.69 seconds. file # 865 (253560 bytes), reading it took: ^C0.49 seconds. file # 866 (253560 bytes), reading it took: ^C0.01 seconds. file # 867 (253560 bytes), reading it took: ^C0.02 seconds. file # 868 (253560 bytes), reading it took: ^C^C0.01 seconds. file # 869 (253560 bytes), reading it took: ^C^C0.04 seconds. file # 870 (253560 bytes), reading it took: ^C^C^C0.03 seconds. file # 871 (253560 bytes), reading it took: ^C0.02 seconds. file # 872 (253560 bytes), reading it took: ^C^C0.02 seconds. file # 873 (253560 bytes), reading it took:
^C^C^C^Caldebaran:~/linux/linux/test-files/src>

Quoted 3 lines

I had to hit Ctrl-C numerous times before Bash would honor it. This to is a very common thing on large SMP systems. I'm willing to test patches until all these problems are fixed. Any takers?

This Bash bug still occurs and is highly annoying when using scripts on SMP Linux boxes and trying to Ctrl-C out of them.

But all in one, the ext3 and IO latency problems seem to be thoroughly cured!

To me the past 1 week has made more difference in filesystems and IO interactivity than all filesystems development done in the past 5+ years, combined. Kudos!

Ingo

#410 lore raw

Re: updated: ext3 IO latency measurements on v2.6.30-rc1

Quoted 9 lines · Ingo Molnar

This is an update to the ext3+CFQ latency measurements i did early in the merge window - originally with a v2.6.29 based kernel.

Today i've repeated my measurements under v2.6.30-rc1, using the exact same system and the exact same workload.

The quick executive summary:

_what a difference a week of upstream development makes_!

Here's a quicktest with xfs+CFQ on one of my testing machines, using fsync-tester while running Linus' "bigfile torture test". It underlines your results, showing significant improvement!

The xfs partition is mounted with the defaults, and I'm expecting slightly more improvement after mounting with noatime and nobarrier.

2.6.30-rc1

fsync time: 0.4674
fsync time: 1.0473
fsync time: 0.4190
fsync time: 1.0800
fsync time: 1.0132
fsync time: 1.0193
fsync time: 1.0191
fsync time: 1.1318
fsync time: 0.9924
fsync time: 1.0568
fsync time: 1.0676
fsync time: 1.0241
fsync time: 1.0530
fsync time: 0.9709
fsync time: 0.4475
fsync time: 0.6320
fsync time: 1.0906
fsync time: 0.6344
fsync time: 1.0632
fsync time: 1.0455
fsync time: 1.0530
fsync time: 1.0655
fsync time: 1.0032
fsync time: 1.0644
fsync time: 1.1573
fsync time: 1.0197
fsync time: 1.0342
fsync time: 1.0643
fsync time: 0.0342
fsync time: 0.7603
fsync time: 1.0905
fsync time: 0.6340

2.6.29.1

fsync time: 2.1255
fsync time: 2.2851
fsync time: 1.9048
fsync time: 1.0999
fsync time: 2.0117
fsync time: 2.0819
fsync time: 2.0819
fsync time: 0.0225
fsync time: 0.2796
fsync time: 0.3879
fsync time: 0.6584
fsync time: 0.9287
fsync time: 0.2488
fsync time: 2.0994
fsync time: 2.0161
fsync time: 1.9736
fsync time: 2.0231
fsync time: 2.2888
fsync time: 2.1719
fsync time: 1.8452
fsync time: 0.3278
fsync time: 1.0881
fsync time: 0.5202
fsync time: 1.3339
fsync time: 0.4295
fsync time: 1.2772
fsync time: 1.9436
fsync time: 2.1048
fsync time: 1.9376
fsync time: 2.0786
fsync time: 1.9202

#411 lore raw

Re: updated: ext3 IO latency measurements on v2.6.30-rc1

In reply to Ingo Molnar

To test the ext3 improvements in 2.6.30-rc5, I tried my previous test of starting an rxvt while findutils or 'aptitude dist-upgrade' is madly spinning the disk.

With vanilla 2.6.29 in those circumstances, starting an rxvt took 4 seconds on an otherwise unloaded system (Thinkpad T60 w/ 1.83GHz dual-core CPU, 1.5GB RAM, 5400 rpm drive). The second rxvt came up right away. So the first rxvt probably stalled until /usr/bin/rxvt could be grabbed from disk.

With 2.6.30-rc5, the rxvt showed up much faster -- either right away (maybe it was still in the cache) or after about 0.5 seconds (probably the uncached case).

Other tests were also snappy, like opening a 10MB PDF file in Emacs, adding a newline at the beginning, and saving it (while findutils was running). I hadn't done the same test with 2.6.29, but I'm pretty sure it would have been painfully slow if findutils was running.

So, relative to 2.6.29, I see a large improvement in interactive behavior under high IO load.

-Sanjoy

`Until lions have their historians, tales of the hunt shall always
glorify the hunters.' --African Proverb

#412 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

Yes, yes, it may need to allocate backing store (a page that was dirtied by mmap), and I'm sure that's the reason for it all,

Hmm. Thinking about that, I'm not so sure. Shouldn't that backing store allocation happen when the page is actually dirtied on ext3?

I _suspect_ that goes back to the fact that ext3 is older than the "aops->set_page_dirty()" callback, and nobody taught ext3 to do the bmap's at dirty time, so now it does it at writeout time.

Anyway, there we are. Old filesystems do the wrong thing (block allocation while doing writeout because they don't do it when dirtying), and newer filesystems do the wrong thing (block allocations during writeout, because they want to do delayed allocation to do the inode dirtying after doing writeback).

And in either case, the VM is screwed, and can't ask for writeout, because it will be randomly throttled by the filesystem. So we do lots of async bdflush threads, which then causes IO ordering problems because now the writeout is all in random order.

Linus

reviewer
#413 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

Yes, yes, it may need to allocate backing store (a page that was dirtied by mmap), and I'm sure that's the reason for it all,

Hmm. Thinking about that, I'm not so sure. Shouldn't that backing store allocation happen when the page is actually dirtied on ext3?

We don't do it currently. We could do it (it would also solve the problem
that we currently silently discard users data when he reaches his quota or filesystem gets ENOSPC) but there are problems with it as well:
1) We have to writeout blocks full of zeros on allocation so that we don't
expose unallocated data => slight slowdown
2) When blocksize < pagesize we must play nasty tricks for this to work
(think about i_size = 1024, set_page_dirty(), truncate(f, 8192), writepage() -> uhuh, not enough space allocated)
3) We'll do allocation in the order in which pages are dirtied. Generally,
I'd suspect this order to be less linear than the order in which writepages submit IO and thus it will result in the larger fragmentation of the file.
So it's not a clear win IMHO.

Quoted 10 lines

I _suspect_ that goes back to the fact that ext3 is older than the "aops->set_page_dirty()" callback, and nobody taught ext3 to do the bmap's at dirty time, so now it does it at writeout time.

Anyway, there we are. Old filesystems do the wrong thing (block allocation while doing writeout because they don't do it when dirtying), and newer filesystems do the wrong thing (block allocations during writeout, because they want to do delayed allocation to do the inode dirtying after doing writeback).

Quoted 4 lines

And in either case, the VM is screwed, and can't ask for writeout, because it will be randomly throttled by the filesystem. So we do lots of async bdflush threads, which then causes IO ordering problems because now the writeout is all in random order.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#414 lore raw
Quoted 3 lines · Jan Kara

1) We have to writeout blocks full of zeros on allocation so that we don't
expose unallocated data => slight slowdown

Why?

This is in _no_ way different from a regular "write()" system call. And there, we just attach the buffers to the page. If something crashes before the page actually gets written out, then we'll have hopefully never written out the metadata (that's what "data=ordered" means).

Quoted 3 lines

2) When blocksize < pagesize we must play nasty tricks for this to work
(think about i_size = 1024, set_page_dirty(), truncate(f, 8192), writepage() -> uhuh, not enough space allocated)

Good point. I suspect not enough people have played around with "set_page_dirty()" to find these kinds of things. The VFS layer probably doesn't help sufficiently with the half-dirty pages, although the FS can obviously always look up the previously last page and do things manually if it wants to.

But yes, this is nasty.

Quoted 4 lines

3) We'll do allocation in the order in which pages are dirtied. Generally,
I'd suspect this order to be less linear than the order in which writepages submit IO and thus it will result in the larger fragmentation of the file.
So it's not a clear win IMHO.

Yes, that may be the case.

Of course, the approach of just checking whether the buffer heads already exists and are mapped (before bothering with anything else) probably works fine in practice. In most loads, pages will have been dirtied by regular "write()" system calls, and then we will have the buffers pre-allocated regardless.

Linus

maintainer
#415 lore raw
Quoted 7 lines · Linus Torvalds

Of course, the approach of just checking whether the buffer heads already exists and are mapped (before bothering with anything else) probably works fine in practice. In most loads, pages will have been dirtied by regular "write()" system calls, and then we will have the buffers pre-allocated regardless.

Yeah, I agree; solving the problem in the case of files being dirtied via write() is going to solve a much percentage of the cases compared to those cases where the pages are dirtied via mmap()'ed pages.

I thought we were doing this already, but clearly I should have looked at the code first. :-(

- Ted

reviewer
#416 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Jan Kara

1) We have to writeout blocks full of zeros on allocation so that we don't
expose unallocated data => slight slowdown

Why?

This is in _no_ way different from a regular "write()" system call. And there, we just attach the buffers to the page. If something crashes before the page actually gets written out, then we'll have hopefully never written out the metadata (that's what "data=ordered" means).

Sorry, I wasn't exact enough. We'll attach buffers to the running
transaction and they'll get written out at the transaction commit which is usually earlier than when the writepage() is called and then later writepage() will write the data again (this is a consequence of the fact that JBD commit code just writes buffers without calling clear_page_dirty_for_io())...
At least ext4 has this fixed because JBD2 already writes out ordered data
via writepages().

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#417 lore raw
In reply to Ingo Molnar
Quoted 12 lines · Ingo Molnar
Quoted 4 lines

"Give kjournald a IOPRIO_CLASS_RT io priority"

October 2007 (yes its that old)

thx. A more recent submission from Arjan would be:

    http://lkml.org/lkml/2008/10/1/405

Resolution was that Tytso indicated it went into some sort of ext4 patch queue:

Quoted 7 lines
| I've ported the patch to the ext4 filesystem, and dropped it into 
| the unstable portion of the ext4 patch queue.
|
|   ext4: akpm's locking hack to fix locking delays

but 6 months down the line and i can find no trace of this upstream anywhere.

Andrew really didn't like Arjan's patch because it forces non-synchronous writes to have a real-time I/O priority. He suggested an alternative approach which I coded up as "akpm's locking hack to fix locking delays"; unfortunately, it doesn't work.

In ext4, I quietly put in a mount option, journal_ioprio, and set the default to be slightly higher than the default I/O priority (but no a real-time class priority) to prevent the write starvation problem. This definitely helps for some workloads (when some task is reading enough to starve out the rights).

More recently (as in this past weekend), I went back to the ext3 problem, and found a better solution, here:

	 http://lkml.org/lkml/2009/3/21/304
	 http://lkml.org/lkml/2009/3/21/302
	 http://lkml.org/lkml/2009/3/21/303

These patches cause the synchronous writes caused by an fsync() to be submitted using WRITE_SYNC, instead of WRITE, which definitely helps in the case where there is a heavy read workload in the background.

They don't solve the problem where there is a *huge* amount of writes going on, though --- if something is dirtying pages at a rate far greater than the local disk can write it out, say, either "dd if=/dev/zero of=/mnt/make-lots-of-writes" or a massive distcc cluster driving a huge amount of data towards a single system or a wget over a local 100 megabit ethernet from a massive NFS server where everything is in cache, then you can have a major delay with the fsync().

However, what I've found, though, is that if you're just doing a local copy from one hard drive to another, or downloading a huge iso file from an ftp server over a wide area network, the fsync() delays really don't get *that* bad, even with ext3. At least, I haven't found a workload that doesn't involve either dd if=/dev/zero or a massive amount of data coming in over the network that will cause fsync() delays in the > 1-2 second category. Ext3 has been around for a long time, and it's only been the last couple of years that people have really complained about this; my theory is that it was the rise of > 10 megabit ethernets and the use of systems like distcc that really made this problem really become visible. The only realistic workload I've found that triggers this requires a fast network dumping data to a local filesystem.

(I'm sure someone will be ingeniuous enough to find something else though, and if they're interested, I've attached an fsync latency tester to this note. If you find something; let me know, I'd be interested.)

Quoted 6 lines

<let-me-rant-too>

The thing is ... this is a _bad_ ext3 design bug affecting ext3 users in the last decade or so of ext3 existence. Why is this issue not handled with the utmost high priority and why wasnt it fixed 5 years ago already? :-)

OK, so there are a couple of solutions to this problem. One is to use ext4 and delayed allocation. This solves the problem by simply not allocating the blocks in the first place, so we don't have to force them out to solve the security problem that data=ordered was trying to solve. Simply mounting an ext3 filesystem using ext4, without making any change to the filesystem format, should solve the problem.

Another is to use the mount option data=writeback. The whole reason for forcing the writes out to disk was simply to prevent a security problem that occurs if your system crashes before the data blocks get forced out to disk. This could expose previously written data, which could belong to another user, and might be his e-mail or p0rn. Historically, this was always a problem with the BSD Fast Filesystem; it sync'ed out data every 30 seconds, and metadata every 5 seconds. (This is where the default ext3 commit interval of 5 seconds, and the default /proc/sys/vm/dirty_expire_centiseconds came from.) After a system crash, it was possible for files written just before the crash to point to blocks that had not yet been written, and which contain some other users' data files. This was the reason for Stephen Tweedie implementing the data=ordered mode, and making it the default.

However, these days, nearly all Linux boxes are single user machines, so the security concern is much less of a problem. So maybe the best solution for now is to make data=writeback the default. This solves the problem too. The only problem with this is that there are a lot of sloppy application writers out there, and they've gotten lazy about using fsync() where it's necessary; combine that with Ubuntu shipping massively unstable video drivers that crash if you breath on the system wrong (or exit World of Goo :-), and you've got the problem which was recently slashdotted, and which I wrote about here:

http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/

Quoted 2 lines

It does not matter whether we have extents or htrees when there are _trivially reproducible_ basic usability problems with ext3.

Try ext4, I think you'll like it. :-)

Failing that, data=writeback for single-user machines is probably your best bet.

- Ted

/*
* fsync-tester.c
*
* Written by Theodore Ts'o, 3/21/09.
*
* This file may be redistributed under the terms of the GNU Public
* License, version 2.
*/

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <time.h>
#include <fcntl.h>
#include <string.h>

#define SIZE (32768*32)

static float timeval_subtract(struct timeval *tv1, struct timeval *tv2)

{
	return ((tv1->tv_sec - tv2->tv_sec) +
		((float) (tv1->tv_usec - tv2->tv_usec)) / 1000000);
}

int main(int argc, char **argv)

{
	int	fd;
	struct timeval tv, tv2;
	char buf[SIZE];

	fd = open("fsync-tester.tst-file", O_WRONLY|O_CREAT);
	if (fd < 0) {
		perror("open");
		exit(1);
	}
	memset(buf, 'a', SIZE);
	while (1) {
		pwrite(fd, buf, SIZE, 0);
		gettimeofday(&tv, NULL);
		fsync(fd);
		gettimeofday(&tv2, NULL);
		printf("fsync time: %5.4f\n", timeval_subtract(&tv2, &tv));
		sleep(1);
	}
}
#418 lore raw
Quoted 12 lines · Theodore Tso

More recently (as in this past weekend), I went back to the ext3 problem, and found a better solution, here:

	 http://lkml.org/lkml/2009/3/21/304
	 http://lkml.org/lkml/2009/3/21/302
	 http://lkml.org/lkml/2009/3/21/303

These patches cause the synchronous writes caused by an fsync() to be submitted using WRITE_SYNC, instead of WRITE, which definitely helps in the case where there is a heavy read workload in the background.

Quoted 8 lines

They don't solve the problem where there is a *huge* amount of writes going on, though --- if something is dirtying pages at a rate far greater than the local disk can write it out, say, either "dd if=/dev/zero of=/mnt/make-lots-of-writes" or a massive distcc cluster driving a huge amount of data towards a single system or a wget over a local 100 megabit ethernet from a massive NFS server where everything is in cache, then you can have a major delay with the fsync().

Nice, thanks for the update! The situation isnt nearly as bleak as i feared they are :)

Quoted 6 lines

However, what I've found, though, is that if you're just doing a local copy from one hard drive to another, or downloading a huge iso file from an ftp server over a wide area network, the fsync() delays really don't get *that* bad, even with ext3. At least, I haven't found a workload that doesn't involve either dd if=/dev/zero or a massive amount of data coming in over the

Quoted 8 lines

network that will cause fsync() delays in the > 1-2 second category. Ext3 has been around for a long time, and it's only been the last couple of years that people have really complained about this; my theory is that it was the rise of > 10 megabit ethernets and the use of systems like distcc that really made this problem really become visible. The only realistic workload I've found that triggers this requires a fast network dumping data to a local filesystem.

i think the problem became visible via the rise in memory size, combined with the non-improvement of the performance of rotational disks.

The disk speed versus RAM size ratio has become dramatically worse - and our "5% of RAM" dirty ratio on a 32 GB box is 1.6 GB - which takes an eternity to write out if you happen to sync on that. When we had 1 GB of RAM 5% meant 51 MB - one or two seconds to flush out
- and worse than that, chances are that it's spread out widely on the disk, the whole thing becoming seek-limited as well.

That's where the main difference in perception of this problem comes from i believe. The problem was always there, but only in the last 1-2 years did 4G/8G systems become really common for people to notice.

SSDs will save us eventually, but they will take up to a decade to trickle through for us to forget about this problem altogether.

Ingo

maintainer
#419 lore raw
Quoted 10 lines · Ingo Molnar

i think the problem became visible via the rise in memory size, combined with the non-improvement of the performance of rotational disks.

The disk speed versus RAM size ratio has become dramatically worse - and our "5% of RAM" dirty ratio on a 32 GB box is 1.6 GB - which takes an eternity to write out if you happen to sync on that. When we had 1 GB of RAM 5% meant 51 MB - one or two seconds to flush out
- and worse than that, chances are that it's spread out widely on the disk, the whole thing becoming seek-limited as well.

That's definitely a problem too, but keep in mind that by default the journal gets committed every 5 seconds, so the data gets flushed out that often. So the question is how quickly can you *dirty* 1.6GB of memory?

"dd if=/dev/zero of=/u1/dirty-me-harder" will certainly do it, but normally we're doing something useful, and so you're either copying data from local disk, at which point you're limited by the read speed of your local disk (I suppose it could be in cache, but how common of a case is that?), *or*, you're copying from the network, and to copy in 1.6GB of data in 5 seconds, that means you're moving 320 megabytes/second, which if we're copying in the data from the network, requires a 10 gigabit ethernet.

Hence my statement that this probably became much more visible with fast ethernets --- but you're right, the huge increase in memory sizes was also a key factor; otherwise, write throttling would have kicked in and the VM would have started pushing the dirty pages to disk much sooner.

- Ted

#420 lore raw
Quoted 11 lines · Theodore Tso
Quoted 5 lines · Ingo Molnar

i think the problem became visible via the rise in memory size, combined with the non-improvement of the performance of rotational disks.

The disk speed versus RAM size ratio has become dramatically worse - and our "5% of RAM" dirty ratio on a 32 GB box is 1.6 GB - which takes an eternity to write out if you happen to sync on that. When we had 1 GB of RAM 5% meant 51 MB - one or two seconds to flush out
- and worse than that, chances are that it's spread out widely on
the disk, the whole thing becoming seek-limited as well.

That's definitely a problem too, but keep in mind that by default the journal gets committed every 5 seconds, so the data gets flushed out that often. So the question is how quickly can you *dirty* 1.6GB of memory?

Say it's a file that you allready have in memory cache read in.. there
is plenty of space in 16GB for that.. then you can dirty it at memory-speed.. that about ½sec. (correct me if I'm wrong).

Ok, this is probably unrealistic, but memory grows the largest we have
at the moment is 32GB and its steadily growing with the core-counts.
Then the available memory is used to cache the "active" portion of the
filsystems. I would even say that in the NFS-servers I depend on it to
do this efficiently. (2.6.29-rc8 delivered 1050MB/s over af 10GbitE using nfsd - send speed to multiple clients).

The current workload is based of an active dataset of 600GB where
index'es are being generated and written back to the same disk. So
there is a fairly high read/write load on the machine (as you said was required). The majority (perhaps 550GB ) is only read once where the rest of the time it is stuff in the last 50GB being rewritten.

Quoted 5 lines

"dd if=/dev/zero of=/u1/dirty-me-harder" will certainly do it, but normally we're doing something useful, and so you're either copying data from local disk, at which point you're limited by the read speed of your local disk (I suppose it could be in cache, but how common of a case is that?),

Increasingly the case as memory sizes grows.

Quoted 4 lines

*or*, you're copying from the network, and to copy
in 1.6GB of data in 5 seconds, that means you're moving 320 megabytes/second, which if we're copying in the data from the network, requires a 10 gigabit ethernet.

or just around being processed on the 16-32 cores on the system.

Jesper

Signature · 1 line
Jesper
#421 lore raw
Quoted 6 lines · Jesper Krogh
Quoted 4 lines · Theodore Tso

That's definitely a problem too, but keep in mind that by default the journal gets committed every 5 seconds, so the data gets flushed out that often. So the question is how quickly can you *dirty* 1.6GB of memory?

Doesn't at least ext4 default to the _insane_ model of "data is less important than meta-data, and it doesn't get journalled"?

And ext3 with "data=writeback" does the same, no?

Both of which are - as far as I can tell - total braindamage. At least with ext3 it's not the _default_ mode.

I never understood how anybody doing filesystems (especially ones that claim to be crash-resistant due to journalling) would _ever_ accept the "writeback" behavior of having "clean fsck, but data loss".

Quoted 3 lines

Say it's a file that you allready have in memory cache read in.. there is plenty of space in 16GB for that.. then you can dirty it at memory-speed.. that about ½sec. (correct me if I'm wrong).

No, you'll still have to get per-page locks etc. If you use mmap(), you'll page-fault on each page, if you use write() you'll do all the page lookups etc. But yes, it can be pretty quick - the biggest cost probably _will_ be the speed of memory itself (doing one-byte writes at each block would change that, and the bottle-neck would become the system call and page lookup/locking path, but it's probably in the same rough cost as cost of writing out one page one page).

That said, this is all why we now have 'dirty_*bytes' limits too.

The problem is that the dirty_[background_]bytes value really should be scaled up by the speed of IO. And we currently have no way to do that. Some machines can write a gigabyte in a second with some fancy RAID setups. Others will take minutes (or hours) to do that (crappy SSD's that get 25kB/s throughput on random writes).

The "dirty_[background_ratio" percentage doesn't scale up by the speed of IO either, of course, but at least historically there was generally a pretty good correlation between amount of memory and speed of IO. The machines that had gigs and gigs of RAM tended to always have fast IO too. So scaling up dirty limits by memory size made sense both in the "we have tons of memory, so allow tons of it to be dirty" sense _and_ in the "we likely have a fast disk, so allow more pending dirty data".

Linus

#422 lore raw
In reply to Theodore Tso
Quoted 1 line · Theodore Tso

So the question is how quickly can you *dirty* 1.6GB of memory?

..

MythTV: rm /some/really/huge/video/file ; sync
## disk light stays on for several minutes..

Note quite the same thing, I suppose, but it does break the shutdown scripts of every major Linux distribution.

Simple solution for MythTV is what people already do: use xfs instead.

#423 lore raw
Quoted 9 lines · Mark Lord
Quoted 1 line · Theodore Tso

So the question is how quickly can you *dirty* 1.6GB of memory?

..

MythTV: rm /some/really/huge/video/file ; sync
## disk light stays on for several minutes..

Note quite the same thing, I suppose, but it does break the shutdown scripts of every major Linux distribution.

It is indeed a different issue. ext3 does a fair bit of IO on a (here 60G file) delete:

http://people.redhat.com/~esandeen/rm_test/ext3_rm.png

ext4 is much better:

http://people.redhat.com/~esandeen/rm_test/ext4_rm.png

Quoted 1 line

Simple solution for MythTV is what people already do: use xfs instead.

and yes, xfs does it very quickly:

http://people.redhat.com/~esandeen/rm_test/xfs_rm.png

-Eric

#424 lore raw
In reply to Theodore Tso
Quoted 2 lines

They don't solve the problem where there is a *huge* amount of writes going on, though --- if something is dirtying pages at a rate far

At very high rates other things seem to go pear shaped. I've not traced it back far enough to be sure but what I suspect occurs from the I/O at disk level is that two people are writing stuff out at once - presumably the vm paging pressure and the file system - as I see two streams of I/O that are each reasonably ordered but are interleaved.

Quoted 4 lines

don't get *that* bad, even with ext3. At least, I haven't found a workload that doesn't involve either dd if=/dev/zero or a massive amount of data coming in over the network that will cause fsync() delays in the > 1-2 second category. Ext3 has been around for a long

I see it with a desktop when it pages hard and also when doing heavy desktop I/O (in my case the repeatable every time case is saving large images in the gimp - A4 at 600-1200dpi).

The other one (#8636) seems to be a bug in the I/O schedulers as it goes away if you use a different I/O sched.

Quoted 2 lines

solve. Simply mounting an ext3 filesystem using ext4, without making any change to the filesystem format, should solve the problem.

I will try this experiment but not with production data just yet 8)

Quoted 2 lines

some other users' data files. This was the reason for Stephen Tweedie implementing the data=ordered mode, and making it the default.

Yes and in the server environment or for typical enterprise customers this is a *big issue*, especially the risk of it being undetected that they just inadvertently did something like put your medical data into the end of something public during a crash.

Quoted 1 line

Try ext4, I think you'll like it. :-)

I need to, so that I can double check none of the open jbd locking bugs are there and close more bugzilla entries (#8147)

Thanks for the reply - I hadn't realised a lot of this was getting fixed but in ext4 and quietly

Alan

maintainer
#425 lore raw
Quoted 6 lines · Alan Cox

At very high rates other things seem to go pear shaped. I've not traced it back far enough to be sure but what I suspect occurs from the I/O at disk level is that two people are writing stuff out at once - presumably the vm paging pressure and the file system - as I see two streams of I/O that are each reasonably ordered but are interleaved.

Surely the elevator should have reordered the writes reasonably? (Or is that what you meant by "the other one -- #8636 (I assume this is a kernel Bugzilla #?) seems to be a bug in the I/O schedulers as it goes away if you use a different I/O sched.?")

Quoted 8 lines
Quoted 4 lines

don't get *that* bad, even with ext3. At least, I haven't found a workload that doesn't involve either dd if=/dev/zero or a massive amount of data coming in over the network that will cause fsync() delays in the > 1-2 second category. Ext3 has been around for a long

I see it with a desktop when it pages hard and also when doing heavy desktop I/O (in my case the repeatable every time case is saving large images in the gimp - A4 at 600-1200dpi).

Yeah, I could see that doing it. How big is the image, and out of curiosity, can you run the fsync-tester.c program I posted while saving the gimp image, and tell me how much of a delay you end up seeing?

Quoted 4 lines
Quoted 2 lines

solve. Simply mounting an ext3 filesystem using ext4, without making any change to the filesystem format, should solve the problem.

I will try this experiment but not with production data just yet 8)

Where's your bravery, man? :-)

I've been using it on my laptop since July, and haven't lost significant amounts of data yet. (The only thing I did lose was bits of a git repository fairly early on, and I was able to repair by replacing the missing objects.)

Quoted 7 lines
Quoted 2 lines

some other users' data files. This was the reason for Stephen Tweedie implementing the data=ordered mode, and making it the default.

Yes and in the server environment or for typical enterprise customers this is a *big issue*, especially the risk of it being undetected that they just inadvertently did something like put your medical data into the end of something public during a crash.

True enough; changing the defaults to be data=writeback for the server environment is probably not a good idea. (Then again, in the server environment most of the workloads generally don't end up hitting the nasty data=ordered failure modes; they tend to be transaction-oriented, and fsync().)

Quoted 4 lines
Quoted 1 line

Try ext4, I think you'll like it. :-)

I need to, so that I can double check none of the open jbd locking bugs are there and close more bugzilla entries (#8147)

More testing would be appreciated --- and yeah, we need to groom the bugzilla. For a long time no one in ext3 land was paying attention to bugzilla, and more recently I've been trying to keep up with the ext4-related bugs, but I don't get to do ext4 work full-time, and occasionally Stacey gets annoyed at me when I work late into night...

Quoted 2 lines

Thanks for the reply - I hadn't realised a lot of this was getting fixed but in ext4 and quietly

Yeah, there are a bunch of things, like the barrier=1 default, which akpm has rejected for ext3, but which we've fixed in ext4. More help in shaking down the bugs would definitely be appreciated.

- Ted

#426 lore raw
Quoted 4 lines

Surely the elevator should have reordered the writes reasonably? (Or is that what you meant by "the other one -- #8636 (I assume this is a kernel Bugzilla #?) seems to be a bug in the I/O schedulers as it goes away if you use a different I/O sched.?")

There are two cases there. One is a bug #8636 (kernel bugzilla) which is where things like dump show awful performance with certain I/O scheduler settings. That seems to be totally not connected to the fs but it is a problem (and has a patch)

The second one the elevator is clearly trying to sort out but its behaving as if someone is writing the file starting at say 0 and someone else is trying to write it back starting some large distance further down the file. The elevator can only do so much then.

Quoted 2 lines

Yeah, I could see that doing it. How big is the image, and out of curiosity, can you run the fsync-tester.c program I posted while

150MB+ for the pnm files from gimp used as temporaries by Eve (Etch Validation Engine), more like 10MB for xcf/tif files.

Quoted 2 lines

saving the gimp image, and tell me how much of a delay you end up seeing?

Added to the TODO list once I can set up a suitable test box (my new dev box is somewhere between Dell and my desk right now)

Quoted 2 lines

More testing would be appreciated --- and yeah, we need to groom the bugzilla.

I'm currently doing this on a large scale (closed about 300 so far this run). Bug 8147 might be worth a look as its a case where the jbd locking and the jbd comments seem to disagree (the comments say you must hold a lock but we don't seem to)

reviewer
#427 lore raw
In reply to Alan Cox
Quoted 8 lines
Quoted 2 lines

They don't solve the problem where there is a *huge* amount of writes going on, though --- if something is dirtying pages at a rate far

At very high rates other things seem to go pear shaped. I've not traced it back far enough to be sure but what I suspect occurs from the I/O at disk level is that two people are writing stuff out at once - presumably the vm paging pressure and the file system - as I see two streams of I/O that are each reasonably ordered but are interleaved.

There are different problems leading to this:
1) JBD commit code writes ordered data on each transaction commit. This is done in dirtied-time order which is not necessarily optimal in case of random access IO. IO scheduler helps here though because we submit a lot of IO at once. ext4 has at least the randomness part of this problem "fixed" because it submits ordered data via writepages(). Doing this change requires non-trivial changes to the journaling layer so I wasn't brave enough to do it with ext3 and JBD as well (although porting the patch is trivial).

2) When we do dirty throttling, there are going to be several threads writing out on the filesystem (if you have more pdflush threads which translates to having more than one CPU). Jens' per-BDI writeback threads could help here (but I haven't yet got to reading his patches in detail to be sure).

These two problems together result in non-optimal IO pattern. At least
that's where I got to when I was looking into why Berkeley DB is so slow. I was trying to somehow serialize more pdflush threads on the filesystem but a stupid solution does not really help much - either I was starving some throttled thread by other threads doing writeback or I didn't quite keep the disk busy. So something like Jens' approach is probably the way to go in the end.

Quoted 25 lines
Quoted 4 lines

don't get *that* bad, even with ext3. At least, I haven't found a workload that doesn't involve either dd if=/dev/zero or a massive amount of data coming in over the network that will cause fsync() delays in the > 1-2 second category. Ext3 has been around for a long

I see it with a desktop when it pages hard and also when doing heavy desktop I/O (in my case the repeatable every time case is saving large images in the gimp - A4 at 600-1200dpi).

The other one (#8636) seems to be a bug in the I/O schedulers as it goes away if you use a different I/O sched.

Quoted 2 lines

solve. Simply mounting an ext3 filesystem using ext4, without making any change to the filesystem format, should solve the problem.

I will try this experiment but not with production data just yet 8)

Quoted 2 lines

some other users' data files. This was the reason for Stephen Tweedie implementing the data=ordered mode, and making it the default.

Yes and in the server environment or for typical enterprise customers this is a *big issue*, especially the risk of it being undetected that they just inadvertently did something like put your medical data into the end of something public during a crash.

Quoted 4 lines
Quoted 1 line

Try ext4, I think you'll like it. :-)

I need to, so that I can double check none of the open jbd locking bugs are there and close more bugzilla entries (#8147)

This one is still there. I'll have a look at it tomorrow and hopefully
will be able to answer...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#428 lore raw
In reply to Theodore Tso
Quoted 5 lines · Theodore Tso

Try ext4, I think you'll like it. :-)

Failing that, data=writeback for single-user machines is probably your best bet.

Isn't that the same fix? ext4 just defaults to the crappy "writeback" behavior, which is insane.

Sure, it makes things _much_ smoother, since now the actual data is no longer in the critical path for any journal writes, but anybody who thinks that's a solution is just incompetent.

We might as well go back to ext2 then. If your data gets written out long after the metadata hit the disk, you are going to hit all kinds of bad issues if the machine ever goes down.

Linus

maintainer
#429 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Theodore Tso

Try ext4, I think you'll like it. :-)

Failing that, data=writeback for single-user machines is probably your best bet.

Isn't that the same fix? ext4 just defaults to the crappy "writeback" behavior, which is insane.

Technically, it's not data=writeback. It's more like XFS's delayed allocation; I've added workarounds so that files that which are replaced via truncate or rename get pushed out right away, which should solve most of the problems involved with files becoming zero-length after a system crash.

Quoted 7 lines

Sure, it makes things _much_ smoother, since now the actual data is no longer in the critical path for any journal writes, but anybody who thinks that's a solution is just incompetent.

We might as well go back to ext2 then. If your data gets written out long after the metadata hit the disk, you are going to hit all kinds of bad issues if the machine ever goes down.

With ext2 after a system crash you need to run fsck. With ext4, fsck isn't an issue, but if the application doesn't use fsync(), yes, there's no guarantee (other than the workarounds for replace-via-truncate and replace-via-rename), but there's plenty of prior history that says that applications that care about data hitting the disk should use fsync(). Otherwise, it will get spread out over a few minutes; and for some files, that really won't make a difference.

For precious files, applications that use fsync() will be safe --- otherwise, even with ext3, you can end up losing the contents of the file if you crash right before 5 second commit window. At least back in the days when people were proud of their Linux systems having 2-3 year uptimes, and where jiffies could actually wrap from time to time, the difference between 5 seconds and 3 minutes really wasn't that big of a deal. People who really care about this can turn off delayed allocation with the nodelalloc mount option. Of course then they will have the ext3 slower fsync() problem.

You are right that data=writeback and delayed allocation do both mean that data can get pushed out much later than the metadata. But that's allowed by POSIX, and it does give some very nice performance benefits.

With either data=writeback or delayed allocation, we can also adjust the default commit interval and the writeback timer settings; if we say, change the default commit interval to be 30 seconds, and change the writeback expire interval to be 15 seconds, it will also smooth out the writes significantly. So that's yet another solution, with a different set of tradeoffs.

Depending on the set of applications someone is running on their system, running and the reliability of their hardware/power/system in general, different tradeoffs will be more or less appropriate for the system administrator in question.

- Ted

#430 lore raw
Quoted 3 lines · Theodore Tso

With ext2 after a system crash you need to run fsck. With ext4, fsck isn't an issue,

Bah. A corrupt filesystem is a corrupt filesystem. Whether you have to fsck it or not should be a secondary concern.

I personally find silent corruption to be _worse_ than the non-silent one. At least if there's some program that says "oops, your inode so-and-so seems to be scrogged" that's better than just silently having bad data in it.

Of course, never having bad data _nor_ needing fsck is clearly optimal. data=ordered gets pretty close (and data=journal is unacceptable for performance reasons).

But I really don't understand filesystem people who think that "fsck" is the important part, regardless of whether the data is valid or not. That's just stupid and _obviously_ bogus.

Linus

#431 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Theodore Tso

With ext2 after a system crash you need to run fsck. With ext4, fsck isn't an issue,

Bah. A corrupt filesystem is a corrupt filesystem. Whether you have to fsck it or not should be a secondary concern.

Quoted 8 lines

I personally find silent corruption to be _worse_ than the non-silent one. At least if there's some program that says "oops, your inode so-and-so seems to be scrogged" that's better than just silently having bad data in it.

Of course, never having bad data _nor_ needing fsck is clearly optimal. data=ordered gets pretty close (and data=journal is unacceptable for performance reasons).

But I really don't understand filesystem people who think that "fsck" is the important part, regardless of whether the data is valid or not. That's just stupid and _obviously_ bogus.

Linus

It is always interesting to try to explain to users that just because fsck ran cleanly does not mean anything that they care about is actually safely on disk. The speed that fsck can run at is important when you are trying to recover data from a really hosed file system, but that is thankfully relatively rare for most people.

Having been involved in many calls with customers after crashes, what they really want to know is pretty routine - do you have all of the data I wrote? can you prove that it is the same data that I wrote? if not, what data is missing and needs to be restored?

We can get help answer those questions with checksums or digital hashes to validate the actual user data of files (open question is when to compute it, where to store, would the SCSI T10 DIF/DIX stuff be sufficient), putting in place some background scrubbers to detect corruptions (which can happen even without an IO error), etc.

Being able to pin point what was impacted is actually enormously useful - for example, being able to map a bad sector back into some meaningful object like a user file, meta-data (translation, run fsck) or so on.

Ric

#432 lore raw
In reply to Linus Torvalds
Quoted 1 line · Linus Torvalds

But I really don't understand filesystem people who think that "fsck" is the important part, regardless of whether the data is valid or not. That's just stupid and _obviously_ bogus.

I think I can understand that point of view, at least:

More customers complain about hours-long fsck times than they do about silent data corruption of non-fsync'd files.

Quoted 3 lines

The point is, if you write your metadata earlier (say, every 5 sec) and the real data later (say, every 30 sec), you're actually MORE LIKELY to see corrupt files than if you try to write them together.

And if you write your data _first_, you're never going to see corruption at all.

Amen.

And, personal filesystem pet peeve: please encourage proper FLUSH CACHE use to give users the data guarantees they deserve. Linux's sync(2) and fsync(2) (and fdatasync, etc.) should poke the block layer to guarantee a media write.

Jeff

P.S. Overall, I am thrilled that this ext3/ext4 transition and associated slashdotting has spurred debate over filesystem data guarantees. This is the kind of discussion that has needed to happen for years, IMO.

#433 lore raw
Quoted 18 lines · Jeff Garzik
Quoted 3 lines · Linus Torvalds

But I really don't understand filesystem people who think that "fsck" is the important part, regardless of whether the data is valid or not. That's just stupid and _obviously_ bogus.

I think I can understand that point of view, at least:

More customers complain about hours-long fsck times than they do about silent data corruption of non-fsync'd files.

Quoted 6 lines

The point is, if you write your metadata earlier (say, every 5 sec) and the real data later (say, every 30 sec), you're actually MORE LIKELY to see corrupt files than if you try to write them together.

And if you write your data _first_, you're never going to see corruption at all.

Quoted 6 lines

Amen.

And, personal filesystem pet peeve: please encourage proper FLUSH CACHE use to give users the data guarantees they deserve. Linux's sync(2) and fsync(2) (and fdatasync, etc.) should poke the block layer to guarantee a media write.

I completely agree. This also applies to nfsd_sync, by the way. What's the right place to implement that?
How about sync_blockdev?

Benny

Quoted 16 lines

Jeff

P.S. Overall, I am thrilled that this ext3/ext4 transition and associated slashdotting has spurred debate over filesystem data guarantees. This is the kind of discussion that has needed to happen for years, IMO.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 9 lines
Benny Halevy
Software Architect
Panasas, Inc.
[email protected]
Tel/Fax: +972-3-647-8340
Mobile: +972-54-802-8340

Panasas: The Leader in Parallel Storage
www.panasas.com
#434 lore raw
In reply to Jeff Garzik
Quoted 18 lines · Jeff Garzik
Quoted 3 lines · Linus Torvalds

But I really don't understand filesystem people who think that "fsck" is the important part, regardless of whether the data is valid or not. That's just stupid and _obviously_ bogus.

I think I can understand that point of view, at least:

More customers complain about hours-long fsck times than they do about silent data corruption of non-fsync'd files.

Quoted 6 lines

The point is, if you write your metadata earlier (say, every 5 sec) and the real data later (say, every 30 sec), you're actually MORE LIKELY to see corrupt files than if you try to write them together.

And if you write your data _first_, you're never going to see corruption at all.

Quoted 6 lines

Amen.

And, personal filesystem pet peeve: please encourage proper FLUSH CACHE use to give users the data guarantees they deserve. Linux's sync(2) and fsync(2) (and fdatasync, etc.) should poke the block layer to guarantee a media write.

fsync already does that, at least if you have barriers enabled on your drive.

Signature · 1 line
Jens Axboe
#435 lore raw
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Jeff Garzik
Quoted 1 line · Linus Torvalds

But I really don't understand filesystem people who think that "fsck" is the important part, regardless of whether the data is valid or not. That's just stupid and _obviously_ bogus.

I think I can understand that point of view, at least:

More customers complain about hours-long fsck times than they do about silent data corruption of non-fsync'd files.

Quoted 2 lines

The point is, if you write your metadata earlier (say, every 5 sec) and the real data later (say, every 30 sec), you're actually MORE LIKELY to see corrupt files than if you try to write them together.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

And if you write your data _first_, you're never going to see corruption at all.

Amen.

And, personal filesystem pet peeve: please encourage proper FLUSH CACHE use to give users the data guarantees they deserve. Linux's sync(2) and fsync(2) (and fdatasync, etc.) should poke the block layer to guarantee a media write.

fsync already does that, at least if you have barriers enabled on your drive.

Erm, no, you don't enable barriers on your drive, they are not a hardware feature. You enable barriers via your filesystem.

Stating "fsync already does that" borders on false, because that assumes
(a) the user has a fs that supports barriers
(b) the user is actually aware of a 'barriers' mount option and what it means
(c) the user has turned on an option normally defaulted to off.

Or in other words, it pretty much never happens.

Furthermore, a blatantly obvious place to flush data to media -- fsync(2), fdatasync(2) and sync_file_range(2) -- should cause the block layer to issue a FLUSH CACHE for __any__ filesystem. But that doesn't happen either.

So, no, for 95% of Linux users, fsync does _not_ already do that. If you are lucky enough to use XFS or ext4, you're covered. That's it.

Jeff

#436 lore raw
Quoted 25 lines · Jeff Garzik
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Jeff Garzik
Quoted 3 lines · Linus Torvalds

But I really don't understand filesystem people who think that "fsck" is the important part, regardless of whether the data is valid or not. That's just stupid and _obviously_ bogus.

I think I can understand that point of view, at least:

More customers complain about hours-long fsck times than they do about silent data corruption of non-fsync'd files.

Quoted 6 lines

The point is, if you write your metadata earlier (say, every 5 sec) and the real data later (say, every 30 sec), you're actually MORE LIKELY to see corrupt files than if you try to write them together.

And if you write your data _first_, you're never going to see corruption at all.

Amen.

And, personal filesystem pet peeve: please encourage proper FLUSH CACHE use to give users the data guarantees they deserve. Linux's sync(2) and fsync(2) (and fdatasync, etc.) should poke the block layer to guarantee a media write.

Quoted 5 lines
Quoted 2 lines

fsync already does that, at least if you have barriers enabled on your drive.

Erm, no, you don't enable barriers on your drive, they are not a hardware feature. You enable barriers via your filesystem.

Thanks for the lesson Jeff, I'm obviously not aware how that stuff works...

Quoted 7 lines

Stating "fsync already does that" borders on false, because that assumes
(a) the user has a fs that supports barriers
(b) the user is actually aware of a 'barriers' mount option and what it means
(c) the user has turned on an option normally defaulted to off.

Or in other words, it pretty much never happens.

That is true, except if you use xfs/ext4. And this discussion is fine, as was the one a few months back that got ext4 to enable barriers by default. If I had submitted patches to do that back in 2001/2 when the barrier stuff was written, I would have been shot for introducing such a slow down. After people found out that it just wasn't something silly, then you have a way to enable it.

I'd still wager that most people would rather have a 'good enough fsync' on their desktops than incur the penalty of barriers or write through caching. I know I do.

Quoted 7 lines

Furthermore, a blatantly obvious place to flush data to media -- fsync(2), fdatasync(2) and sync_file_range(2) -- should cause the block layer to issue a FLUSH CACHE for __any__ filesystem. But that doesn't happen either.

So, no, for 95% of Linux users, fsync does _not_ already do that. If you are lucky enough to use XFS or ext4, you're covered. That's it.

The point is that you need to expose this choice somewhere, and that 'somewhere' isn't manually editing fstab and enabling barriers or fsync-for-real. And it should be easier.

Another problem is that FLUSH_CACHE sucks. Really. And not just on ext3/ordered, generally. Write a 50 byte file, fsync, flush cache and wit for the world to finish. Pretty hard to teach people to use a nicer fdatasync(), when the majority of the cost now becomes flushing the cache of that 1TB drive you happen to have 8 partitions on. Good luck with that.

Signature · 1 line
Jens Axboe
#437 lore raw
Quoted 63 lines · Jens Axboe
Quoted 18 lines · Jeff Garzik
Quoted 16 lines · Jens Axboe
Quoted 14 lines · Jeff Garzik
Quoted 2 lines · Linus Torvalds

But I really don't understand filesystem people who think that "fsck" is the important part, regardless of whether the data is valid or not. That's just stupid and _obviously_ bogus.

I think I can understand that point of view, at least:

More customers complain about hours-long fsck times than they do about silent data corruption of non-fsync'd files.

Quoted 4 lines

The point is, if you write your metadata earlier (say, every 5 sec) and the real data later (say, every 30 sec), you're actually MORE LIKELY to see corrupt files than if you try to write them together.

And if you write your data _first_, you're never going to see corruption at all.

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

Amen.

And, personal filesystem pet peeve: please encourage proper FLUSH CACHE use to give users the data guarantees they deserve. Linux's sync(2) and fsync(2) (and fdatasync, etc.) should poke the block layer to guarantee a media write.

fsync already does that, at least if you have barriers enabled on your drive.

Erm, no, you don't enable barriers on your drive, they are not a hardware feature. You enable barriers via your filesystem.

Thanks for the lesson Jeff, I'm obviously not aware how that stuff works...

Quoted 7 lines

Stating "fsync already does that" borders on false, because that assumes
(a) the user has a fs that supports barriers
(b) the user is actually aware of a 'barriers' mount option and what it means
(c) the user has turned on an option normally defaulted to off.

Or in other words, it pretty much never happens.

That is true, except if you use xfs/ext4. And this discussion is fine, as was the one a few months back that got ext4 to enable barriers by default. If I had submitted patches to do that back in 2001/2 when the barrier stuff was written, I would have been shot for introducing such a slow down. After people found out that it just wasn't something silly, then you have a way to enable it.

I'd still wager that most people would rather have a 'good enough fsync' on their desktops than incur the penalty of barriers or write through caching. I know I do.

Quoted 4 lines

Furthermore, a blatantly obvious place to flush data to media -- fsync(2), fdatasync(2) and sync_file_range(2) -- should cause the block layer to issue a FLUSH CACHE for __any__ filesystem. But that doesn't happen either.

So, no, for 95% of Linux users, fsync does _not_ already do that. If you are lucky enough to use XFS or ext4, you're covered. That's it.

The point is that you need to expose this choice somewhere, and that 'somewhere' isn't manually editing fstab and enabling barriers or fsync-for-real. And it should be easier.

Quoted 8 lines

Another problem is that FLUSH_CACHE sucks. Really. And not just on ext3/ordered, generally. Write a 50 byte file, fsync, flush cache and wit for the world to finish. Pretty hard to teach people to use a nicer fdatasync(), when the majority of the cost now becomes flushing the cache of that 1TB drive you happen to have 8 partitions on. Good luck with that.

And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

ric

#438 lore raw
Quoted 88 lines · Ric Wheeler
Quoted 79 lines · Jens Axboe
Quoted 32 lines · Jeff Garzik
Quoted 30 lines · Jens Axboe
Quoted 28 lines · Jeff Garzik
Quoted 4 lines · Linus Torvalds

But I really don't understand filesystem people who think that "fsck" is the important part, regardless of whether the data is valid or not. That's just stupid and _obviously_ bogus.

I think I can understand that point of view, at least:

More customers complain about hours-long fsck times than they do about silent data corruption of non-fsync'd files.

Quoted 8 lines

The point is, if you write your metadata earlier (say, every 5 sec) and the real data later (say, every 30 sec), you're actually MORE LIKELY to see corrupt files than if you try to write them together.

And if you write your data _first_, you're never going to see corruption at all.

Amen.

And, personal filesystem pet peeve: please encourage proper FLUSH CACHE use to give users the data guarantees they deserve. Linux's sync(2) and fsync(2) (and fdatasync, etc.) should poke the block layer to guarantee a media write.

Quoted 6 lines
Quoted 3 lines

fsync already does that, at least if you have barriers enabled on your drive.

Erm, no, you don't enable barriers on your drive, they are not a hardware feature. You enable barriers via your filesystem.

Thanks for the lesson Jeff, I'm obviously not aware how that stuff works...

Quoted 8 lines

Stating "fsync already does that" borders on false, because that assumes
(a) the user has a fs that supports barriers
(b) the user is actually aware of a 'barriers' mount option and what it means
(c) the user has turned on an option normally defaulted to off.

Or in other words, it pretty much never happens.

That is true, except if you use xfs/ext4. And this discussion is fine, as was the one a few months back that got ext4 to enable barriers by default. If I had submitted patches to do that back in 2001/2 when the barrier stuff was written, I would have been shot for introducing such a slow down. After people found out that it just wasn't something silly, then you have a way to enable it.

I'd still wager that most people would rather have a 'good enough fsync' on their desktops than incur the penalty of barriers or write through caching. I know I do.

Quoted 8 lines

Furthermore, a blatantly obvious place to flush data to media -- fsync(2), fdatasync(2) and sync_file_range(2) -- should cause the block layer to issue a FLUSH CACHE for __any__ filesystem. But that doesn't happen either.

So, no, for 95% of Linux users, fsync does _not_ already do that. If
you are lucky enough to use XFS or ext4, you're covered. That's it.

The point is that you need to expose this choice somewhere, and that 'somewhere' isn't manually editing fstab and enabling barriers or fsync-for-real. And it should be easier.

Quoted 8 lines

Another problem is that FLUSH_CACHE sucks. Really. And not just on ext3/ordered, generally. Write a 50 byte file, fsync, flush cache and wit for the world to finish. Pretty hard to teach people to use a nicer fdatasync(), when the majority of the cost now becomes flushing the cache of that 1TB drive you happen to have 8 partitions on. Good luck with that.

Quoted 5 lines

And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

Exactly, that's what my (vague) 8 partition reference was for :-) A range flush would be so much more palatable.

Signature · 1 line
Jens Axboe
#439 lore raw
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Ric Wheeler
Quoted 8 lines · Jens Axboe

Another problem is that FLUSH_CACHE sucks. Really. And not just on ext3/ordered, generally. Write a 50 byte file, fsync, flush cache and wit for the world to finish. Pretty hard to teach people to use a nicer fdatasync(), when the majority of the cost now becomes flushing the cache of that 1TB drive you happen to have 8 partitions on. Good luck with that.

Quoted 8 lines
Quoted 5 lines

And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

Exactly, that's what my (vague) 8 partition reference was for :-) A range flush would be so much more palatable.

Tangential question, but am I right in thinking that BIO_RW_BARRIER similarly bars across all partitions, whereas its WRITE_BARRIER and DISCARD_BARRIER users would actually prefer it to apply to just one?

Hugh

#440 lore raw
Quoted 17 lines · Hugh Dickins
Quoted 16 lines · Jens Axboe
Quoted 14 lines · Ric Wheeler
Quoted 8 lines · Jens Axboe

Another problem is that FLUSH_CACHE sucks. Really. And not just on ext3/ordered, generally. Write a 50 byte file, fsync, flush cache and wit for the world to finish. Pretty hard to teach people to use a nicer fdatasync(), when the majority of the cost now becomes flushing the cache of that 1TB drive you happen to have 8 partitions on. Good luck with that.

And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

Quoted 6 lines
Quoted 2 lines

Exactly, that's what my (vague) 8 partition reference was for :-) A range flush would be so much more palatable.

Tangential question, but am I right in thinking that BIO_RW_BARRIER similarly bars across all partitions, whereas its WRITE_BARRIER and DISCARD_BARRIER users would actually prefer it to apply to just one?

All the barriers refer to just that range which the barrier itself references. The problem with the full device flushes is implementation on the hardware side, since we can't do small range flushes. So it's not as-designed, but rather the best we can do...

Signature · 1 line
Jens Axboe
#441 lore raw
Quoted 8 lines · Jens Axboe
Quoted 4 lines · Hugh Dickins

Tangential question, but am I right in thinking that BIO_RW_BARRIER similarly bars across all partitions, whereas its WRITE_BARRIER and DISCARD_BARRIER users would actually prefer it to apply to just one?

All the barriers refer to just that range which the barrier itself references.

Ah, thank you: then I had a fundamental misunderstanding of them, and need to go away and work that out some more.

Though I didn't read it before asking, doesn't the I/O Barriers section of Documentation/block/biodoc.txt give a very different impression?

Quoted 3 lines

The problem with the full device flushes is implementation on the hardware side, since we can't do small range flushes. So it's not as-designed, but rather the best we can do...

Right, that part of it I did get.

Hugh

#442 lore raw
Quoted 7 lines · Hugh Dickins
Quoted 6 lines · Jens Axboe
Quoted 4 lines · Hugh Dickins

Tangential question, but am I right in thinking that BIO_RW_BARRIER similarly bars across all partitions, whereas its WRITE_BARRIER and DISCARD_BARRIER users would actually prefer it to apply to just one?

Quoted 8 lines
Quoted 2 lines

All the barriers refer to just that range which the barrier itself references.

Ah, thank you: then I had a fundamental misunderstanding of them, and need to go away and work that out some more.

Though I didn't read it before asking, doesn't the I/O Barriers section of Documentation/block/biodoc.txt give a very different impression?

I'm sensing a miscommunication here... The ordering constraint is across devices, at least that is how it is implemented. For file system barriers (like BIO_RW_BARRIER), it could be per-partition instead. Doing so would involve some changes at the block layer side, not necessarily trivial. So I think you were asking about ordering, I was answering about the write guarantee :-)

Signature · 1 line
Jens Axboe
#443 lore raw
Quoted 17 lines · Jens Axboe
Quoted 7 lines · Hugh Dickins
Quoted 6 lines · Jens Axboe
Quoted 4 lines · Hugh Dickins

Tangential question, but am I right in thinking that BIO_RW_BARRIER similarly bars across all partitions, whereas its WRITE_BARRIER and DISCARD_BARRIER users would actually prefer it to apply to just one?

Quoted 8 lines
Quoted 2 lines

All the barriers refer to just that range which the barrier itself references.

Ah, thank you: then I had a fundamental misunderstanding of them, and need to go away and work that out some more.

Though I didn't read it before asking, doesn't the I/O Barriers section of Documentation/block/biodoc.txt give a very different impression?

Quoted 6 lines

I'm sensing a miscommunication here... The ordering constraint is across devices, at least that is how it is implemented. For file system barriers (like BIO_RW_BARRIER), it could be per-partition instead. Doing so would involve some changes at the block layer side, not necessarily trivial. So I think you were asking about ordering, I was answering about the write guarantee :-)

Ah, thank you again, perhaps I did understand after all.

So, directing a barrier (WRITE_BARRIER or DISCARD_BARRIER) to a range of sectors in one partition interposes a barrier into the queue of I/O across (all partitions of) that whole device.

I think that's not how filesystems really want barriers to behave, and might tend to discourage us from using barriers more freely. But I have zero appreciation of whether it's a significant issue worth non-trivial change - just wanted to get it out into the open.

Hugh

#444 lore raw
Quoted 25 lines · Hugh Dickins
Quoted 17 lines · Jens Axboe
Quoted 7 lines · Hugh Dickins
Quoted 6 lines · Jens Axboe
Quoted 4 lines · Hugh Dickins

Tangential question, but am I right in thinking that BIO_RW_BARRIER similarly bars across all partitions, whereas its WRITE_BARRIER and DISCARD_BARRIER users would actually prefer it to apply to just one?

Quoted 8 lines
Quoted 2 lines

All the barriers refer to just that range which the barrier itself references.

Ah, thank you: then I had a fundamental misunderstanding of them, and need to go away and work that out some more.

Though I didn't read it before asking, doesn't the I/O Barriers section of Documentation/block/biodoc.txt give a very different impression?

Quoted 6 lines

I'm sensing a miscommunication here... The ordering constraint is across devices, at least that is how it is implemented. For file system barriers (like BIO_RW_BARRIER), it could be per-partition instead. Doing so would involve some changes at the block layer side, not necessarily trivial. So I think you were asking about ordering, I was answering about the write guarantee :-)

Quoted 5 lines

Ah, thank you again, perhaps I did understand after all.

So, directing a barrier (WRITE_BARRIER or DISCARD_BARRIER) to a range of sectors in one partition interposes a barrier into the queue of I/O across (all partitions of) that whole device.

Correct

Quoted 4 lines

I think that's not how filesystems really want barriers to behave, and might tend to discourage us from using barriers more freely. But I have zero appreciation of whether it's a significant issue worth non-trivial change - just wanted to get it out into the open.

Per-partition definitely makes sense. The problem is that we do sorting on a per-device basis right now. But it's a good point, I'll try and take a look at how much work it would be to make it per-partition instead. It wont be trivial :-)

Signature · 1 line
Jens Axboe
#445 lore raw
Quoted 13 lines · Jens Axboe
Quoted 4 lines · Hugh Dickins

So, directing a barrier (WRITE_BARRIER or DISCARD_BARRIER) to a range of sectors in one partition interposes a barrier into the queue of I/O across (all partitions of) that whole device.

Correct

Quoted 4 lines

I think that's not how filesystems really want barriers to behave, and might tend to discourage us from using barriers more freely. But I have zero appreciation of whether it's a significant issue worth non-trivial change - just wanted to get it out into the open.

Quoted 4 lines

Per-partition definitely makes sense. The problem is that we do sorting on a per-device basis right now. But it's a good point, I'll try and take a look at how much work it would be to make it per-partition instead. It wont be trivial :-)

Thanks, that would be interesting.
Trivial bores you anyway, doesn't it?

Hugh

#446 lore raw
Quoted 14 lines · Hugh Dickins
Quoted 13 lines · Jens Axboe
Quoted 4 lines · Hugh Dickins

So, directing a barrier (WRITE_BARRIER or DISCARD_BARRIER) to a range of sectors in one partition interposes a barrier into the queue of I/O across (all partitions of) that whole device.

Correct

Quoted 4 lines

I think that's not how filesystems really want barriers to behave, and might tend to discourage us from using barriers more freely. But I have zero appreciation of whether it's a significant issue worth non-trivial change - just wanted to get it out into the open.

Quoted 7 lines
Quoted 4 lines

Per-partition definitely makes sense. The problem is that we do sorting on a per-device basis right now. But it's a good point, I'll try and take a look at how much work it would be to make it per-partition instead. It wont be trivial :-)

Thanks, that would be interesting.
Trivial bores you anyway, doesn't it?

You're a good motivator, Hugh!

Signature · 1 line
Jens Axboe
#447 lore raw
In reply to Ric Wheeler

Ric Wheeler wrote:> And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE

Quoted 3 lines

is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

SCSI'S SYNCHRONIZE CACHE command already accepts an (LBA, length) pair.
We could make use of that.

And I bet we could convince T13 to add FLUSH CACHE RANGE, if we could demonstrate clear benefit.

Jeff

#448 lore raw
Quoted 10 lines · Jeff Garzik

Ric Wheeler wrote:> And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE

Quoted 3 lines

is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

SCSI'S SYNCHRONIZE CACHE command already accepts an (LBA, length) pair. We could make use of that.

And I bet we could convince T13 to add FLUSH CACHE RANGE, if we could demonstrate clear benefit.

Jeff

How well supported is this in SCSI? Can we try it out with a commodity SAS drive?

Ric

#449 lore raw
Quoted 11 lines · Ric Wheeler
Quoted 10 lines · Jeff Garzik

Ric Wheeler wrote:> And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE

Quoted 4 lines

is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

SCSI'S SYNCHRONIZE CACHE command already accepts an (LBA, length) pair. We could make use of that.

Quoted 7 lines
Quoted 4 lines

And I bet we could convince T13 to add FLUSH CACHE RANGE, if we could demonstrate clear benefit.

Jeff

How well supported is this in SCSI? Can we try it out with a commodity SAS drive?

What do you mean by well supported? The way the SCSI standard is written, a device can do a complete cache flush when a range flush is requested and still be fully standards compliant. There's no easy way to tell if it does a complete cache flush every time other than by taking the firmware apart (or asking the manufacturer).

James

#450 lore raw
Quoted 17 lines · James Bottomley
Quoted 16 lines · Ric Wheeler
Quoted 14 lines · Jeff Garzik

Ric Wheeler wrote:> And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE

Quoted 4 lines

is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

SCSI'S SYNCHRONIZE CACHE command already accepts an (LBA, length) pair. We could make use of that.

And I bet we could convince T13 to add FLUSH CACHE RANGE, if we could demonstrate clear benefit.

Jeff

Quoted 8 lines
Quoted 2 lines

How well supported is this in SCSI? Can we try it out with a commodity SAS drive?

What do you mean by well supported? The way the SCSI standard is written, a device can do a complete cache flush when a range flush is requested and still be fully standards compliant. There's no easy way to tell if it does a complete cache flush every time other than by taking the firmware apart (or asking the manufacturer).

That's the fear of range flushes, if it was added to t13 as well. Unless that Other OS uses range flushes, most firmware writers would most likely implement any range as 0...-1 and it wouldn't help us at all. In fact it would make things worse, as we would have done extra work to actually find these ranges, unless you went cheap and said 'just flush this partition'.

Signature · 1 line
Jens Axboe
#451 lore raw

range-based cache flushing (was Re: Linux 2.6.29)

In reply to James Bottomley
Quoted 10 lines · James Bottomley
Quoted 8 lines · Ric Wheeler
Quoted 7 lines · Jeff Garzik

Ric Wheeler wrote:> And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE

Quoted 3 lines

is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

SCSI'S SYNCHRONIZE CACHE command already accepts an (LBA, length) pair. We could make use of that.

And I bet we could convince T13 to add FLUSH CACHE RANGE, if we could demonstrate clear benefit.

Quoted 7 lines
Quoted 1 line

How well supported is this in SCSI? Can we try it out with a commodity SAS drive?

What do you mean by well supported? The way the SCSI standard is written, a device can do a complete cache flush when a range flush is requested and still be fully standards compliant. There's no easy way to tell if it does a complete cache flush every time other than by taking the firmware apart (or asking the manufacturer).

Quite true, though wondering aloud...

How difficult would it be to pass the "lower-bound" LBA to SYNCHRONIZE CACHE, where "lower bound" is defined as the lowest sector in the range of sectors to be flushed?

That seems like a reasonable optimization -- it gives the drive an easy way to skip sync'ing sectors lower than the lower-bound LBA, if it is capable. Otherwise, a standards-compliant firmware will behave as you describe, and do what our code currently expects today -- a full cache flush.

This seems like a good way to speed up cache flush [on SCSI], while also perhaps experimenting with a more fine-grained way to pass down write barriers to the device.

Not a high priority thing overall, but OTOH, consider the case of placing your journal at the end of the disk. You could then issue a cache flush with a non-zero starting offset:

SYNCHRONIZE CACHE (max sectors - JOURNAL_SIZE, ~0)

That should be trivial even for dumb disk firmwares to optimize.

Jeff

#452 lore raw

Re: range-based cache flushing (was Re: Linux 2.6.29)

Quoted 24 lines · Jeff Garzik
Quoted 14 lines · James Bottomley
Quoted 12 lines · Ric Wheeler
Quoted 11 lines · Jeff Garzik

Ric Wheeler wrote:> And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE

Quoted 4 lines

is per device (not file system).

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

SCSI'S SYNCHRONIZE CACHE command already accepts an (LBA, length) pair. We could make use of that.

And I bet we could convince T13 to add FLUSH CACHE RANGE, if we could demonstrate clear benefit.

Quoted 8 lines
Quoted 2 lines

How well supported is this in SCSI? Can we try it out with a commodity SAS drive?

What do you mean by well supported? The way the SCSI standard is written, a device can do a complete cache flush when a range flush is requested and still be fully standards compliant. There's no easy way to tell if it does a complete cache flush every time other than by taking the firmware apart (or asking the manufacturer).

Quoted 5 lines

Quite true, though wondering aloud...

How difficult would it be to pass the "lower-bound" LBA to SYNCHRONIZE CACHE, where "lower bound" is defined as the lowest sector in the range of sectors to be flushed?

Actually, the implementation is designed to allow this. The standard says if the number of blocks is zero that means flush from the specified LBA to the end of the device. The sync cache we currently use has LBA 0 and number of blocks zero (which means flush everything).

Quoted 10 lines

That seems like a reasonable optimization -- it gives the drive an easy way to skip sync'ing sectors lower than the lower-bound LBA, if it is capable. Otherwise, a standards-compliant firmware will behave as you describe, and do what our code currently expects today -- a full cache flush.

This seems like a good way to speed up cache flush [on SCSI], while also perhaps experimenting with a more fine-grained way to pass down write barriers to the device.

Quoted 7 lines

Not a high priority thing overall, but OTOH, consider the case of placing your journal at the end of the disk. You could then issue a cache flush with a non-zero starting offset:

SYNCHRONIZE CACHE (max sectors - JOURNAL_SIZE, ~0)

That should be trivial even for dumb disk firmwares to optimize.

We could try it ... I'm still not sure how we'd tell the device is actually implementing it and not flushing the entire device.

James

#453 lore raw

Re: range-based cache flushing (was Re: Linux 2.6.29)

Quoted 15 lines · James Bottomley
Quoted 14 lines · Jeff Garzik
Quoted 13 lines · James Bottomley
Quoted 7 lines · Ric Wheeler
Quoted 5 lines · Jeff Garzik

Ric Wheeler wrote:> And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE

Quoted 2 lines

is per device (not file system).
When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

SCSI'S SYNCHRONIZE CACHE command already accepts an (LBA, length) pair. We could make use of that. And I bet we could convince T13 to add FLUSH CACHE RANGE, if we could demonstrate clear benefit.

How well supported is this in SCSI? Can we try it out with a commodity SAS drive?

What do you mean by well supported? The way the SCSI standard is written, a device can do a complete cache flush when a range flush is requested and still be fully standards compliant. There's no easy way to tell if it does a complete cache flush every time other than by taking the firmware apart (or asking the manufacturer).

Quoted 8 lines
Quoted 3 lines

Quite true, though wondering aloud...

How difficult would it be to pass the "lower-bound" LBA to SYNCHRONIZE CACHE, where "lower bound" is defined as the lowest sector in the range of sectors to be flushed?

Actually, the implementation is designed to allow this. The standard says if the number of blocks is zero that means flush from the specified LBA to the end of the device. The sync cache we currently use has LBA 0 and number of blocks zero (which means flush everything).

Yeah, that feature of the spec was what got me thinking.

"difficult" was referring more to the kernel side of things... if calculating the lowest LBA of a write barrier is difficult and/or CPU-consuming, the effort may not be worth it.

But if we could stick a

if (LBA < barrier-lower-bound)
barrier-lower-bound = LBA

somewhere, then pass that to SYNCHRONIZE CACHE, it could be a cheap way to increase sync-cache speed.

It seems extremely unlikely that sync-cache speed would _decrease_: for flush-everything firmwares, the sync-cache speed would remain unchanged.

Quoted 4 lines
Quoted 4 lines

That seems like a reasonable optimization -- it gives the drive an easy way to skip sync'ing sectors lower than the lower-bound LBA, if it is capable. Otherwise, a standards-compliant firmware will behave as you describe, and do what our code currently expects today -- a full cache flush.

This seems like a good way to speed up cache flush [on SCSI], while also perhaps experimenting with a more fine-grained way to pass down write barriers to the device.

Quoted 8 lines
Quoted 5 lines

Not a high priority thing overall, but OTOH, consider the case of placing your journal at the end of the disk. You could then issue a cache flush with a non-zero starting offset:

SYNCHRONIZE CACHE (max sectors - JOURNAL_SIZE, ~0)

That should be trivial even for dumb disk firmwares to optimize.

We could try it ... I'm still not sure how we'd tell the device is actually implementing it and not flushing the entire device.

Is that knowledge necessary?

Assuming the lower-bound is super-cheap to calculate, then the two most likely outcomes are: sync-cache speed remains the same, or sync-cache speed increases.

If the calculation of lower-bound is costly, I could see the need for that knowledge -- but if the cost is too high, the entire effort it likely to be scuttled, rather than worrying about detecting flush-everything firmwares.

Jeff

#454 lore raw

Re: range-based cache flushing (was Re: Linux 2.6.29)

Quoted 40 lines · Jeff Garzik
Quoted 22 lines · James Bottomley
Quoted 21 lines · Jeff Garzik
Quoted 11 lines · James Bottomley
Quoted 10 lines · Ric Wheeler
Quoted 9 lines · Jeff Garzik

Ric Wheeler wrote:> And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE

Quoted 3 lines

is per device (not file system).
When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

SCSI'S SYNCHRONIZE CACHE command already accepts an (LBA, length) pair. We could make use of that.
And I bet we could convince T13 to add FLUSH CACHE RANGE, if we could demonstrate clear benefit.

Quoted 7 lines
Quoted 2 lines

How well supported is this in SCSI? Can we try it out with a commodity SAS drive?

What do you mean by well supported? The way the SCSI standard is written, a device can do a complete cache flush when a range flush is requested and still be fully standards compliant. There's no easy way to tell if it does a complete cache flush every time other than by taking the firmware apart (or asking the manufacturer).

Quite true, though wondering aloud...

Quoted 8 lines
Quoted 3 lines

How difficult would it be to pass the "lower-bound" LBA to SYNCHRONIZE CACHE, where "lower bound" is defined as the lowest sector in the range of sectors to be flushed?

Actually, the implementation is designed to allow this. The standard says if the number of blocks is zero that means flush from the specified LBA to the end of the device. The sync cache we currently use has LBA 0 and number of blocks zero (which means flush everything).

Yeah, that feature of the spec was what got me thinking.

"difficult" was referring more to the kernel side of things... if calculating the lowest LBA of a write barrier is difficult and/or CPU-consuming, the effort may not be worth it.

But if we could stick a

Quoted 8 lines

if (LBA < barrier-lower-bound)
barrier-lower-bound = LBA

somewhere, then pass that to SYNCHRONIZE CACHE, it could be a cheap way to increase sync-cache speed.

It seems extremely unlikely that sync-cache speed would _decrease_: for flush-everything firmwares, the sync-cache speed would remain unchanged.

It's not impossible, though ... since the drive fw processor is probably pretty slow, but yes, it should hopefully be as fast or faster than full sync cache.

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines

That seems like a reasonable optimization -- it gives the drive an easy way to skip sync'ing sectors lower than the lower-bound LBA, if it is capable. Otherwise, a standards-compliant firmware will behave as you describe, and do what our code currently expects today -- a full cache flush.

This seems like a good way to speed up cache flush [on SCSI], while also perhaps experimenting with a more fine-grained way to pass down write barriers to the device.

Not a high priority thing overall, but OTOH, consider the case of placing your journal at the end of the disk. You could then issue a cache flush with a non-zero starting offset:

SYNCHRONIZE CACHE (max sectors - JOURNAL_SIZE, ~0)

That should be trivial even for dumb disk firmwares to optimize.

Quoted 8 lines
Quoted 2 lines

We could try it ... I'm still not sure how we'd tell the device is actually implementing it and not flushing the entire device.

Is that knowledge necessary?

Assuming the lower-bound is super-cheap to calculate, then the two most likely outcomes are: sync-cache speed remains the same, or sync-cache speed increases.

Yes, agreed ... we might as well tell the FW if it's cheap to know whether or not it acts on it.

Quoted 4 lines

If the calculation of lower-bound is costly, I could see the need for that knowledge -- but if the cost is too high, the entire effort it likely to be scuttled, rather than worrying about detecting flush-everything firmwares.

I really think, though, it's time to look again at how we implement barriers. Even properly implemented range flushing (if we can do it) is only decreasing the amount of overhead in a flush barrier.

If we could make the filesystems tolerant or at least aware that there might be very rare periods during operation when barriers get violated (during error processing or queue full handling) we could look again at implementing barriers via ordered tags.

James

#455 lore raw
In reply to Jeff Garzik
Quoted 4 lines · Jeff Garzik

Ric Wheeler wrote:> And, as I am sure that you do know, to add insult to injury, FLUSH_CACHE

Quoted 2 lines

is per device (not file system).

Quoted 8 lines
Quoted 2 lines

When you issue an fsync() on a disk with multiple partitions, you will flush the data for all of its partitions from the write cache....

SCSI'S SYNCHRONIZE CACHE command already accepts an (LBA, length) pair.
We could make use of that.

And I bet we could convince T13 to add FLUSH CACHE RANGE, if we could demonstrate clear benefit.

One more example of flexible, fine grain flush (though quite far out) are T10 OSDs with which you can flush a byte range of a single object (or collection, partition, or the whole device LUN)

Benny

Quoted 10 lines

Jeff

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#456 lore raw
In reply to Jens Axboe
Quoted 15 lines · Jens Axboe
Quoted 6 lines · Jeff Garzik

Stating "fsync already does that" borders on false, because that assumes
(a) the user has a fs that supports barriers
(b) the user is actually aware of a 'barriers' mount option and what it means
(c) the user has turned on an option normally defaulted to off.

Or in other words, it pretty much never happens.

That is true, except if you use xfs/ext4. And this discussion is fine, as was the one a few months back that got ext4 to enable barriers by default. If I had submitted patches to do that back in 2001/2 when the barrier stuff was written, I would have been shot for introducing such a slow down. After people found out that it just wasn't something silly, then you have a way to enable it.

Quoted 3 lines

I'd still wager that most people would rather have a 'good enough fsync' on their desktops than incur the penalty of barriers or write through caching. I know I do.

That's a strawman argument: The choice is not between "good enough fsync" and full use of barriers / write-through caching, at all.

It is clearly possible to implement an fsync(2) that causes FLUSH CACHE to be issued, without adding full barrier support to a filesystem. It is likely doable to avoid touching per-filesystem code at all, if we issue the flush from a generic fsync(2) code path in the kernel.

Thus, you have a "third way": fsync(2) gives the guarantee it is supposed to, but you do not take the full performance hit of barriers-all-the-time.

Remember, fsync(2) means that the user _expects_ a performance hit.

And they took the extra step to call fsync(2) because they want a guarantee, not a lie.

Jeff

#457 lore raw
Quoted 5 lines · Jeff Garzik

It is clearly possible to implement an fsync(2) that causes FLUSH CACHE to be issued, without adding full barrier support to a filesystem. It is likely doable to avoid touching per-filesystem code at all, if we issue the flush from a generic fsync(2) code path in the kernel.

We could easily do that. It would even work for most cases. The problematic ones are where filesystems do their own disk management, but I guess those people can do their own fsync() management too.

Somebody send me the patch, we can try it out.

Quoted 1 line

Remember, fsync(2) means that the user _expects_ a performance hit.

Within reason, though.

OS X, for example, doesn't do the disk barrier. It requires you to do a separate FULL_FSYNC (or something similar) ioctl to get that. Apparently exactly because users don't expect quite _that_ big of a performance hit.

(Or maybe just because it was easier to do that way. Never attribute to malice what can be sufficiently explained by stupidity).

Linus

#458 lore raw
Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Jeff Garzik

It is clearly possible to implement an fsync(2) that causes FLUSH CACHE to be issued, without adding full barrier support to a filesystem. It is likely doable to avoid touching per-filesystem code at all, if we issue the flush from a generic fsync(2) code path in the kernel.

We could easily do that. It would even work for most cases. The problematic ones are where filesystems do their own disk management, but I guess those people can do their own fsync() management too.

One concern with doing this above the file system is that you are not in the context of a transaction so you have no clean promises about what is on disk and persistent when. Flushing the cache is primitive at best, but the way barriers work today is designed to give the transactions some pretty critical ordering semantics for journalling file systems at least.

I don't see how you could use this approach to make a really robust, failure proof storage system, but it might appear to work most of the time for most people :-)

ric

Quoted 15 lines

Somebody send me the patch, we can try it out.

Quoted 2 lines

Remember, fsync(2) means that the user _expects_ a performance hit.

Within reason, though.

OS X, for example, doesn't do the disk barrier. It requires you to do a separate FULL_FSYNC (or something similar) ioctl to get that. Apparently exactly because users don't expect quite _that_ big of a performance hit.

(Or maybe just because it was easier to do that way. Never attribute to malice what can be sufficiently explained by stupidity).

Linus

#459 lore raw
Quoted 10 lines · Ric Wheeler

One concern with doing this above the file system is that you are not in the context of a transaction so you have no clean promises about what is on disk and persistent when. Flushing the cache is primitive at best, but the way barriers work today is designed to give the transactions some pretty critical ordering semantics for journalling file systems at least.

I don't see how you could use this approach to make a really robust, failure proof storage system, but it might appear to work most of the time for most people :-)

You just do a write barrier after doing all the filesystem writing, and you return with the guarantee that all the writes the filesystem did are actually on disk.

No gray areas. No questions. No "might appear to work".

Sure, there might be other writes that got flushed _too_, but nobody cares. If you have a crash later on, that's always true - you don't get crashes at nice well-defined points.

Linus

#460 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · Ric Wheeler

One concern with doing this above the file system is that you are not in the context of a transaction so you have no clean promises about what is on disk and persistent when. Flushing the cache is primitive at best, but the way barriers work today is designed to give the transactions some pretty critical ordering semantics for journalling file systems at least.

Quoted 8 lines
Quoted 4 lines

I don't see how you could use this approach to make a really robust, failure proof storage system, but it might appear to work most of the time for most people :-)

You just do a write barrier after doing all the filesystem writing, and you return with the guarantee that all the writes the filesystem did are actually on disk.

In this case, you have not gained anything - same number of barrier operations/cache flushes and looser semantics for the transactions?

Quoted 6 lines

No gray areas. No questions. No "might appear to work".

Sure, there might be other writes that got flushed _too_, but nobody cares. If you have a crash later on, that's always true - you don't get crashes at nice well-defined points.

Linus

This is pretty much how write barriers work today - you carry down other transactions (even for other partitions on the same disk) with you...

ric

#461 lore raw
Quoted 3 lines · Ric Wheeler

In this case, you have not gained anything - same number of barrier operations/cache flushes and looser semantics for the transactions?

Um. Except you gained the fact that the filesystem doesn't have to care and screw it up. And then we can know that it gets done, regardless of what odd things the low-level fs does.

Linus

#462 lore raw

PATCH 1/1 issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Jeff Garzik

It is clearly possible to implement an fsync(2) that causes FLUSH CACHE to be issued, without adding full barrier support to a filesystem. It is likely doable to avoid touching per-filesystem code at all, if we issue the flush from a generic fsync(2) code path in the kernel.

We could easily do that. It would even work for most cases. The problematic ones are where filesystems do their own disk management, but I guess those people can do their own fsync() management too.

Somebody send me the patch, we can try it out.

This is a simple step that would cover a lot of cases... sync(2) calls sync_blockdev(), and many filesystems do as well via the generic filesystem helper file_fsync (fs/sync.c).

XFS code calls sync_blockdev() a "big hammer", so I hope my patch follows with known practice.

Looking over every use of sync_blockdev(), its most frequent use is through fsync(2), for the selected filesystems that use the generic file_fsync helper.

Most callers of sync_blockdev() in the kernel do so infrequently, when removing and invalidating volumes (MD) or storing the superblock prior to release (put_super) in some filesystems.

Compile-tested only, of course :) But it should be work :)

My main concern is some hidden area that calls sync_blockdev() with a high-enough frequency that the performance hit is bad.

fs/buffer.c +8 -3
index 891e1c7..7b9f74a 100644
@@ -173,9 +173,14 @@ int sync_blockdev(struct block_device *bdev)
 {
 	int ret = 0;
 
-	if (bdev)
-		ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
-	return ret;
+	if (!bdev)
+		return 0;
+	
+	ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
+	if (ret)
+		return ret;
+	
+	return blkdev_issue_flush(bdev, NULL);
 }
 EXPORT_SYMBOL(sync_blockdev);
 
#463 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

Quoted 45 lines · Jeff Garzik
Quoted 10 lines · Linus Torvalds
Quoted 4 lines · Jeff Garzik

It is clearly possible to implement an fsync(2) that causes FLUSH CACHE to be issued, without adding full barrier support to a filesystem. It is likely doable to avoid touching per-filesystem code at all, if we issue the flush from a generic fsync(2) code path in the kernel.

We could easily do that. It would even work for most cases. The problematic ones are where filesystems do their own disk management, but I guess those people can do their own fsync() management too.

Somebody send me the patch, we can try it out.

This is a simple step that would cover a lot of cases... sync(2) calls sync_blockdev(), and many filesystems do as well via the generic filesystem helper file_fsync (fs/sync.c).

XFS code calls sync_blockdev() a "big hammer", so I hope my patch follows with known practice.

Looking over every use of sync_blockdev(), its most frequent use is through fsync(2), for the selected filesystems that use the generic file_fsync helper.

Most callers of sync_blockdev() in the kernel do so infrequently, when removing and invalidating volumes (MD) or storing the superblock prior to release (put_super) in some filesystems.

Compile-tested only, of course :) But it should be work :)

My main concern is some hidden area that calls sync_blockdev() with a high-enough frequency that the performance hit is bad.

fs/buffer.c +0 -3
index 891e1c7..7b9f74a 100644
@@ -173,9 +173,14 @@ int sync_blockdev(struct block_device *bdev)
 {
 	int ret = 0;
 
-	if (bdev)
-		ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
-	return ret;

fs/buffer.c int sync_blockdev(struct block_device *bdev) · line 185

Quoted 10 lines
fs/buffer.c +8 -0
+	if (!bdev)
+		return 0;
+	
+	ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
+	if (ret)
+		return ret;
+	
+	return blkdev_issue_flush(bdev, NULL);
 }
 EXPORT_SYMBOL(sync_blockdev);

What about when you're running over a big raid device with battery-backed cache, and you trust the cache as much as much as the disks. Wouldn't this unconditional cache flush be painful there on any of the callers even if they're rare? (fs unmounts, freezes, unmounts, etc? Or a fat filesystem on that device doing an fsync?)

xfs, reiserfs, ext4 all avoid the blkdev flush on fsync if barriers are not enabled, I think for that reason...

(I'm assuming these raid devices still honor a cache flush request even if they're battery-backed? I dunno.)

-Eric

#464 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

Quoted 51 lines · Eric Sandeen
Quoted 50 lines · Jeff Garzik
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Jeff Garzik

It is clearly possible to implement an fsync(2) that causes FLUSH CACHE to be issued, without adding full barrier support to a filesystem. It is likely doable to avoid touching per-filesystem code at all, if we issue the flush from a generic fsync(2) code path in the kernel.

We could easily do that. It would even work for most cases. The problematic ones are where filesystems do their own disk management, but I guess those people can do their own fsync() management too.

Somebody send me the patch, we can try it out.

This is a simple step that would cover a lot of cases... sync(2) calls sync_blockdev(), and many filesystems do as well via the generic filesystem helper file_fsync (fs/sync.c).

XFS code calls sync_blockdev() a "big hammer", so I hope my patch follows with known practice.

Looking over every use of sync_blockdev(), its most frequent use is through fsync(2), for the selected filesystems that use the generic file_fsync helper.

Most callers of sync_blockdev() in the kernel do so infrequently, when removing and invalidating volumes (MD) or storing the superblock prior to release (put_super) in some filesystems.

Compile-tested only, of course :) But it should be work :)

My main concern is some hidden area that calls sync_blockdev() with a high-enough frequency that the performance hit is bad.

fs/buffer.c +8 -3
index 891e1c7..7b9f74a 100644
@@ -173,9 +173,14 @@ int sync_blockdev(struct block_device *bdev)
 {
 	int ret = 0;
 
-	if (bdev)
-		ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
-	return ret;
+	if (!bdev)
+		return 0;
+	
+	ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
+	if (ret)
+		return ret;
+	
+	return blkdev_issue_flush(bdev, NULL);
Quoted 8 lines

fs/buffer.c int sync_blockdev(struct block_device *bdev) · line 185

Quoted 2 lines
fs/buffer.c +0 -0
 }
 EXPORT_SYMBOL(sync_blockdev);

What about when you're running over a big raid device with battery-backed cache, and you trust the cache as much as much as the disks. Wouldn't this unconditional cache flush be painful there on any of the callers even if they're rare? (fs unmounts, freezes, unmounts, etc? Or a fat filesystem on that device doing an fsync?)

What exactly do you think sync_blockdev() does? :)

It is used right before a volume goes away. If that's not a time to flush the cache, I dunno what is.

The _whole purpose_ of sync_blockdev() is to push out the data to permanent storage. Look at the users -- unmount volume, journal close, etc. Things that are OK to occur after those points include: power off, device unplug, etc.

A secondary purpose of sync_blockdev() is as a hack, for simple/ancient bdev-based filesystems that do not wish to bother with barriers and all the associated complexity to tracking what writes do/do not need flushing.

Quoted 2 lines

xfs, reiserfs, ext4 all avoid the blkdev flush on fsync if barriers are not enabled, I think for that reason...

Enabling barriers causes slowdowns far greater than that of simply causing fsync(2) to trigger FLUSH CACHE, because barriers imply FLUSH CACHE issuance for all in-kernel filesystem journalled/atomic transactions, in addition to whatever syscalls userspace is issuing.

The number of FLUSH CACHES w/ barriers is orders of magnitude larger than the number of fsync/fdatasync calls.

Jeff

#465 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

Quoted 9 lines · Jeff Garzik
Quoted 5 lines · Eric Sandeen

What about when you're running over a big raid device with battery-backed cache, and you trust the cache as much as much as the disks. Wouldn't this unconditional cache flush be painful there on any of the callers even if they're rare? (fs unmounts, freezes, unmounts, etc? Or a fat filesystem on that device doing an fsync?)

What exactly do you think sync_blockdev() does? :)

It used to push os cached data to the storage. Now it tells the storage to flush cache too (with your patch). This seems fine in general, although it's not a panacea for all the various data integrity issues that are being tossed about in this thread. :)

Quoted 7 lines

It is used right before a volume goes away. If that's not a time to flush the cache, I dunno what is.

The _whole purpose_ of sync_blockdev() is to push out the data to permanent storage. Look at the users -- unmount volume, journal close, etc. Things that are OK to occur after those points include: power off, device unplug, etc.

Sure. But I was thinking about enterprise raids with battery backup which may last for days.

But, ok, I wasn't thinking quite right about the unmount situations etc; even on enterprise raids like this, flushing things out on unmount makes sense in the case where you lose power post-unmount and can't restore power before the battery backup dies.

I also wondered if a cache flush on one lun issues a cache flush for the entire controller, or just for that lun. Hopefully the latter, in which case it's not as big a deal.

Quoted 3 lines

A secondary purpose of sync_blockdev() is as a hack, for simple/ancient bdev-based filesystems that do not wish to bother with barriers and all the associated complexity to tracking what writes do/do not need flushing.

Yep, I get that and it seems reasonable except ....

Quoted 10 lines
Quoted 2 lines

xfs, reiserfs, ext4 all avoid the blkdev flush on fsync if barriers are not enabled, I think for that reason...

Enabling barriers causes slowdowns far greater than that of simply causing fsync(2) to trigger FLUSH CACHE, because barriers imply FLUSH CACHE issuance for all in-kernel filesystem journalled/atomic transactions, in addition to whatever syscalls userspace is issuing.

The number of FLUSH CACHES w/ barriers is orders of magnitude larger than the number of fsync/fdatasync calls.

I understand all that.

My point is that the above filesystems (xfs, reiserfs, ext4) skip the blkdev flush on fsync when barriers are explicitly disabled.

They do this because if an admin disables barriers, they are trusting that the write cache is nonvolatile and will be able to destage fully even if external power is lost for some time.

In that case you don't need a blkdev_issue_flush on fsync either (or are at least willing to live with the diminished risk, thanks to the battery backup), and on xfs, ext4 etc you can turn it off (it goes away w/ the barriers off setting). With this change to the simple generic fsync path, you can't turn it off for those filesystems that use it for fsync.

But I suppose it's rare that anybody ever uses a filesystem which uses this generic sync method on any sort of interesting storage like I'm talking about, and it's not a big deal... (or maybe that interesting storage just ignores cache flushes anyway, I dunno).

My main concerns were that these extra cache flushes for fsync aren't tunable, and that flushes on one lun might affect other luns. I guess I've talked myself out of those concerns in a couple different ways now. ;)

-Eric

#466 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

Quoted 30 lines · Eric Sandeen
Quoted 4 lines · Jeff Garzik

Eric Sandeen wrote:

Quoted 8 lines
Quoted 6 lines

What about when you're running over a big raid device with battery-backed cache, and you trust the cache as much as much as the disks. Wouldn't this unconditional cache flush be painful there on any of the callers even if they're rare? (fs unmounts, freezes, unmounts, etc? Or a fat filesystem on that device doing an fsync?)

What exactly do you think sync_blockdev() does? :)

It used to push os cached data to the storage. Now it tells the storage to flush cache too (with your patch). This seems fine in general, although it's not a panacea for all the various data integrity issues that are being tossed about in this thread. :)

Quoted 8 lines

It is used right before a volume goes away. If that's not a time to flush the cache, I dunno what is.

The _whole purpose_ of sync_blockdev() is to push out the data to permanent storage. Look at the users -- unmount volume, journal close, etc. Things that are OK to occur after those points include: power off, device unplug, etc.

Quoted 8 lines

Sure. But I was thinking about enterprise raids with battery backup which may last for days.

But, ok, I wasn't thinking quite right about the unmount situations etc; even on enterprise raids like this, flushing things out on unmount makes sense in the case where you lose power post-unmount and can't restore power before the battery backup dies.

Enterprise raid systems don't have this issue. They all have sufficient battery power to safely destage the volatile cache to persistent storage on power outage (i.e., they keep enough drives spinning and so on to empty the cache).

Hardware RAID cards that you have in some servers (not external RAID arrays) would have this issue though and are probably fairly common.

Quoted 4 lines

I also wondered if a cache flush on one lun issues a cache flush for the entire controller, or just for that lun. Hopefully the latter, in which case it's not as big a deal.

Probably just for that LUN - LUN's are usually independent in almost all ways. Firmware of course could do anything, I would assume that most high end arrays would probably ignore a flush command in any case.

Quoted 41 lines
Quoted 5 lines

A secondary purpose of sync_blockdev() is as a hack, for simple/ancient bdev-based filesystems that do not wish to bother with barriers and all the associated complexity to tracking what writes do/do not need flushing.

Yep, I get that and it seems reasonable except ....

Quoted 11 lines
Quoted 3 lines

xfs, reiserfs, ext4 all avoid the blkdev flush on fsync if barriers are not enabled, I think for that reason...

Enabling barriers causes slowdowns far greater than that of simply causing fsync(2) to trigger FLUSH CACHE, because barriers imply FLUSH CACHE issuance for all in-kernel filesystem journalled/atomic transactions, in addition to whatever syscalls userspace is issuing.

The number of FLUSH CACHES w/ barriers is orders of magnitude larger than the number of fsync/fdatasync calls.

I understand all that.

My point is that the above filesystems (xfs, reiserfs, ext4) skip the blkdev flush on fsync when barriers are explicitly disabled.

They do this because if an admin disables barriers, they are trusting that the write cache is nonvolatile and will be able to destage fully even if external power is lost for some time.

In that case you don't need a blkdev_issue_flush on fsync either (or are at least willing to live with the diminished risk, thanks to the battery backup), and on xfs, ext4 etc you can turn it off (it goes away w/ the barriers off setting). With this change to the simple generic fsync path, you can't turn it off for those filesystems that use it for fsync.

But I suppose it's rare that anybody ever uses a filesystem which uses this generic sync method on any sort of interesting storage like I'm talking about, and it's not a big deal... (or maybe that interesting storage just ignores cache flushes anyway, I dunno).

Quoted 6 lines

My main concerns were that these extra cache flushes for fsync aren't tunable, and that flushes on one lun might affect other luns. I guess I've talked myself out of those concerns in a couple different ways now. ;)

-Eric

I do enthusiastically agree that we should not be doing barriers and the blkdev flush for file systems that do barriers correctly.

ric

#467 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

In reply to Eric Sandeen
Quoted 66 lines · Eric Sandeen
Quoted 47 lines · Jeff Garzik
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Jeff Garzik

It is clearly possible to implement an fsync(2) that causes FLUSH CACHE to be issued, without adding full barrier support to a filesystem. It is likely doable to avoid touching per-filesystem code at all, if we issue the flush from a generic fsync(2) code path in the kernel.

We could easily do that. It would even work for most cases. The problematic ones are where filesystems do their own disk management, but I guess those people can do their own fsync() management too.

Somebody send me the patch, we can try it out.

This is a simple step that would cover a lot of cases... sync(2) calls sync_blockdev(), and many filesystems do as well via the generic filesystem helper file_fsync (fs/sync.c).

XFS code calls sync_blockdev() a "big hammer", so I hope my patch follows with known practice.

Looking over every use of sync_blockdev(), its most frequent use is through fsync(2), for the selected filesystems that use the generic file_fsync helper.

Most callers of sync_blockdev() in the kernel do so infrequently, when removing and invalidating volumes (MD) or storing the superblock prior to release (put_super) in some filesystems.

Compile-tested only, of course :) But it should be work :)

My main concern is some hidden area that calls sync_blockdev() with a high-enough frequency that the performance hit is bad.

fs/buffer.c +1 -3
index 891e1c7..7b9f74a 100644
@@ -173,9 +173,14 @@ int sync_blockdev(struct block_device *bdev)
 {
 	int ret = 0;
 
-	if (bdev)
-		ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
-	return ret;
+	if (!bdev)

fs/buffer.c int sync_blockdev(struct block_device *bdev) · line 186

Quoted 10 lines
fs/buffer.c +7 -0
+		return 0;
+	
+	ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
+	if (ret)
+		return ret;
+	
+	return blkdev_issue_flush(bdev, NULL);
 }
 EXPORT_SYMBOL(sync_blockdev);
 

What about when you're running over a big raid device with battery-backed cache, and you trust the cache as much as much as the disks. Wouldn't this unconditional cache flush be painful there on any of the callers even if they're rare? (fs unmounts, freezes, unmounts, etc? Or a fat filesystem on that device doing an fsync?)

Quoted 8 lines

xfs, reiserfs, ext4 all avoid the blkdev flush on fsync if barriers are not enabled, I think for that reason...

(I'm assuming these raid devices still honor a cache flush request even if they're battery-backed? I dunno.)

-Eric

I think that Jeff's patch misses the whole need to protect transactions, including meta data, in a precise way. Useful for thing like unmount, not to give us strong protection for transactions or for fsync().

This patch will be adding overhead here - you will still need flushing at the transaction commit layer of the specific file systems to get any reliable transactions.

Having looked at the timing of barrier flushes on slow s-ata drives with an analyser a few years back, the first one is expensive (as you would expect with a large drive cache of 16 or 32 MB) and the second was nearly free.

Moving the expensive flush to this layer guts the transaction building blocks and costs the same....

ric

#468 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

Quoted 70 lines · Ric Wheeler
Quoted 69 lines · Eric Sandeen
Quoted 47 lines · Jeff Garzik
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Jeff Garzik

It is clearly possible to implement an fsync(2) that causes FLUSH CACHE to be issued, without adding full barrier support to a filesystem. It is likely doable to avoid touching per-filesystem code at all, if we issue the flush from a generic fsync(2) code path in the kernel.

We could easily do that. It would even work for most cases. The problematic ones are where filesystems do their own disk management, but I guess those people can do their own fsync() management too.

Somebody send me the patch, we can try it out.

This is a simple step that would cover a lot of cases... sync(2) calls sync_blockdev(), and many filesystems do as well via the generic filesystem helper file_fsync (fs/sync.c).

XFS code calls sync_blockdev() a "big hammer", so I hope my patch follows with known practice.

Looking over every use of sync_blockdev(), its most frequent use is through fsync(2), for the selected filesystems that use the generic file_fsync helper.

Most callers of sync_blockdev() in the kernel do so infrequently, when removing and invalidating volumes (MD) or storing the superblock prior to release (put_super) in some filesystems.

Compile-tested only, of course :) But it should be work :)

My main concern is some hidden area that calls sync_blockdev() with a high-enough frequency that the performance hit is bad.

fs/buffer.c +1 -3
index 891e1c7..7b9f74a 100644
@@ -173,9 +173,14 @@ int sync_blockdev(struct block_device *bdev)
 {
     int ret = 0;
 
-    if (bdev)
-        ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
-    return ret;
+    if (!bdev)

fs/buffer.c int sync_blockdev(struct block_device *bdev) · line 186

Quoted 10 lines
fs/buffer.c +7 -0
+        return 0;
+   
+    ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
+    if (ret)
+        return ret;
+   
+    return blkdev_issue_flush(bdev, NULL);
 }
 EXPORT_SYMBOL(sync_blockdev);
 

What about when you're running over a big raid device with battery-backed cache, and you trust the cache as much as much as the disks. Wouldn't this unconditional cache flush be painful there on any of the callers even if they're rare? (fs unmounts, freezes, unmounts, etc? Or a fat filesystem on that device doing an fsync?)

xfs, reiserfs, ext4 all avoid the blkdev flush on fsync if barriers are not enabled, I think for that reason...

Quoted 6 lines
Quoted 5 lines

(I'm assuming these raid devices still honor a cache flush request even if they're battery-backed? I dunno.)

-Eric

I think that Jeff's patch misses the whole need to protect transactions, including meta data, in a precise way. Useful for thing like unmount, not to give us strong protection for transactions or for fsync().

What do you think sync_blockdev() does? What is its purpose?

Twofold:
(1) guarantee all user data is flushed out before a major event (unmount, journal close, unplug, poweroff, explosion, ...)

(2) As a sledgehammer hack for simple or legacy filesystems that do not wish or need the complexity of transactional protection. sync_blockdev() is intentionally used in lieu of complexity for the following filesystems: HFS, HFS+, ADFS, AFFS, FAT, bfs, UFS, NTFS, qnx4.

My patch adds needed guarantees, only for the above filesystems, where none were present before.

Quoted 1 line

This patch will be adding overhead here - you will still need flushing at the transaction commit layer of the specific file systems to get any reliable transactions.

sync_blockdev() is used as fsync(2) only in simple or legacy filesystems that do not want a transaction commit layer!

Read the patch :)

Jeff

#469 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

Quoted 9 lines · Jeff Garzik

Ric Wheeler wrote:
What do you think sync_blockdev() does? What is its purpose?

Twofold:
(1) guarantee all user data is flushed out before a major event (unmount, journal close, unplug, poweroff, explosion, ...)

(2) As a sledgehammer hack for simple or legacy filesystems that do not wish or need the complexity of transactional protection. sync_blockdev() is intentionally used in lieu of complexity for the following filesystems: HFS, HFS+, ADFS, AFFS, FAT, bfs, UFS, NTFS, qnx4.

My patch adds needed guarantees, only for the above filesystems, where none were present before.

To be specific, I was referring to fsync(2) guarantees being added to HFS, HFS+, ADFS, AFFS, FAT, bfs, UFS, NTFS, and qnx4.

Other filesystems, besides those in the list, gain the flush-on-unmount action (a rare but useful addition) with my patch.

Jeff

#470 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

Quoted 9 lines · Jeff Garzik
Quoted 7 lines · Jeff Garzik

Ric Wheeler wrote:
What do you think sync_blockdev() does? What is its purpose?

Twofold:
(1) guarantee all user data is flushed out before a major event (unmount, journal close, unplug, poweroff, explosion, ...)

(2) As a sledgehammer hack for simple or legacy filesystems that do not wish or need the complexity of transactional protection. sync_blockdev() is intentionally used in lieu of complexity for the following filesystems: HFS, HFS+, ADFS, AFFS, FAT, bfs, UFS, NTFS, qnx4.

Quoted 10 lines
Quoted 1 line

My patch adds needed guarantees, only for the above filesystems, where none were present before.

To be specific, I was referring to fsync(2) guarantees being added to HFS, HFS+, ADFS, AFFS, FAT, bfs, UFS, NTFS, and qnx4.

Other filesystems, besides those in the list, gain the flush-on-unmount action (a rare but useful addition) with my patch.

Jeff

Sorry for misunderstanding the scope of this before - this is certainly a net win for the file systems that don't have proper barrier support baked in already. Thanks!

Ric

#471 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

In reply to Jeff Garzik
Quoted 1 line · Jeff Garzik

What do you think sync_blockdev() does? What is its purpose?

It writes out data in the block device inode. Which does not include any user data, and might not contain anything at all for filesystems that have their own address space for metadata. It's defintively the wrong place for this kind of hack.

#472 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

In reply to Eric Sandeen
Quoted 63 lines · Eric Sandeen
Quoted 45 lines · Jeff Garzik
Quoted 10 lines · Linus Torvalds
Quoted 4 lines · Jeff Garzik

It is clearly possible to implement an fsync(2) that causes FLUSH CACHE to be issued, without adding full barrier support to a filesystem. It is likely doable to avoid touching per-filesystem code at all, if we issue the flush from a generic fsync(2) code path in the kernel.

We could easily do that. It would even work for most cases. The problematic ones are where filesystems do their own disk management, but I guess those people can do their own fsync() management too.

Somebody send me the patch, we can try it out.

This is a simple step that would cover a lot of cases... sync(2) calls sync_blockdev(), and many filesystems do as well via the generic filesystem helper file_fsync (fs/sync.c).

XFS code calls sync_blockdev() a "big hammer", so I hope my patch follows with known practice.

Looking over every use of sync_blockdev(), its most frequent use is through fsync(2), for the selected filesystems that use the generic file_fsync helper.

Most callers of sync_blockdev() in the kernel do so infrequently, when removing and invalidating volumes (MD) or storing the superblock prior to release (put_super) in some filesystems.

Compile-tested only, of course :) But it should be work :)

My main concern is some hidden area that calls sync_blockdev() with a high-enough frequency that the performance hit is bad.

fs/buffer.c +0 -3
index 891e1c7..7b9f74a 100644
@@ -173,9 +173,14 @@ int sync_blockdev(struct block_device *bdev)
 {
 	int ret = 0;
 
-	if (bdev)
-		ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
-	return ret;

fs/buffer.c int sync_blockdev(struct block_device *bdev) · line 185

Quoted 10 lines
fs/buffer.c +8 -0
+	if (!bdev)
+		return 0;
+	
+	ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
+	if (ret)
+		return ret;
+	
+	return blkdev_issue_flush(bdev, NULL);
 }
 EXPORT_SYMBOL(sync_blockdev);

What about when you're running over a big raid device with battery-backed cache, and you trust the cache as much as much as the disks. Wouldn't this unconditional cache flush be painful there on any of the callers even if they're rare? (fs unmounts, freezes, unmounts, etc? Or a fat filesystem on that device doing an fsync?)

Quoted 5 lines

xfs, reiserfs, ext4 all avoid the blkdev flush on fsync if barriers are not enabled, I think for that reason...

(I'm assuming these raid devices still honor a cache flush request even if they're battery-backed? I dunno.)

I think most don't, as they realize it's a data integrity thing and that doesn't apply if you don't lose data on powerloss. But, I'm sure there are also "dumb" ones that DO flush the cache. In which case the flush is utterly hopeless and should not be done.

Signature · 1 line
Jens Axboe
#473 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

In reply to Jeff Garzik
Quoted 3 lines

This is a simple step that would cover a lot of cases... sync(2) calls sync_blockdev(), and many filesystems do as well via the generic filesystem helper file_fsync (fs/sync.c).

file_fsync probably needs to pass down more information so you can make this a mount option. It's going to depend on the application whether the flush is good bad or indifferent.

#474 lore raw

Re: [PATCH] issue storage device flush via sync_blockdev() (was Re: Linux 2.6.29)

Quoted 7 lines · Alan Cox
Quoted 3 lines

This is a simple step that would cover a lot of cases... sync(2) calls sync_blockdev(), and many filesystems do as well via the generic filesystem helper file_fsync (fs/sync.c).

file_fsync probably needs to pass down more information so you can make this a mount option. It's going to depend on the application whether the flush is good bad or indifferent.

file_fsync is only used by ancient legacy filesystems, who specifically don't want to bother with anything more complicated: HFS, HFS+, ADFS, AFFS, FAT, bfs, UFS, NTFS, qnx4.

IOW they _already_ consciously implement fsync(2) as "flush ENTIRE blockdev".

I think it is worth it to simply wait and see if mount options are even wanted.

Jeff

#475 lore raw

PATCH v2 1/1 issue storage device flush via sync_blockdev()

In reply to Jeff Garzik

Flush storage dev writeback cache, for each call to sync_blockdev().

sync_blockdev() is used primarily for two purposes:

1) To flush all data to permanent storage prior to a major event, such as: unmount, journal close, unplug, poweroff, explosion, ...

2) As a "sledgehammer hack" to provide fsync(2) via file_fsync to filesystems, generally simple or legacy filesystems, such as HFS, HFS+, ADFS, AFFS, bfs, UFS, NTFS, qnx4 and FAT.

This change guarantees that the underlying storage device will have flushed any pending data in its writeback cache to permanent media, before it returns (...if underlying storage dev supports flushes).

---
Changes since last patch:
- do not return error, if storage dev does not support flushes at all (-EOPNOTSUPP)

fs/buffer.c +11 -2
index 891e1c7..e04d7a4 100644
@@ -173,8 +173,17 @@ int sync_blockdev(struct block_device *bdev)
 {
 	int ret = 0;
 
-	if (bdev)
-		ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
+	if (!bdev)
+		return 0;
+	
+	ret = filemap_write_and_wait(bdev->bd_inode->i_mapping);
+	if (ret)
+		return ret;
+	
+	ret = blkdev_issue_flush(bdev, NULL);
+	if (ret == -EOPNOTSUPP)
+		ret = 0;
+	
 	return ret;
 }
 EXPORT_SYMBOL(sync_blockdev);
maintainer
#476 lore raw

Re: [PATCH v2] issue storage device flush via sync_blockdev()

Jeff,

FYI, I tried your patch; it causes the lvm process called out of the initramfs from an Ubuntu 8.10 system to blow up while trying to set up the root filesystem. The stack trace was:

generic_make_request+0x2a3/0x2e6
trace_hardirqs_on_caller+0x111/0x135
mempool_alloc_slab+0xe/0x10
mempool_alloc+0x42/0xe0
submit_bio+0xad/0xb5
bio_alloc_bioset+0x21/0xfc
blkdev_issue_flush+0x7f/0xfc
syn_blockdev+0x2a/0x36
__blkdev_put_0x44/0x131
blkdev_put+0xa/0xc
blkdev_close+0x2e/0x32
__fput+0xcf/0x15f
fput+0x19/0x1b
filp_close+0x51/0x5b
sys_close+0x73/0xad

- Ted

#477 lore raw

Re: [PATCH v2] issue storage device flush via sync_blockdev()

Quoted 6 lines · Theodore Tso

Jeff,

FYI, I tried your patch; it causes the lvm process called out of the initramfs from an Ubuntu 8.10 system to blow up while trying to set up the root filesystem. The stack trace was:

Quoted 15 lines
generic_make_request+0x2a3/0x2e6
trace_hardirqs_on_caller+0x111/0x135
mempool_alloc_slab+0xe/0x10
mempool_alloc+0x42/0xe0
submit_bio+0xad/0xb5
bio_alloc_bioset+0x21/0xfc
blkdev_issue_flush+0x7f/0xfc
syn_blockdev+0x2a/0x36
__blkdev_put_0x44/0x131
blkdev_put+0xa/0xc
blkdev_close+0x2e/0x32
__fput+0xcf/0x15f
fput+0x19/0x1b
filp_close+0x51/0x5b
sys_close+0x73/0xad

hmmm, I wonder if DM/LVM doesn't like blkdev_issue_flush, or it's too early, or what. I'll toss Ubuntu onto a VM and check it out...

Jeff

maintainer
#478 lore raw

Re: [PATCH v2] issue storage device flush via sync_blockdev()

Quoted 23 lines · Jeff Garzik
Quoted 6 lines · Theodore Tso

Jeff,

FYI, I tried your patch; it causes the lvm process called out of the initramfs from an Ubuntu 8.10 system to blow up while trying to set up the root filesystem. The stack trace was:

Quoted 15 lines
generic_make_request+0x2a3/0x2e6
trace_hardirqs_on_caller+0x111/0x135
mempool_alloc_slab+0xe/0x10
mempool_alloc+0x42/0xe0
submit_bio+0xad/0xb5
bio_alloc_bioset+0x21/0xfc
blkdev_issue_flush+0x7f/0xfc
syn_blockdev+0x2a/0x36
__blkdev_put_0x44/0x131
blkdev_put+0xa/0xc
blkdev_close+0x2e/0x32
__fput+0xcf/0x15f
fput+0x19/0x1b
filp_close+0x51/0x5b
sys_close+0x73/0xad
Quoted 2 lines

hmmm, I wonder if DM/LVM doesn't like blkdev_issue_flush, or it's too early, or what. I'll toss Ubuntu onto a VM and check it out...

I forgot to mention. The failure was the EIP was NULL; so it looks like we called a null function pointer. The only function pointer derference I can find is q->make_request_fn() in line 1460 of blk-core.c, in __generic_make_request. But that doesn't seem make any sense....

Anyway, maybe you can figure out what's going on. The problem disappeared as soon as I popped off this patch, though, so it was pretty clearly the culprit.

- Ted

#479 lore raw

PATCH 1/1 issue storage dev flush from generic file_fsync helper

In reply to Jeff Garzik

Simple and legacy blkdev-based filesystems such HFS, HFS+, ADFS, AFFS, FAT, bfs, UFS, NTFS, and qnx4 all use file_fsync as their fsync(2) VFS helper implementation.

Add a storage dev cache flush, to actually provide the guarantees that are promised with fsync(2).

---

Out of 18 other places that call sync_blockdev(), only 3-4 are in filesystems that arguably do not need or want a blkdev flush. This patch below clearly only addresses 1 out of ~15 callsites that really do want metadata, data, and everything in between flushed to disk at the sync_blockdev() callsite.

It should be noted that other calls are NOT used in fsync(2), but rather than with guaranteed written data prior to major events such as unmount, journal close, MD consistency check, etc.

fs/sync.c +8 -0
index a16d53e..24bb2f4 100644
@@ -5,6 +5,7 @@
 #include <linux/kernel.h>
 #include <linux/file.h>
 #include <linux/fs.h>
+#include <linux/blkdev.h>
 #include <linux/module.h>
 #include <linux/sched.h>
 #include <linux/writeback.h>
@@ -72,6 +73,13 @@ int file_fsync(struct file *filp, struct dentry *dentry, int datasync)
 	err = sync_blockdev(sb->s_bdev);
 	if (!ret)
 		ret = err;
+
+	err = blkdev_issue_flush(sb->s_bdev, NULL);
+	if (err == -EOPNOTSUPP)
+		err = 0;
+	if (!ret)
+		ret = err;
+
 	return ret;
 }
 
#480 lore raw

Re: [PATCH] issue storage dev flush from generic file_fsync helper

Looks good except that we still need a tuning know for it. Preferably one that works for these filesystems and all the existing barrier using ones.

#481 lore raw

Re: [PATCH] issue storage dev flush from generic file_fsync helper

Quoted 3 lines · Christoph Hellwig

Looks good except that we still need a tuning know for it. Preferably one that works for these filesystems and all the existing barrier using ones.

Christoph, I reworked my previous fsync() patches so that what was a mount option to trigger a storage device writeback cache flush becomes a sysfs knob.

I still need to add support for automatic detection of underlying device's flushing capabilities but first I would like to know if you agree with the general approach.

I'll be replying to this email with the new patches.

- Fernando

#482 lore raw

PATCH 1/5 block: Add block_flush_device()

This patch adds a helper function that should be used by filesystems that need to flush the underlying block device on fsync()/fdatasync().

---

fs/buffer.c +11 -0
diff -urNp linux-2.6.29-orig/fs/buffer.c linux-2.6.29/fs/buffer.c
@@ -165,6 +165,17 @@ void end_buffer_write_sync(struct buffer
 	put_bh(bh);
 }
 
+/* Issue flush of write caches on the block device */
+int block_flush_device(struct super_block *sb)
+{
+	int ret = 0;
+
+	ret = blkdev_issue_flush(sb->s_bdev, NULL);
+
+	return (ret == -EOPNOTSUPP) ? 0 : ret;
+}
+EXPORT_SYMBOL(block_flush_device);
+
 /*
  * Write out and wait upon all the dirty data associated with a block
  * device via its mapping.  Does not take the superblock lock.
include/linux/buffer_head.h +1 -0
diff -urNp linux-2.6.29-orig/include/linux/buffer_head.h linux-2.6.29/include/linux/buffer_head.h
@@ -238,6 +238,7 @@ int nobh_write_end(struct file *, struct
 int nobh_truncate_page(struct address_space *, loff_t, get_block_t *);
 int nobh_writepage(struct page *page, get_block_t *get_block,
                         struct writeback_control *wbc);
+int block_flush_device(struct super_block *sb);
 
 void buffer_init(void);
 
#483 lore raw

PATCH 2/5 ext3: call blkdev_issue_flush on fsync()

In reply to Fernando Luis Vázquez Cao

To ensure that bits are truly on-disk after an fsync or fdatasync, we should force a disk flush explicitly when there is dirty data/metadata and the journal didn't emit a write barrier (either because metadata is not being synched or barriers are disabled).

---

fs/ext3/fsync.c +14 -5
diff -urNp linux-2.6.29-orig/fs/ext3/fsync.c linux-2.6.29/fs/ext3/fsync.c
@@ -45,6 +45,8 @@
 int ext3_sync_file(struct file * file, struct dentry *dentry, int datasync)
 {
 	struct inode *inode = dentry->d_inode;
+	journal_t *journal = EXT3_SB(inode->i_sb)->s_journal;
+	unsigned long i_state = inode->i_state;
 	int ret = 0;
 
 	J_ASSERT(ext3_journal_current_handle() == NULL);
@@ -69,23 +71,30 @@ int ext3_sync_file(struct file * file, s
 	 */
 	if (ext3_should_journal_data(inode)) {
 		ret = ext3_force_commit(inode->i_sb);
-		goto out;
+		if (!(journal->j_flags & JFS_BARRIER))
+			block_flush_device(inode->i_sb);
+		return ret;
 	}
 
-	if (datasync && !(inode->i_state & I_DIRTY_DATASYNC))
-		goto out;
+	if (datasync && !(i_state & I_DIRTY_DATASYNC)) {
+		if (i_state & I_DIRTY_PAGES)
+			block_flush_device(inode->i_sb);
+		return ret;
+	}
 
 	/*
 	 * The VFS has written the file data.  If the inode is unaltered
 	 * then we need not start a commit.
 	 */
-	if (inode->i_state & (I_DIRTY_SYNC|I_DIRTY_DATASYNC)) {
+	if (i_state & (I_DIRTY_SYNC|I_DIRTY_DATASYNC)) {
 		struct writeback_control wbc = {
 			.sync_mode = WB_SYNC_ALL,
 			.nr_to_write = 0, /* sys_fsync did this */
 		};
 		ret = sync_inode(inode, &wbc);
+		if (journal && !(journal->j_flags & JFS_BARRIER))
+			block_flush_device(inode->i_sb);
 	}
-out:
+
 	return ret;
 }
#484 lore raw

Re: [PATCH 2/5] ext3: call blkdev_issue_flush on fsync()

Quoted 6 lines · Fernando Luis Vázquez Cao

To ensure that bits are truly on-disk after an fsync or fdatasync, we should force a disk flush explicitly when there is dirty data/metadata and the journal didn't emit a write barrier (either because metadata is not being synched or barriers are disabled).

Your patches do not seem to propagate the issue-flush error code, even when it is easily available.

Jeff

#485 lore raw

Re: [PATCH 2/5] ext3: call blkdev_issue_flush on fsync()

Quoted 9 lines · Jeff Garzik
Quoted 6 lines · Fernando Luis Vázquez Cao

To ensure that bits are truly on-disk after an fsync or fdatasync, we should force a disk flush explicitly when there is dirty data/metadata and the journal didn't emit a write barrier (either because metadata is not being synched or barriers are disabled).

Your patches do not seem to propagate the issue-flush error code, even when it is easily available.

Oops... you are right. I will fix that.

Thanks!

- Fernando

#486 lore raw

Re: [PATCH 2/5] ext3: call blkdev_issue_flush on fsync()

Quoted 7 lines · Fernando Luis Vázquez Cao
Quoted 6 lines · Jeff Garzik
Quoted 5 lines · Fernando Luis Vázquez Cao

To ensure that bits are truly on-disk after an fsync or fdatasync, we should force a disk flush explicitly when there is dirty data/metadata and the journal didn't emit a write barrier (either because metadata is not being synched or barriers are disabled).

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Your patches do not seem to propagate the issue-flush error code, even when it is easily available.

Oops... you are right. I will fix that.

I reflected your comments in the new version of the patch set.While at it I also modified the respective reiserfs and xfs fsync functions so that, at least to some extent,they propagate the issue-flush error code.

I'll be replying to this email with the new patches.

Thanks,

Fernando

#487 lore raw

PATCH 1/7 block: Add block_flush_device()

This patch adds a helper function that should be used by filesystems that need to flush the underlying block device on fsync()/fdatasync().

---

fs/buffer.c +11 -0
diff -urNp linux-2.6.29-orig/fs/buffer.c linux-2.6.29/fs/buffer.c
@@ -165,6 +165,17 @@ void end_buffer_write_sync(struct buffer
 	put_bh(bh);
 }
 
+/* Issue flush of write caches on the block device */
+int block_flush_device(struct block_device *bdev)
+{
+	int ret = 0;
+
+	ret = blkdev_issue_flush(bdev, NULL);
+
+	return (ret == -EOPNOTSUPP) ? 0 : ret;
+}
+EXPORT_SYMBOL(block_flush_device);
+
 /*
  * Write out and wait upon all the dirty data associated with a block
  * device via its mapping.  Does not take the superblock lock.
include/linux/buffer_head.h +1 -0
diff -urNp linux-2.6.29-orig/include/linux/buffer_head.h linux-2.6.29/include/linux/buffer_head.h
@@ -238,6 +238,7 @@ int nobh_write_end(struct file *, struct
 int nobh_truncate_page(struct address_space *, loff_t, get_block_t *);
 int nobh_writepage(struct page *page, get_block_t *get_block,
                         struct writeback_control *wbc);
+int block_flush_device(struct block_device *bdev);
 
 void buffer_init(void);
 
#488 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 10 lines · Fernando Luis Vázquez Cao

This patch adds a helper function that should be used by filesystems that need to flush the underlying block device on fsync()/fdatasync().

---

diff -urNp linux-2.6.29-orig/fs/buffer.c linux-2.6.29/fs/buffer.c

fs/buffer.c void end_buffer_write_sync(struct buffer · line 169

Quoted 5 lines
fs/buffer.c +2 -0
@@ -165,6 +165,17 @@ void end_buffer_write_sync(struct buffer
  	put_bh(bh);
  }
 
+/* Issue flush of write caches on the block device */
+int block_flush_device(struct block_device *bdev)

I don't consider this an improvement over using blkdev_issue_flush().

fs/buffer.c void end_buffer_write_sync(struct buffer · line 173

Quoted 4 lines
fs/buffer.c +4 -0
+{
+	int ret = 0;
+
+	ret = blkdev_issue_flush(bdev, NULL);

The problem lies in using NULL for error_sector argument which shows a subtle deficiency of the current implementation/usage of barriers based on a write cache flushing.

I intend to document the issue with adding the FIXME to the current users of blkdev_issue_flush() so the problem is at least known and not forgotten (fixing it would require some work from both block and fs sides and unfortunately there wasn't even a willingness to discuss possible solutions few years back when the original code was added).

Thanks,
Bart

#489 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 14 lines · Bartlomiej Zolnierkiewicz
Quoted 13 lines · Fernando Luis Vázquez Cao

This patch adds a helper function that should be used by filesystems that need to flush the underlying block device on fsync()/fdatasync().

---

diff -urNp linux-2.6.29-orig/fs/buffer.c linux-2.6.29/fs/buffer.c

fs/buffer.c +0 -0
@@ -165,6 +165,17 @@ void end_buffer_write_sync(struct buffer
  	put_bh(bh);
  }
 
Quoted 4 lines

fs/buffer.c void end_buffer_write_sync(struct buffer · line 169

Quoted 2 lines
fs/buffer.c +2 -0
+/* Issue flush of write caches on the block device */
+int block_flush_device(struct block_device *bdev)

I don't consider this an improvement over using blkdev_issue_flush().

The reason I used a wrapper is that I did not like the semantics provided by blkdev_issue_flush(). On the one hand, I did not want to pass -EOPNOTSUPP to filesystems (it is not an error filesystems should care about). On the other hand it is weird that some filesystems use blkdev_issue_flush() when they want emit a barrier. blkdev_issue_flush() happens to be implemented as an empty (block layer) barrier, but I think that is an implementation detail filesystems should not neet to know about. Indeed I am working on a patch that implements blkdev_issue_empty_barrier(), so that we can optimize fsync() flushes and filesystem-originated barriers independently in the block layer.

Judging from your comments below, it seems we are in the same page regarding this issue.

Again, thank you for you feedback!

- Fernando

Quoted 17 lines

fs/buffer.c void end_buffer_write_sync(struct buffer · line 173

Quoted 4 lines
fs/buffer.c +4 -0
+{
+	int ret = 0;
+
+	ret = blkdev_issue_flush(bdev, NULL);

The problem lies in using NULL for error_sector argument which shows a subtle deficiency of the current implementation/usage of barriers based on a write cache flushing.

I intend to document the issue with adding the FIXME to the current users of blkdev_issue_flush() so the problem is at least known and not forgotten (fixing it would require some work from both block and fs sides and unfortunately there wasn't even a willingness to discuss possible solutions few years back when the original code was added).

Thanks,
Bart

#490 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

In reply to Fernando Luis Vázquez Cao

fs/buffer.c void end_buffer_write_sync(struct buffer · line 175

Quoted 5 lines · Fernando Luis Vázquez Cao
fs/buffer.c +5 -0
+	int ret = 0;
+
+	ret = blkdev_issue_flush(bdev, NULL);
+
+	return (ret == -EOPNOTSUPP) ? 0 : ret;

Btw, why do we do that silly EOPNOTSUPP at all?

If the device doesn't support flushing, we should

- set a flag in the device saying so, and not ever try to flush again on that device (who knows how long it took for the device to say "I can't do this"? We don't want to keep on doing it)

- return "done". There's nothing sane the caller can do with the error code anyway, it just has to assume that the device basically doesn't reorder writes.

So wouldn't it be better to just fix blkdev_issue_flush() to not do those crazy error codes?

[ The same thing probably goes for those ENXIO errors, btw. If we don't
have a bd_disk or a queue, why would the caller care about it? ]

Jens?

Linus

#491 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 1 line · Linus Torvalds

So wouldn't it be better to just fix blkdev_issue_flush() to not do those crazy error codes?

Yes, much.

Jeff

#492 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

In reply to Linus Torvalds
Quoted 17 lines · Linus Torvalds

fs/buffer.c void end_buffer_write_sync(struct buffer · line 175

Quoted 5 lines · Fernando Luis Vázquez Cao
fs/buffer.c +5 -0
+	int ret = 0;
+
+	ret = blkdev_issue_flush(bdev, NULL);
+
+	return (ret == -EOPNOTSUPP) ? 0 : ret;

Btw, why do we do that silly EOPNOTSUPP at all?

If the device doesn't support flushing, we should

- set a flag in the device saying so, and not ever try to flush again on that device (who knows how long it took for the device to say "I can't do this"? We don't want to keep on doing it)

Quoted 6 lines

- return "done". There's nothing sane the caller can do with the error code anyway, it just has to assume that the device basically doesn't reorder writes.

So wouldn't it be better to just fix blkdev_issue_flush() to not do those crazy error codes?

The problem is that we may not know upfront, so it sort-of has to be this trial approach where the first barrier issued will notice and fail with -EOPNOTSUPP. Sure, we could cache this value, but it's pretty pointless since the filesystem will stop sending barriers in this case. As it also modifed fs behaviour, we need to pass the info back.

For blkdev_issue_flush() it may not be very interesting, since there's not much we can do about that. Just seems like very bad style to NOT return an error in such a case. You can assume that ordering is fine, but it definitely wont be in all case (eg devices that have write back caching on by default and don't support flush). So the nice thing to do there is actually tell the caller about it. So the same error is reused as we do for actualy write barriers that have data attached.

Quoted 2 lines

[ The same thing probably goes for those ENXIO errors, btw. If we don't
have a bd_disk or a queue, why would the caller care about it? ]

Right, that is pretty pointless.

Signature · 1 line
Jens Axboe
#493 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 4 lines · Jens Axboe

The problem is that we may not know upfront, so it sort-of has to be this trial approach where the first barrier issued will notice and fail with -EOPNOTSUPP.

Well, absolutely. Except I don't think you shoul use ENOTSUPP, you should just set a bit in the "struct request_queue", and then return 0.

IOW, something like this

	--- a/block/blk-barrier.c
	+++ b/block/blk-barrier.c
	@@ -318,6 +318,9 @@ int blkdev_issue_flush(struct block_device *bdev, sector_t *error_sector)
	 	if (!q)
	 		return -ENXIO;
	 
	+	if (is_queue_noflush(q))
	+		return 0;
	+
	 	bio = bio_alloc(GFP_KERNEL, 0);
	 	if (!bio)
	 		return -ENOMEM;
	@@ -339,7 +342,7 @@ int blkdev_issue_flush(struct block_device *bdev, sector_t *error_sector)
	 
	 	ret = 0;
	 	if (bio_flagged(bio, BIO_EOPNOTSUPP))
	-		ret = -EOPNOTSUPP;
	+		set_queue_noflush(q);
	 	else if (!bio_flagged(bio, BIO_UPTODATE))
	 		ret = -EIO;

which just returns 0 if we don't support flushing on that queue.

(Obviously incomplete patch, which is why I also intentionally whitespace-broke it).

Quoted 2 lines

Sure, we could cache this value, but it's pretty
pointless since the filesystem will stop sending barriers in this case.

Well no, it won't. Or rather, it will have to have such a stupid per-filesystem flag, for no good reason.

Quoted 5 lines

For blkdev_issue_flush() it may not be very interesting, since there's not much we can do about that. Just seems like very bad style to NOT return an error in such a case. You can assume that ordering is fine, but it definitely wont be in all case (eg devices that have write back caching on by default and don't support flush).

So?

The thing is, you can't _do_ anything about it. So what's the point in returning an error? The caller cannot possibly care - because there is nothing the caller can really do.

Sure, the device may or may not re-order things, but since the caller can't know, and can't really do a thing about it _anyway_, you're just better off not even confusing anybody.

Linus

#494 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 37 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe

The problem is that we may not know upfront, so it sort-of has to be this trial approach where the first barrier issued will notice and fail with -EOPNOTSUPP.

Well, absolutely. Except I don't think you shoul use ENOTSUPP, you should just set a bit in the "struct request_queue", and then return 0.

IOW, something like this

	--- a/block/blk-barrier.c
	+++ b/block/blk-barrier.c
	@@ -318,6 +318,9 @@ int blkdev_issue_flush(struct block_device *bdev, sector_t *error_sector)
	 	if (!q)
	 		return -ENXIO;
	 
	+	if (is_queue_noflush(q))
	+		return 0;
	+
	 	bio = bio_alloc(GFP_KERNEL, 0);
	 	if (!bio)
	 		return -ENOMEM;
	@@ -339,7 +342,7 @@ int blkdev_issue_flush(struct block_device *bdev, sector_t *error_sector)
	 
	 	ret = 0;
	 	if (bio_flagged(bio, BIO_EOPNOTSUPP))
	-		ret = -EOPNOTSUPP;
	+		set_queue_noflush(q);
	 	else if (!bio_flagged(bio, BIO_UPTODATE))
	 		ret = -EIO;

which just returns 0 if we don't support flushing on that queue.

Quoted 8 lines

(Obviously incomplete patch, which is why I also intentionally whitespace-broke it).

Quoted 2 lines

Sure, we could cache this value, but it's pretty
pointless since the filesystem will stop sending barriers in this case.

Well no, it won't. Or rather, it will have to have such a stupid per-filesystem flag, for no good reason.

Sorry, I just don't see much point to doing it this way instead. So now the fs will have to check a queue bit after it has issued the flush, how is that any better than having the 'error' returned directly?

Quoted 11 lines
Quoted 5 lines

For blkdev_issue_flush() it may not be very interesting, since there's not much we can do about that. Just seems like very bad style to NOT return an error in such a case. You can assume that ordering is fine, but it definitely wont be in all case (eg devices that have write back caching on by default and don't support flush).

So?

The thing is, you can't _do_ anything about it. So what's the point in returning an error? The caller cannot possibly care - because there is nothing the caller can really do.

Not for blkdev_issue_flush(), all they can do is report about the device. And even that would be a vague "Your data may or may not be safe, we don't know".

Quoted 3 lines

Sure, the device may or may not re-order things, but since the caller can't know, and can't really do a thing about it _anyway_, you're just better off not even confusing anybody.

I'd call that a pretty reckless approach to data integrity, honestly. You HAVE to issue an error in this case. Then the user/admin can at least check up on the device stack in question, and determine whether this is an issue or not. That goes for both blkdev_issue_flush() and the actual barrier write. And perhaps the cached value is then of some use, since you then know when to warn (bit not already set) and you can keep the warning in blkdev_issue_flush() instead of putting it in every call site.

Signature · 1 line
Jens Axboe
#495 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 36 lines · Jens Axboe
Quoted 35 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe

The problem is that we may not know upfront, so it sort-of has to be this trial approach where the first barrier issued will notice and fail with -EOPNOTSUPP.

Well, absolutely. Except I don't think you shoul use ENOTSUPP, you should just set a bit in the "struct request_queue", and then return 0.

IOW, something like this

	--- a/block/blk-barrier.c
	+++ b/block/blk-barrier.c
	@@ -318,6 +318,9 @@ int blkdev_issue_flush(struct block_device *bdev, sector_t *error_sector)
	 	if (!q)
	 		return -ENXIO;
	 
	+	if (is_queue_noflush(q))
	+		return 0;
	+
	 	bio = bio_alloc(GFP_KERNEL, 0);
	 	if (!bio)
	 		return -ENOMEM;
	@@ -339,7 +342,7 @@ int blkdev_issue_flush(struct block_device *bdev, sector_t *error_sector)
	 
	 	ret = 0;
	 	if (bio_flagged(bio, BIO_EOPNOTSUPP))
	-		ret = -EOPNOTSUPP;
	+		set_queue_noflush(q);
	 	else if (!bio_flagged(bio, BIO_UPTODATE))
	 		ret = -EIO;

which just returns 0 if we don't support flushing on that queue.

(Obviously incomplete patch, which is why I also intentionally whitespace-broke it).

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Sure, we could cache this value, but it's pretty
pointless since the filesystem will stop sending barriers in this case.

Well no, it won't. Or rather, it will have to have such a stupid per-filesystem flag, for no good reason.

Sorry, I just don't see much point to doing it this way instead. So now the fs will have to check a queue bit after it has issued the flush, how is that any better than having the 'error' returned directly?

AFAICS, the aim is simply to return zero rather than EOPNOTSUPP, for the not-supported case, rather than burdening all callers with such checks.

Which is quite reasonable for Fernando's patch -- the direct call fsync case.

But that leaves open the possibility that some people really do want the EOPNOTSUPP return value, I guess? Do existing callers need that?

Quoted 15 lines
Quoted 8 lines
Quoted 5 lines

For blkdev_issue_flush() it may not be very interesting, since there's not much we can do about that. Just seems like very bad style to NOT return an error in such a case. You can assume that ordering is fine, but it definitely wont be in all case (eg devices that have write back caching on by default and don't support flush).

So?

The thing is, you can't _do_ anything about it. So what's the point in returning an error? The caller cannot possibly care - because there is nothing the caller can really do.

Not for blkdev_issue_flush(), all they can do is report about the device. And even that would be a vague "Your data may or may not be safe, we don't know".

Quoted 1 line

Sure, the device may or may not re-order things, but since the caller can't know, and can't really do a thing about it _anyway_, you're just better off not even confusing anybody.

Quoted 8 lines

I'd call that a pretty reckless approach to data integrity, honestly. You HAVE to issue an error in this case. Then the user/admin can at least check up on the device stack in question, and determine whether this is an issue or not. That goes for both blkdev_issue_flush() and the actual barrier write. And perhaps the cached value is then of some use, since you then know when to warn (bit not already set) and you can keep the warning in blkdev_issue_flush() instead of putting it in every call site.

Indeed -- if the drive tells us it failed the cache flush, it seems self-evident that we should be passing that failure back to userspace where possible.

And as the patches show, it is definitely possible to return a FLUSH CACHE error back to an fsync(2) caller [though, yes, I certainly recognize fsync is not the only generator of these requests].

Jeff

#496 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 48 lines · Jeff Garzik
Quoted 38 lines · Jens Axboe
Quoted 37 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe

The problem is that we may not know upfront, so it sort-of has to be this trial approach where the first barrier issued will notice and fail with -EOPNOTSUPP.

Well, absolutely. Except I don't think you shoul use ENOTSUPP, you should just set a bit in the "struct request_queue", and then return 0.

IOW, something like this

	--- a/block/blk-barrier.c
	+++ b/block/blk-barrier.c
	@@ -318,6 +318,9 @@ int blkdev_issue_flush(struct block_device *bdev, sector_t *error_sector)
	 	if (!q)
	 		return -ENXIO;
	 
	+	if (is_queue_noflush(q))
	+		return 0;
	+
	 	bio = bio_alloc(GFP_KERNEL, 0);
	 	if (!bio)
	 		return -ENOMEM;
	@@ -339,7 +342,7 @@ int blkdev_issue_flush(struct block_device *bdev, sector_t *error_sector)
	 
	 	ret = 0;
	 	if (bio_flagged(bio, BIO_EOPNOTSUPP))
	-		ret = -EOPNOTSUPP;
	+		set_queue_noflush(q);
	 	else if (!bio_flagged(bio, BIO_UPTODATE))
	 		ret = -EIO;

which just returns 0 if we don't support flushing on that queue.

(Obviously incomplete patch, which is why I also intentionally whitespace-broke it).

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Sure, we could cache this value, but it's pretty
pointless since the filesystem will stop sending barriers in this case.

Well no, it won't. Or rather, it will have to have such a stupid per-filesystem flag, for no good reason.

Sorry, I just don't see much point to doing it this way instead. So now the fs will have to check a queue bit after it has issued the flush, how is that any better than having the 'error' returned directly?

Quoted 9 lines

AFAICS, the aim is simply to return zero rather than EOPNOTSUPP, for the not-supported case, rather than burdening all callers with such checks.

Which is quite reasonable for Fernando's patch -- the direct call fsync case.

But that leaves open the possibility that some people really do want the EOPNOTSUPP return value, I guess? Do existing callers need that?

As far as I know, reiserfs is the only one actively using it to choose different code. It moves a single wait_on_buffer when barriers are on, which I took out once to simplify the code. Ric saw it in some benchmark numbers and I put it back in.

Given that it was a long time ago, I don't have a problem with changing it to work like all the other filesystems.

-chris

#497 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 8 lines · Chris Mason

As far as I know, reiserfs is the only one actively using it to choose different code. It moves a single wait_on_buffer when barriers are on, which I took out once to simplify the code. Ric saw it in some benchmark numbers and I put it back in.

Given that it was a long time ago, I don't have a problem with changing it to work like all the other filesystems.

When it was a win on reiserfs back then maybe it would be a win on ext4 or xfs today too?

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#498 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 7 lines · Andi Kleen
Quoted 6 lines · Chris Mason

As far as I know, reiserfs is the only one actively using it to choose different code. It moves a single wait_on_buffer when barriers are on, which I took out once to simplify the code. Ric saw it in some benchmark numbers and I put it back in.

Quoted 5 lines
Quoted 2 lines

Given that it was a long time ago, I don't have a problem with changing it to work like all the other filesystems.

When it was a win on reiserfs back then maybe it would be a win on ext4 or xfs today too?

It could be, but you get into some larger changes. The theory behind the code was that writeback cache is on, so wait_on_buffer isn't really going to give you a worthwhile error return anyway. Might as well do the wait_on_buffer some time later and fix up the commit blocks if it didn't work out.

We're still arguing about barriers being a good idea all these years later, and the drives are better at them than they used to be. So, I'd rather see less complex code in the filesystems than more.

-chris

#499 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

In reply to Jeff Garzik
Quoted 4 lines · Jeff Garzik

Indeed -- if the drive tells us it failed the cache flush, it seems self-evident that we should be passing that failure back to userspace where possible.

That's not what EOPNOTSUPP means!

EOPNOTSUPP doesn't mean "the cache flush failed". It just means "I don't support cache flushing".

No failure anywhere. See?

Maybe the operation isn't supported becasue there are no caches? Who the hell knows? Nobody. The layer just said "I don't support this". For example, maybe it just cannot translate the "flush cache" op into its own command set, because the thing doesn't _do_ anything like that.

For a concrete example, look at the "loop" driver. It literally returns EOPNOTSUPP if the filesystem doesn't have a "fsync()" thing. Ok, so it can't do serialization - does that mean that the caller should fail entirely? No. But it means that the caller cannot serialize, so now the caller has two choices:

- not work at all

- ignore it, and assume that a device without serialization is serialized enough as-is.

Those are the two only choices. The caller knows that it can't flush. What would you _suggest_ it do? Just stop, and do nothing at all? I rally don't think that's a useful or valid approach.

And notice - at NO TIME did anythign actually fail. It's just that the particular protocol didn't support that empty flush op.

(Also note that block/blk-barrier.c really does an empty barrier command. If we were to be talking about a real IO with a real payload and the "barrier" bit set, that would be different. But we really aren't.)

Linus

#500 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik

Indeed -- if the drive tells us it failed the cache flush, it seems self-evident that we should be passing that failure back to userspace where possible.

EOPNOTSUPP doesn't mean "the cache flush failed". It just means "I don't support cache flushing".

No failure anywhere. See?

Hence my statement of

the aim is simply to return zero rather than EOPNOTSUPP [...] which is quite reasonable

I think we are all getting a bit confused whether we are discussing

(a) EOPNOTSUPP return value,
or
(b) _all possible_ blkdev_issue_flush() error return values.

As I read it, you are talking about (a) and Jens responded to (b). But maybe I am wrong.

So I have these observations:

1) fsync(2) should not return EOPNOTSUPP, if the block device does not support cache flushing. This seems to agree with Linus's patch.

2) A Linux filesystem MIGHT care about EOPNOTSUPP return value, as that return value does provide information about the future value of cache flushes.

3) However, at present NONE of the blkdev_issue_flush() callers use EOPNOTSUPP in any way. In fact, none of the current callers check the return value at all.

4) Furthermore, handling lack of cache flush support at the block layer, rather than per-filesystem, makes more sense to me.

But I am biased towards storage, so what do I know :)

5) Based on observation #3, the current kernel should be changed to return USEFUL blkdev_issue_flush() return values back to userspace. Fernando's patches head in this direction, as does my most recent file_fsync patch.

Jeff

#501 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

In reply to Jens Axboe
Quoted 4 lines · Jens Axboe

Sorry, I just don't see much point to doing it this way instead. So now the fs will have to check a queue bit after it has issued the flush, how is that any better than having the 'error' returned directly?

No.

Now the fs SHOULD NEVER CHECK AT ALL.

Either it did the ordering, or the FS cannot do anything about it.

That's the point. EOPNOTSUPP is n ot a useful error message. You can't _do_ anything about it.

Quoted 5 lines
Quoted 3 lines

Sure, the device may or may not re-order things, but since the caller can't know, and can't really do a thing about it _anyway_, you're just better off not even confusing anybody.

I'd call that a pretty reckless approach to data integrity, honestly.

It has _nothing_ to do with 'reckless'. It has everything to do with 'you can't do anything about it'.

Quoted 1 line

You HAVE to issue an error in this case.

No. Returning an error just means that now the box is useless. Nobody can do anything about it. Not the admin, not the driver writer, not anybody.

Ok, so a device didn't support flushing. We don't know why, we don't know if it needed it, we simply don't know. There's nothing to do. But returning an error to user mode is unacceptable, because that will result in everything just -failing-.

And total failure is much worse than "we don't know whether the thing serialized".

Linus

#502 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe

Sorry, I just don't see much point to doing it this way instead. So now the fs will have to check a queue bit after it has issued the flush, how is that any better than having the 'error' returned directly?

Quoted 8 lines

No.

Now the fs SHOULD NEVER CHECK AT ALL.

Either it did the ordering, or the FS cannot do anything about it.

That's the point. EOPNOTSUPP is n ot a useful error message. You can't _do_ anything about it.

My point is that some file systems may or may not have different paths or optimizations depending on whether barriers are enabled and working or not. Apparently that's just reiserfs and Chris says we can remove it, so it is probably a moot point.

And that is for the barrier write btw, NOT blkdev_issue_flush(). For the latter it obviously doesn't matter if you return -EOPNOTSUPP or not, as long as you warn. And before you yell on, please read on below...

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Sure, the device may or may not re-order things, but since the caller can't know, and can't really do a thing about it _anyway_, you're just better off not even confusing anybody.

I'd call that a pretty reckless approach to data integrity, honestly.

It has _nothing_ to do with 'reckless'. It has everything to do with 'you can't do anything about it'.

No, but you better damn well inform of such a discovery!

Quoted 4 lines
Quoted 1 line

You HAVE to issue an error in this case.

No. Returning an error just means that now the box is useless. Nobody can do anything about it. Not the admin, not the driver writer, not anybody.

What, that's nonsense. The admin can certainly check whether it's an issue or not, and he should. That's different from handling it in the kernel or in the application, but you have to inform about it. I honestly cannot fathom why you don't think that is important.

Quoted 7 lines

Ok, so a device didn't support flushing. We don't know why, we don't know if it needed it, we simply don't know. There's nothing to do. But returning an error to user mode is unacceptable, because that will result in everything just -failing-.

And total failure is much worse than "we don't know whether the thing serialized".

That is not what I meant with returning it to the user. My point was that you have to notify that the error occured, which means putting a printk() (or whatever) in that blkdev_issue_flush(). I guess most of the miscommunication stems from this, I don't want -EOPNOTSUPP returned to user space, but I want some notification that tells the admin that this device doesn't support flushes. And if the file systems use the same path for barrier or no barriers, then it's perfectly fine to have them share the very same "flush doesn't work bit" and the same single warning that we don't know whether ordering is preserved on this device or not.

IOW, what I'm advocating is just a simple:

@@

        if (err == -EOPNOTSUPP) {
+               if (!is_queue_noflush(q)) {
+                       warn();
                        set_queue_noflush(q);
                }
        }

change to the pseudo-patch you posted.

Signature · 1 line
Jens Axboe
#503 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 5 lines · Jens Axboe
Quoted 3 lines

It has _nothing_ to do with 'reckless'. It has everything to do with 'you can't do anything about it'.

No, but you better damn well inform of such a discovery!

Well, if that's the issue, then just add a printk to that 'blkdev_issue_flush()', and now you have that informational message in _one_ place, instead of havign each filesystem having to do it over and over again.

Quoted 5 lines
Quoted 2 lines

No. Returning an error just means that now the box is useless. Nobody can do anything about it. Not the admin, not the driver writer, not anybody.

What, that's nonsense. The admin can certainly check whether it's an issue or not, and he should.

If it's just informational, then again - why should the filesystem care?

Returning an error to the caller is never the right thing to do. The caller can't do anything sane about it.

If you argue that the admin wants to know, then sure, make that

if (bio_flagged(bio, BIO_EOPNOTSUPP))
- ret = -EOPNOTSUPP;
+ set_queue_noflush(q);

"set_queue_noflush()" function print a warning message when it sets the bit.

Problem solved.

Quoted 3 lines

That's different from handling it in the kernel or in the application, but you have to inform about it. I honestly cannot fathom why you don't think that is important.

I cannot fathom why you can _possibly_ think that this is something that can and must be done something about in the caller. When the caller obviously has no real option except to ignore the error _anyway_.

That was always my point. Returning an error is INSANE, because ther is no valid thing that the caller can possibly do.

If you want it logged, fine. But THAT DOES NOT CHANGE ANYTHING. It would still be wrong to return the error, since the caller _still_ can't do anything about it.

Linus

#504 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 34 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe
Quoted 2 lines

It has _nothing_ to do with 'reckless'. It has everything to do with 'you can't do anything about it'.

No, but you better damn well inform of such a discovery!

Well, if that's the issue, then just add a printk to that 'blkdev_issue_flush()', and now you have that informational message in _one_ place, instead of havign each filesystem having to do it over and over again.

Quoted 5 lines
Quoted 2 lines

No. Returning an error just means that now the box is useless. Nobody can do anything about it. Not the admin, not the driver writer, not anybody.

What, that's nonsense. The admin can certainly check whether it's an issue or not, and he should.

If it's just informational, then again - why should the filesystem care?

Returning an error to the caller is never the right thing to do. The caller can't do anything sane about it.

If you argue that the admin wants to know, then sure, make that

if (bio_flagged(bio, BIO_EOPNOTSUPP))
- ret = -EOPNOTSUPP;
+ set_queue_noflush(q);

"set_queue_noflush()" function print a warning message when it sets the bit.

Problem solved.

Quoted 2 lines

That's different from handling it in the kernel or in the application, but you have to inform about it. I honestly cannot fathom why you don't think that is important.

Quoted 8 lines

I cannot fathom why you can _possibly_ think that this is something that can and must be done something about in the caller. When the caller obviously has no real option except to ignore the error _anyway_.

That was always my point. Returning an error is INSANE, because ther is no valid thing that the caller can possibly do.

If you want it logged, fine. But THAT DOES NOT CHANGE ANYTHING. It would still be wrong to return the error, since the caller _still_ can't do anything about it.

Linus

One thing the caller could do is to disable the write cache on the device. A second would be to stop using the transactions - skip the journal, just go back to ext2 mode or BSD like soft updates.

Basically, it lets the file system know that its data integrity building blocks are not really there and allows it (if it cares) to try and minimize the chance of data loss.

Ric

#505 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 2 lines · Ric Wheeler

One thing the caller could do is to disable the write cache on the device.

First off, that's not the callers job. If the sysadmin enabled it, some random filesystem shouldn't disable it.

Secondly, this whole insane belief that "write cache" has anything to do with "unable to flush" is just bogus.

Quoted 2 lines

A second would be to stop using the transactions - skip the journal, just go back to ext2 mode or BSD like soft updates.

f*ck me, what's so hard with understanding that EOPNOTSUPP doesn't mean "no ordering". It means what it says - the op isn't supported. For all you know, ALL WRITES MAY BE TOTALLY ORDERED, but perhaps there is no way to make a _single_ write totally atomic (ie the "set barrier on a command that actually does IO").

Besides, why the hell do you think the filesystem (again) should do something that the admin didn't ask it to do.

If the admin wants the thing to fall back to ext2, then he can ask to disable the journal.

Quoted 3 lines

Basically, it lets the file system know that its data integrity building blocks are not really there and allows it (if it cares) to try and minimize the chance of data loss.

Your whole idiotic "as a filesystem designer I know better than everybody else" model where the filesystem is in total control is total crap.

The fact is, it's not the filesystems job to make that decision. If the admin wants to have write caching enabled, the filesystem should get the hell out of the way.

What about laptop mode? Do you expect your filesystem to always decide that "ok, the user wanted to spin down disks, but I know better"?

What about people who have UPS's and don't worry about that part? They want write caching on the disk, and simply don't want to sync? They still worry about OS crashing, since they run random -git development kernels?

In short, stop this IDIOTIC notion that you know better. YOU DO NOT KNOW BETTER. The filesystem DOES NOT KNOW BETTER. It should damn well not do those kinds of decisions that are simply not filesystem decisions to make!

Linus

#506 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Ric Wheeler

One thing the caller could do is to disable the write cache on the device.

First off, that's not the callers job. If the sysadmin enabled it, some random filesystem shouldn't disable it.

Completely agree with that, that is why I want the error logged instead of returned. The write cache MAY be involved, but it may also be something entirely different. The cache may be perfectly fine and ordered but just not supporting flush cache because it doesn't need to (it has battery backing). The important bit is informing the admin of the situation, then it's up to the admin to look into the storage stack and determine if this is a real problem or not. There's nothing the kernel can do about it.

Signature · 1 line
Jens Axboe
#507 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 2 lines · Ric Wheeler

One thing the caller could do is to disable the write cache on the device.

First off, that's not the callers job. If the sysadmin enabled it, some random filesystem shouldn't disable it.

Secondly, this whole insane belief that "write cache" has anything to do with "unable to flush" is just bogus.

First I have heard anyone (other than you above) claim that "unable to flush" is tied to the write cache on disks.

What I was responding to is your objection to exposing the proper error codes to the file system layer instead of hiding them in the block layer. True, the write cache example I used is pretty contrived, but it would be a valid strategy if your sacred sys admin had mounted with the "I do care about my data" mount option and left it up to the file system to make it happen.

Quoted 6 lines
Quoted 3 lines

A second would be to stop using the transactions - skip the journal, just go back to ext2 mode or BSD like soft updates.

f*ck me, what's so hard with understanding that EOPNOTSUPP doesn't mean "no ordering". It means what it says - the op isn't supported. For all you know, ALL WRITES MAY BE TOTALLY ORDERED, but perhaps there is no way to make a _single_ write totally atomic (ie the "set barrier on a command that actually does IO").

Now you are just being silly. The drive and the write cache - without barriers or similar tagged operations - will almost certainly reorder all of the IO's internally.

No one designs code based on the "it might be ordered" basis.

The way the barriers work does absolutely give you full ordering. All previous IO's are sent to the drive and flushed (barrier flush 1), the commit record is sent down followed by a second barrier flush. There is no way that the commit block will pass its dependent IO's.

Quoted 5 lines

Besides, why the hell do you think the filesystem (again) should do something that the admin didn't ask it to do.

If the admin wants the thing to fall back to ext2, then he can ask to disable the journal.

Quoted 9 lines
Quoted 4 lines

Basically, it lets the file system know that its data integrity building blocks are not really there and allows it (if it cares) to try and minimize the chance of data loss.

Your whole idiotic "as a filesystem designer I know better than everybody else" model where the filesystem is in total control is total crap.

The fact is, it's not the filesystems job to make that decision. If the admin wants to have write caching enabled, the filesystem should get the hell out of the way.

This is not me being snotty - this is really very basic to how transactions work. You need ordering and file systems (or data bases) that use transactions must have these building blocks to do the job right.

Your argument seems to be, "Well, it will mostly be ordered anyway, as long as you don't lose power" which I simply don't agree is a good assumption.

The logic conclusion of that argument is that we really should not use transactions at all - basically remove the journal from ext3/4, xfs, btrfs, etc. That is a point of view - drives are crap, journalling does not help anyway, why bother.

Quoted 2 lines

What about laptop mode? Do you expect your filesystem to always decide that "ok, the user wanted to spin down disks, but I know better"?

Laptop mode is pretty much a red herring here. Mount it without barriers enabled - your drive will still spin up occasionally, but as you argued above, that existing options allows you the user/admin to make that trade off.

Quoted 2 lines

What about people who have UPS's and don't worry about that part? They want write caching on the disk, and simply don't want to sync? They still worry about OS crashing, since they run random -git development kernels?

If you run with a UPS or have a battery backed write cache, you should run without barriers since both of those mechanisms give you the required promise of ordering even in face of power outage. Again, mount with barriers disabled (or rely on the storage target to ignore your cache flush commands, which higher end gear will do on a cache flush command).

Not hard to do, no additional code needed. We can even automate it as it is done in some of the linux based home storage boxes.

Quoted 4 lines

In short, stop this IDIOTIC notion that you know better. YOU DO NOT KNOW BETTER. The filesystem DOES NOT KNOW BETTER. It should damn well not do those kinds of decisions that are simply not filesystem decisions to make!

Linus

Not surprisingly, I still disagree with you. Based, strangely enough, on looking at real data over many years, not just my personal experience with a small handful of drives.

If you don't want to run with the data integrity that we have painfully baked into the file & storage stack over many years, you can simply mount without barriers.

Why tear down & attack the infrastructure for those users who do care?

ric

#508 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 4 lines · Ric Wheeler

Now you are just being silly. The drive and the write cache - without barriers or similar tagged operations - will almost certainly reorder all of the IO's internally.

You do realize that the "drive" may not be a drive at all?

But apparently you don't. You really seem to see just your own case, and have blinders on for everything else.

That "drive" may be some virtualized device. It may be some super-fancy memory mapped and largely undocumented random flash thing. It might be a network block device, it may be somebody's IO trace dummy layer, it may be anything at all.

Your filesystem doesn't know. It damn well not even _try_ to know, because it isn't the low-level driver.

The low-level driver - which you don't have a friggin clue about - may say that it doesn't support barrier IO for any random reason that has absolutely _nothing_ to do with any write caches or anything else. Maybe the device has the same ordering semantics as an Intel CPU has: writes are always seen in order on the disk, and reads are always speculated but will snoop in write buffers, and ther is no way to not do that.

See? EOPNOTSUPP means just that - it means that the driver doesn't support the notion of ordered IO. But that does not necessarily mean that the writes aren't always in order. It may well just mean that the drive is a thin shimmy layer over something else (for example, just a user level pipe), and the driver has NO IDEA what the end result is, and the protocol is simplistic and is just 'read' and 'write' and absolutely nothing else.

But you seem to NOT UNDERSTAND THIS.

I'm not interested in your inane drivel. Let's just say that your lack of understanding just means that your input is irrelevant, and leave it at that. Ok? Until you can see the bigger picture, just don't bother.

Linus

#509 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Ric Wheeler

Now you are just being silly. The drive and the write cache - without barriers or similar tagged operations - will almost certainly reorder all of the IO's internally.

You do realize that the "drive" may not be a drive at all?

Quoted 8 lines

But apparently you don't. You really seem to see just your own case, and have blinders on for everything else.

That "drive" may be some virtualized device. It may be some super-fancy memory mapped and largely undocumented random flash thing. It might be a network block device, it may be somebody's IO trace dummy layer, it may be anything at all.

The part that we seem to be skipping over in talking about EOPNOTSUPP is not what do we do when a barrier isn't supported (print a warning and move on), it's what do we do when a barrier works. I very much agree that EOPNOTSUPP tells us almost nothing.

The idea behind the original implementation was that when barriers did work, we could make some assumptions about how IO would be ordered around the barrier, and those assumptions would let us optimize things for the lying cheating cache enabled storage that we all know and love.

It turns out 6 years later that very few people are interested in those optimizations, and we're probably better off skipping them in favor of reducing the complexity of the code involved.

Jens has a little burial site all prepped for pdflush in his yard, dumping EOPNOTSUPP in there too wouldn't be a bad thing.

-chris

#510 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Ric Wheeler

Now you are just being silly. The drive and the write cache - without barriers or similar tagged operations - will almost certainly reorder all of the IO's internally.

You do realize that the "drive" may not be a drive at all?

But apparently you don't. You really seem to see just your own case, and have blinders on for everything else.

That "drive" may be some virtualized device. It may be some super-fancy memory mapped and largely undocumented random flash thing. It might be a network block device, it may be somebody's IO trace dummy layer, it may be anything at all.

Of course I realize that.

Most of the SSD devices, including ones that don't speak normal S-ATA/SCSI/etc, they have a write cache and will combine and re-order IO's.

Some of them have non-volatile write caches and those don't need barriers (flush, fua, what ever) because of batteries, capacitors or other magic hardware people came up with.

For the ones that do have a volatile write cache and can reorder IO's, transactions will still need the ordering primitives to survive a power failure reliably.

If you don't need or want to pay the price of ordering, you can today easily disable this by mounting without barriers.

As Mark pointed out, most S-ATA/SAS drives will flush the write cache when they see a bus reset so even without barriers, the cache will be preserved (or flushed) after a reboot or panic. Power outages are the problem barriers/flushes are meant to help with.

Quoted 5 lines

Your filesystem doesn't know. It damn well not even _try_ to know, because it isn't the low-level driver.

The low-level driver - which you don't have a friggin clue about - may say that it doesn't support barrier IO for any random reason that has absolutely _nothing_ to do with any write caches or anything else. Maybe the device has the same ordering semantics as an Intel CPU has: writes are always seen in order on the disk, and reads are always speculated but will snoop in write buffers, and ther is no way to not do that.

Quoted 7 lines

See? EOPNOTSUPP means just that - it means that the driver doesn't support the notion of ordered IO. But that does not necessarily mean that the writes aren't always in order. It may well just mean that the drive is a thin shimmy layer over something else (for example, just a user level pipe), and the driver has NO IDEA what the end result is, and the protocol is simplistic and is just 'read' and 'write' and absolutely nothing else.

But you seem to NOT UNDERSTAND THIS.

I'm not interested in your inane drivel. Let's just say that your lack of understanding just means that your input is irrelevant, and leave it at that. Ok? Until you can see the bigger picture, just don't bother.

Linus

If the low level device returns EOPNOTSUPP on a barrier op, that is fine. Running a transactional file system on that storage might or might not be a good idea, but at least we can log that and move on.

I agree with Chris that what happens when the device does not support the primitives is not the core issue.

The question is really what we do when you have a storage device in your box with a volatile write cache that does support flush or fua or similar. Using barriers & ordered transactions for these types of devices will give you a more reliable file system - less fsck time needed and better data integrity support for the (few?) applications that use fsync properly.

Ric

#511 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 3 lines · Ric Wheeler

The question is really what we do when you have a storage device in your box with a volatile write cache that does support flush or fua or similar.

Ok. Then you are talking about a different case - not EOPNOTSUPP.

[ Although it may be related in that maybe the admin can _force_ a
EOPNOTSUPP thing for when he wants to disable any "write barrier implies flush" thing.

IOW, we may end up with an _implementation_ detail where we overload a potential QUEUE_FLUSH_EOPNOTSUPP flag with two meanings - either "the driver told me a barrier isn't supported" or "the admin set that same flag by hand to disable barrier-related flush commands".

But that's just an implementation detail, of course. We could use two different flags, we could do the flags at different levels, whatever. ]

Quoted 4 lines

Using barriers & ordered transactions for these types of devices will give you a more reliable file system - less fsck time needed and better data integrity support for the (few?) applications that use fsync properly.

Sure. And it still shouldn't be the filesystem that _requires_ use of it.

The user (or low-level driver) may simply know better. The user may know that he trusts the disk more than anything else, and prefers to not actually emit the "FLUSH" command. Again, that's not something that the filesystem should know about, or care about. If the user trusts the disk subsystem and wants the performance, it's the users choice.

Even the _driver_ may know better.

Knowing the kinds of firmware bugs those drives have, it could even be a driver that simply black-lists certain disks as having known-broken FLUSH commands. We have _CPU's_ that corrupt memory on cache writeback ("wbinvl"), and those things are a lot more tested than most driver firmware is.

Do you realize just how buggy some of those flash drives are? Some of them will literally (a) report the wrong size and (b) lock up if you try to read from the last sector. Oops. Do you really expect such crap to even bother to honor some flush command? Good luck with that. They're designed as a floppy replacement.

Now, you can tell me that I shouldn't put a reliable filesystem on an el-cheapo flash drive and expect it to work, but I'm sorry, you're wrong. People _are_ supposed to be able to move their data around, and the filesystem shouldn't make judgement calls. If you want judgement calls, call your mom. Not your filesystem.

For another example, the driver might be a driver for a high-end battery-backup SCSI RAID controller. It knows that the controller _will_ write things out in the right order even in the case of a crash, but it may also know that the controller _also_ has a way to force a flush to actual hardware.

When do you want to force a flush? For hotplug events, for example. Maybe the disks won't be _connected_ any more afterwards - then the battery backup on the controller won't be helping, will it? So there may well be a flush event thing, but it's really up to the admin to decide whether it should be connected to a write barrier thing, or be a separate admin activity.

Maybe the admin is extra careful and anal, and decides that he wants to flush to disk platters _despite_ the battery backup. Maybe he doesn't trust the card. Maybe he does. Whatever. The point is that the admin might want to set a driver flag that does the flush or not, adn it's totally not a filesystem issue.

See? The filesystem has absolutely _no_place_ deciding these kinds of things. The only thing it can ask for is "please serialize", but what _level_ of serialization is simply not a filesystem decision to make.

And that very much includes the level of serialization that says "no serialization what-so-ever, and please go absolutely crazy with your cache". Not your choice.

So no, you can't have a pony.

Linus

#512 lore raw

Re: [PATCH 1/7] block: Add block_flush_device()

Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Ric Wheeler

The question is really what we do when you have a storage device in your box with a volatile write cache that does support flush or fua or similar.

Ok. Then you are talking about a different case - not EOPNOTSUPP.

So here's a test patch that attempts to just ignore such a failure to flush the caches. It will still flag the bio as BIO_EOPNOTSUPP, but that's merely maintaining the information in case the caller does want to see if that barrier failed or not. It may not actually be useful, in which case we can just kill that flag.

But it'll return 0 for a write, getting rid of hard retry logic in the file systems. It'll also ensure that blkdev_issue_flush() does not see the -EOPNOTSUPP and pass it back.

The first time we see such a failed barrier, we'll log a warning in dmesg about the block device. Subsequent failed barriers with -EOPNOTSUPP will bit warn.

Now, there's a follow up to this. If the device doesn't support barriers and the block layer fails them early, we should still do the ordering inside the block layer. Then we will at least not reorder there, even if the device may or may not order. I'll test this patch and provide a follow up patch that does that as well, before asking for any of this to be included. So that's a note to not apply this patch, it hasn't been tested!

commit 78ab31910c8c7b8853c1fd4d78c5f4ce2aebb516

Author: Jens Axboe <[email protected]>
Date:   Tue Mar 31 18:42:42 2009 +0200

barrier: Don't return -EOPNOTSUPP to the caller if the device does not support barriers

The caller cannot really do much about the situation anyway. Instead log a warning if this is the first such failed barrier we see, so that the admin can look into whether this poses a data integrity problem or not.

block/blk-barrier.c +2 -6
index f7dae57..8660146 100644
@@ -338,9 +338,7 @@ int blkdev_issue_flush(struct block_device *bdev, sector_t *error_sector)
 		*error_sector = bio->bi_sector;
 
 	ret = 0;
-	if (bio_flagged(bio, BIO_EOPNOTSUPP))
-		ret = -EOPNOTSUPP;
-	else if (!bio_flagged(bio, BIO_UPTODATE))
+	if (!bio_flagged(bio, BIO_UPTODATE))
 		ret = -EIO;
 
 	bio_put(bio);
@@ -408,9 +406,7 @@ int blkdev_issue_discard(struct block_device *bdev,
 		submit_bio(DISCARD_BARRIER, bio);
 
 		/* Check if it failed immediately */
-		if (bio_flagged(bio, BIO_EOPNOTSUPP))
-			ret = -EOPNOTSUPP;
-		else if (!bio_flagged(bio, BIO_UPTODATE))
+		if (!bio_flagged(bio, BIO_UPTODATE))
 			ret = -EIO;
 		bio_put(bio);
 	}
block/blk-settings.c +13 -0
index 59fd05d..0a81466 100644
@@ -463,6 +463,19 @@ void blk_queue_update_dma_alignment(struct request_queue *q, int mask)
 }
 EXPORT_SYMBOL(blk_queue_update_dma_alignment);
 
+void blk_queue_set_noflush(struct block_device *bdev)
+{
+	struct request_queue *q = bdev_get_queue(bdev);
+	char b[BDEVNAME_SIZE];
+
+	if (test_and_set_bit(QUEUE_FLAG_NOFLUSH, &q->queue_flags))
+		return;
+
+	printk(KERN_ERR "Device %s does not appear to honor cache flushes. "	
+			"This may mean that file system ordering guarantees "
+			"are not met.", bdevname(bdev, b));
+}
+
 static int __init blk_settings_init(void)
 {
 	blk_max_low_pfn = max_low_pfn - 1;
block/ioctl.c +1 -3
index 0f22e62..769f7be 100644
@@ -166,9 +166,7 @@ static int blk_ioctl_discard(struct block_device *bdev, uint64_t start,
 
 		wait_for_completion(&wait);
 
-		if (bio_flagged(bio, BIO_EOPNOTSUPP))
-			ret = -EOPNOTSUPP;
-		else if (!bio_flagged(bio, BIO_UPTODATE))
+		if (!bio_flagged(bio, BIO_UPTODATE))
 			ret = -EIO;
 		bio_put(bio);
 	}
fs/bio.c +11 -1
index a040cde..79e3cec 100644
@@ -1380,7 +1380,17 @@ void bio_check_pages_dirty(struct bio *bio)
  **/
 void bio_endio(struct bio *bio, int error)
 {
-	if (error)
+	/*
+	 * Special case here - hide the -EOPNOTSUPP from the driver or
+	 * block layer, dump a warning the first time this happens so that
+	 * the admin knows that we may not provide the ordering guarantees
+	 * that are needed. Don't clear the uptodate bit.
+	 */
+	if (error == -EOPNOTSUPP && bio_barrier(bio)) {
+		set_bit(BIO_EOPNOTSUPP, &bio->bi_flags);
+		blk_queue_set_noflush(bio->bi_bdev);
+		error = 0;
+	} else if (error)
 		clear_bit(BIO_UPTODATE, &bio->bi_flags);
 	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))
 		error = -EIO;
fs/btrfs/extent_io.c +2 -7
index ebe6b29..d696d26 100644
@@ -1834,7 +1834,6 @@ extent_bio_alloc(struct block_device *bdev, u64 first_sector, int nr_vecs,
 static int submit_one_bio(int rw, struct bio *bio, int mirror_num,
 			  unsigned long bio_flags)
 {
-	int ret = 0;
 	struct bio_vec *bvec = bio->bi_io_vec + bio->bi_vcnt - 1;
 	struct page *page = bvec->bv_page;
 	struct extent_io_tree *tree = bio->bi_private;
@@ -1846,17 +1845,13 @@ static int submit_one_bio(int rw, struct bio *bio, int mirror_num,
 
 	bio->bi_private = NULL;
 
-	bio_get(bio);
-
 	if (tree->ops && tree->ops->submit_bio_hook)
 		tree->ops->submit_bio_hook(page->mapping->host, rw, bio,
 					   mirror_num, bio_flags);
 	else
 		submit_bio(rw, bio);
-	if (bio_flagged(bio, BIO_EOPNOTSUPP))
-		ret = -EOPNOTSUPP;
-	bio_put(bio);
-	return ret;
+
+	return 0;
 }
 
 static int submit_extent_page(int rw, struct extent_io_tree *tree,
fs/buffer.c +2 -21
index a2fd743..6f50e08 100644
@@ -147,17 +147,9 @@ void end_buffer_read_sync(struct buffer_head *bh, int uptodate)
 
 void end_buffer_write_sync(struct buffer_head *bh, int uptodate)
 {
-	char b[BDEVNAME_SIZE];
-
 	if (uptodate) {
 		set_buffer_uptodate(bh);
 	} else {
-		if (!buffer_eopnotsupp(bh) && !quiet_error(bh)) {
-			buffer_io_error(bh);
-			printk(KERN_WARNING "lost page write due to "
-					"I/O error on %s\n",
-				       bdevname(bh->b_bdev, b));
-		}
 		set_buffer_write_io_error(bh);
 		clear_buffer_uptodate(bh);
 	}
@@ -2828,7 +2820,7 @@ static void end_bio_bh_io_sync(struct bio *bio, int err)
 
 	if (err == -EOPNOTSUPP) {
 		set_bit(BIO_EOPNOTSUPP, &bio->bi_flags);
-		set_bit(BH_Eopnotsupp, &bh->b_state);
+		err = 0;
 	}
 
 	if (unlikely (test_bit(BIO_QUIET,&bio->bi_flags)))
@@ -2841,7 +2833,6 @@ static void end_bio_bh_io_sync(struct bio *bio, int err)
 int submit_bh(int rw, struct buffer_head * bh)
 {
 	struct bio *bio;
-	int ret = 0;
 
 	BUG_ON(!buffer_locked(bh));
 	BUG_ON(!buffer_mapped(bh));
@@ -2879,14 +2870,8 @@ int submit_bh(int rw, struct buffer_head * bh)
 	bio->bi_end_io = end_bio_bh_io_sync;
 	bio->bi_private = bh;
 
-	bio_get(bio);
 	submit_bio(rw, bio);
-
-	if (bio_flagged(bio, BIO_EOPNOTSUPP))
-		ret = -EOPNOTSUPP;
-
-	bio_put(bio);
-	return ret;
+	return 0;
 }
 
 /**
@@ -2965,10 +2950,6 @@ int sync_dirty_buffer(struct buffer_head *bh)
 		bh->b_end_io = end_buffer_write_sync;
 		ret = submit_bh(WRITE, bh);
 		wait_on_buffer(bh);
-		if (buffer_eopnotsupp(bh)) {
-			clear_buffer_eopnotsupp(bh);
-			ret = -EOPNOTSUPP;
-		}
 		if (!ret && !buffer_uptodate(bh))
 			ret = -EIO;
 	} else {
fs/xfs/linux-2.6/xfs_aops.c +0 -1
index de3a198..c0cacb2 100644
@@ -406,7 +406,6 @@ xfs_submit_ioend_bio(
 	bio->bi_end_io = xfs_end_bio;
 
 	submit_bio(WRITE, bio);
-	ASSERT(!bio_flagged(bio, BIO_EOPNOTSUPP));
 	bio_put(bio);
 }
 
include/linux/blkdev.h +2 -0
index 465d6ba..ea2e15a 100644
@@ -452,6 +452,7 @@ struct request_queue
 #define QUEUE_FLAG_NONROT      14	/* non-rotational device (SSD) */
 #define QUEUE_FLAG_VIRT        QUEUE_FLAG_NONROT /* paravirt device */
 #define QUEUE_FLAG_IO_STAT     15	/* do IO stats */
+#define QUEUE_FLAG_NOFLUSH     16	/* device doesn't do flushes */
 
 #define QUEUE_FLAG_DEFAULT	((1 << QUEUE_FLAG_IO_STAT) |		\
 				 (1 << QUEUE_FLAG_CLUSTER) |		\
@@ -789,6 +790,7 @@ extern int blk_execute_rq(struct request_queue *, struct gendisk *,
 extern void blk_execute_rq_nowait(struct request_queue *, struct gendisk *,
 				  struct request *, int, rq_end_io_fn *);
 extern void blk_unplug(struct request_queue *q);
+extern void blk_queue_set_noflush(struct block_device *bdev);
 
 static inline struct request_queue *bdev_get_queue(struct block_device *bdev)
 {
include/linux/buffer_head.h +0 -2
index f19fd90..8adcaa4 100644
@@ -33,7 +33,6 @@ enum bh_state_bits {
 	BH_Boundary,	/* Block is followed by a discontiguity */
 	BH_Write_EIO,	/* I/O error on write */
 	BH_Ordered,	/* ordered write */
-	BH_Eopnotsupp,	/* operation not supported (barrier) */
 	BH_Unwritten,	/* Buffer is allocated on disk but not written */
 	BH_Quiet,	/* Buffer Error Prinks to be quiet */
 
@@ -126,7 +125,6 @@ BUFFER_FNS(Delay, delay)
 BUFFER_FNS(Boundary, boundary)
 BUFFER_FNS(Write_EIO, write_io_error)
 BUFFER_FNS(Ordered, ordered)
-BUFFER_FNS(Eopnotsupp, eopnotsupp)
 BUFFER_FNS(Unwritten, unwritten)
 
 #define bh_offset(bh)		((unsigned long)(bh)->b_data & ~PAGE_MASK)
Signature · 1 line
Jens Axboe