Skip to contentSkip to conversation

Change in default vm_dirty_ratio

DISCUSSION Tim Chen 13 participants 29 messages Jun 18, 2007 – Jun 25, 2007

lore
Share & tools
lore

Tim Chen

author
#0 lore raw

Andrew,

The default vm_dirty_ratio changed from 40 to 10
for the 2.6.22-rc kernels in this patch:

http://git.kernel.org/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=07db59bd6b0f279c31044cba6787344f63be87ea;hp=de46c33745f5e2ad594c72f2cf5f490861b16ce1

IOZone write drops by about 60% when test file size is 50 percent of memory. Rand-write drops by 90%.

Is there a good reason for turning down the default dirty ratio? How will it help for most cases? Intuitively, it seems like a less aggressive writeback will have better performance.

Thanks.

Tim

#1 lore raw
Quoted 10 lines · On Mon, 18 Jun 2007 14:14:30 -0700 Tim Chen <[email protected]> wrote:

Andrew,

The default vm_dirty_ratio changed from 40 to 10
for the 2.6.22-rc kernels in this patch:

http://git.kernel.org/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=07db59bd6b0f279c31044cba6787344f63be87ea;hp=de46c33745f5e2ad594c72f2cf5f490861b16ce1

IOZone write drops by about 60% when test file size is 50 percent of memory. Rand-write drops by 90%.

heh.

(Or is that an inappropriate reaction?)

Quoted 1 line

Is there a good reason for turning down the default dirty ratio?

It seems too large. Memory sizes are going up faster than disk throughput and it seems wrong to keep vast amounts of dirty data floating about in memory like this. It can cause long stalls while the system writes back huge amounts of data and is generally ill-behaved.

Quoted 2 lines

How will it help for most cases? Intuitively, it seems like a less aggressive writeback will have better performance.

I assume that iozone is either doing a lot of file overwrites or is unlinking/truncating files shortly after having written them.

And some benchmarks are silly. You have just demonstrated that IOZone should have been called RAMZone....

Some workloads will work more nicely with this change and others will be hurt. Where does the optimum lie? Don't know. Nowhere, really.

Frankly, I find it very depressing that the kernel defaults matter. These things are trivially tunable and you'd think that after all these years, distro initscripts would be establishing the settings, based upon expected workload, amount of memory, number and bandwidth of attached devices, etc.

Heck, there should even be userspace daemons which observe ongoing system behaviour and which adaptively tune these things to the most appropriate level.

But nope, nothing.

#2 lore raw
Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Mon, 18 Jun 2007 14:14:30 -0700 Tim Chen <[email protected]> wrote:

Andrew,

The default vm_dirty_ratio changed from 40 to 10
for the 2.6.22-rc kernels in this patch:

Yup.

Quoted 6 lines
Quoted 2 lines

IOZone write drops by about 60% when test file size is 50 percent of memory. Rand-write drops by 90%.

heh.

(Or is that an inappropriate reaction?)

I think it's probably appropriate.

I don't know what else to say.

For pure write testing, where writeback caching is good, you should probably run all benchmarks with vm_dirty_ratio set as high as possible. That's fairly obvious.

What's equally obvious is that for actual real-life use, such tuning is not a good idea, and setting the vm_dirty_ratio down causes a more pleasant user experience, thanks to smoother IO load behavoiur.

Is it good to keep tons of dirty stuff around? Sure. It allows overwriting (and thus avoiding doing the write in the first place), but it also allows for a more aggressive IO scheduling, in that you have more writes that you can schedule.

It does sound like IOZone just isn't a good benchmark. It doesn't actually measure disk throughput, it really measures how good the OS is at *not* doing the IO. And yes, in that case, set vm_dirty_ratio high to get better numbers.

I'd rather have the defaults at something that is "pleasant", and then make it easy for benchmarkers to put it at something "unpleasant, but gives better numbers". And it's not like it's all that hard to just do

echo 50 > /proc/sys/vm/dirty_ratio

in your /etc/rc.local or something, if you know you want this.

Maybe somebody can make a small graphical config app, and the distros could even skip it? Dunno. I *suspect* very few people actually end up caring.

Linus

#3 lore raw
Quoted 5 lines

Is it good to keep tons of dirty stuff around? Sure. It allows overwriting (and thus avoiding doing the write in the first place), but it also allows for a more aggressive IO scheduling, in that you have more writes that you can schedule.

it also allows for an elevator that can merge more so that there are less seeks...

so it's not all pure artificial ;(

I really don't like doing just-for-benchmark tuning ... but I wonder how much real workloads this will get too (like installing or upgrading a bunch of rpms)

As for the smoother IO thing.. there's already a kernel process that writes this lot out after 5 seconds... so that ought to smooth some of this out already.... I would hope.

(I'm not arguing this change is wrong, I'm just grinding my teeth on how long updating rpms already takes... for no apparent reason)

#4 lore raw
In reply to Andrew Morton
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Andrew" == Andrew Morton <[email protected]> writes:

Andrew> On Mon, 18 Jun 2007 14:14:30 -0700

Quoted 2 lines · Andrew> Tim Chen <[email protected]> wrote:
Quoted 2 lines

IOZone write drops by about 60% when test file size is 50 percent of memory. Rand-write drops by 90%.

Andrew> heh.

Andrew> (Or is that an inappropriate reaction?)

Quoted 1 line
Quoted 1 line

Is there a good reason for turning down the default dirty ratio?

Andrew> It seems too large. Memory sizes are going up faster than Andrew> disk throughput and it seems wrong to keep vast amounts of Andrew> dirty data floating about in memory like this. It can cause Andrew> long stalls while the system writes back huge amounts of data Andrew> and is generally ill-behaved.

Shouldn't the vm_dirty_ratio be based on the speed of the device, and not the size of memory? So slower devices can't keep as much in memory as fast devices? That would seem to be a better metric. And of course those with hundreds of disks will then complain we're taking too much memory as well, even though they can handle it.

So per-device vm_dirty_ratio, capped with a vm_dirty_total_ratio seems to be what we want, right?

John

#8 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton

It seems too large. Memory sizes are going up faster than disk throughput and it seems wrong to keep vast amounts of dirty data floating about in memory like this. It can cause long stalls while the system writes back huge amounts of data and is generally ill-behaved.

A more continuous write out would be better I think. Perhaps the dirty ratio needs to be per address space?

Quoted 3 lines

things are trivially tunable and you'd think that after all these years, distro initscripts would be establishing the settings, based upon expected workload, amount of memory, number and bandwidth of attached devices, etc.

Distro initscripts normally don't have any better clue about any of this than the kernel.

-Andi

#5 lore raw
In reply to John Stoffel
Quoted 3 lines · John Stoffel

Shouldn't the vm_dirty_ratio be based on the speed of the device, and not the size of memory?

Yes. It should depend on:
- speed of the device(s) in question
- seekiness of the workload
- wishes of the user as per the latency of other operations.

However, nobody has ever found the required algorithm.

So "at most 10% of memory dirty" is a simple (and fairly _good_) heuristic. Nobody has actually ever ended up complaining about the change from 40% -> 10%, and as far as I know this was the first report (and it's not so much because the change was bad, but because it showed up on a benchmark - and I don't think that actually says anythign about anything else then the behaviour of the benchmark itself)

So are there better algorithms in theory? Probably lots of them.

Linus

#6 lore raw
Quoted 3 lines · Linus Torvalds

Yes. It should depend on:
- speed of the device(s) in question

Btw, this one can be quite a big deal. Try connecting an iPod and syncing 8GB of data to it. Oops.

So yes, it would be nice to have some per-device logic too. Tested patches would be very welcome ;)

Linus

#7 lore raw
In reply to Linus Torvalds

From: Linus Torvalds <[email protected]>
Date: Tue, 19 Jun 2007 12:04:33 -0700 (PDT)

Quoted 12 lines · John Stoffel
Quoted 3 lines

Shouldn't the vm_dirty_ratio be based on the speed of the device, and not the size of memory?

Yes. It should depend on:
- speed of the device(s) in question
- seekiness of the workload
- wishes of the user as per the latency of other operations.

Quoted 8 lines

However, nobody has ever found the required algorithm.

So "at most 10% of memory dirty" is a simple (and fairly _good_) heuristic. Nobody has actually ever ended up complaining about the change from 40% -> 10%, and as far as I know this was the first report (and it's not so much because the change was bad, but because it showed up on a benchmark - and I don't think that actually says anythign about anything else then the behaviour of the benchmark itself)

I complained very early on that it makes my workstation basically just hang doing disk writes when I do big git tree operations on my sparc64 workstation.

Restoring the old values makes that go away.

It's an arbitrary number because the correct setting is dependant upon what you're doing and how fast your disks are.

So I really think the "only benchmarks like the old settings" argument doesn't really apply.

#9 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

Frankly, I find it very depressing that the kernel defaults matter. These things are trivially tunable and you'd think that after all these years, distro initscripts would be establishing the settings, based upon expected workload, amount of memory, number and bandwidth of attached devices, etc.

"This is hard, lets make it someone else's problem" shouldn't ever be the answer, especially if the end result is that we become even more dependant on bits of userspace running before the system becomes useful.

Quoted 5 lines

Heck, there should even be userspace daemons which observe ongoing system behaviour and which adaptively tune these things to the most appropriate level.

But nope, nothing.

See the 'libtune' crack that people have been trying to get distros to adopt for a long time.
If we need some form of adaptive behaviour, the kernel needs to be doing this monitoring/adapting, not some userspace daemon that may not get scheduled before its too late.

If the kernel can't get the defaults right, what makes you think
userspace can do better ? Just as the kernel can't get
"one size fits all" right, there's no silver bullet just by clicking "this is a database server" button to have it configure random sysctls etc. These things require thought and planning that daemons will never get right in every case. And when they get it wrong, the results can be worse than the stock defaults.

libtune is the latest in a series of attempts to do this dynamic runtime adjustment (hell, I even started such a project myself back circa 2000 which thankfully never really took off). It's a bad idea that just won't die.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#10 lore raw
Quoted 9 lines · On Wed, 20 Jun 2007 00:24:34 -0400 Dave Jones <[email protected]> wrote:
Quoted 4 lines · Andrew Morton

Frankly, I find it very depressing that the kernel defaults matter. These things are trivially tunable and you'd think that after all these years, distro initscripts would be establishing the settings, based upon expected workload, amount of memory, number and bandwidth of attached devices, etc.

"This is hard, lets make it someone else's problem" shouldn't ever be the answer,

Bovine droppings. Nobody has even tried.

Quoted 2 lines

especially if the end result is that we become even more dependant on bits of userspace running before the system becomes useful.

Cattle excreta. The kernel remains as it presently is. No less useful that it is now.

Quoted 11 lines
Quoted 5 lines

Heck, there should even be userspace daemons which observe ongoing system behaviour and which adaptively tune these things to the most appropriate level.

But nope, nothing.

See the 'libtune' crack that people have been trying to get distros to adopt for a long time.
If we need some form of adaptive behaviour, the kernel needs to be doing this monitoring/adapting, not some userspace daemon that may not get scheduled before its too late.

Userspace has just as much info as the kernel has and there is no latency concern here.

Quoted 2 lines

If the kernel can't get the defaults right, what makes you think userspace can do better ?

Because userspace can implement more sophisticated algorithms and is more easily configured.

For example, userspace can take a hotplug event for the just-added usb-storage device then go look up its IO characteristics in a database and then apply that to the confgured policy. If the device was not found, userspace can perform a test run to empirically measure that device's IO characteristics and then record them in the database. I don't think we'll be doing this in-kernel any time soon.

(And to preempt lkml-games: this is just an _example_. There are others)

Quoted 7 lines

Just as the kernel can't get
"one size fits all" right, there's no silver bullet just by clicking "this is a database server" button to have it configure random sysctls etc. These things require thought and planning that daemons will never get right in every case. And when they get it wrong, the results can be worse than the stock defaults.

Quoted 5 lines

libtune is the latest in a series of attempts to do this dynamic runtime adjustment (hell, I even started such a project myself back circa 2000 which thankfully never really took off). It's a bad idea that just won't die.

So libtune is the only possible way of implementing any of this?

If choosing the optimum settings cannot be done in userspace then it sure as heck cannot be done in-kernel.

Anyway, this is all arse-about. What is the design? What algorithms do we need to implement to do this successfully? Answer me that, then we can decide upon these implementation details.

#11 lore raw
Quoted 3 lines · Andrew Morton

Anyway, this is all arse-about. What is the design? What algorithms do we need to implement to do this successfully? Answer me that, then we can decide upon these implementation details.

Building on the per BDI patches, how about integrating feedback from the full-ness of device queues. That is, when we are happily doing IO and we cannot possibly saturate the active devices (as measured by their queue never reaching 75%?) then we can safely increase the total dirty limit.

OTOH, when even with the per BDI dirty limit the device queue is constantly saturated (contended) we ought to lower the total dirty limit.

Lots of detail here to work out, but does this sound workable?

#12 lore raw
Quoted 12 lines · On Wed, 20 Jun 2007 10:35:36 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 3 lines · Andrew Morton

Anyway, this is all arse-about. What is the design? What algorithms do we need to implement to do this successfully? Answer me that, then we can decide upon these implementation details.

Building on the per BDI patches, how about integrating feedback from the full-ness of device queues. That is, when we are happily doing IO and we cannot possibly saturate the active devices (as measured by their queue never reaching 75%?) then we can safely increase the total dirty limit.

Quoted 5 lines

OTOH, when even with the per BDI dirty limit the device queue is constantly saturated (contended) we ought to lower the total dirty limit.

Lots of detail here to work out, but does this sound workable?

It's pretty easy to fill the queues - I'd expect that there are a lot of not-very-heavy workloads which cause the kernel to shove a lot of little writes into the queue when it visits the blockdev mapping: a shower of inodes, directory entries, indirect blocks, etc. With very little dirty memory associated with it.

But back away further.

What do we actually want the kernel to *do*? Stated in terms of "when the dirty memory state is A, do B" and "when userspace does C, the kernel should do D".

Top-level statement: "when userspace does anything, the kernel should not suck" ;) Some refinement is needed there.

I _think_ the problem is basically one of latency: a) writes starving reads and b) dirty memory causing page reclaim to stall and c) inter-device contention on the global memory limits.

Hard. If the device isn't doing anything else then we can shove data at it freely. If reads (or synchronous writes) come in then perhaps the VM should back off and permit dirty memory to go higher.

The anticipatory scheduler(s) are supposed to fix this.

Perhaps our queues are too long - if the VFS _does_ back off, it'll take some time for that to have an effect.

Perhaps the fact that the queue size knows nothing about the _size_ of the requests in the queue is a problem.

Back away even further here.

What user-visible problem(s) are we attemping to fix?

#13 lore raw
Quoted 5 lines · Andrew Morton

Perhaps our queues are too long - if the VFS _does_ back off, it'll take some time for that to have an effect.

Perhaps the fact that the queue size knows nothing about the _size_ of the requests in the queue is a problem.

It's complicated, the size may not matter a lot. 128 sequential 512kb IO may complete faster than 128 random 4kb IO's.

Quoted 3 lines

Back away even further here.

What user-visible problem(s) are we attemping to fix?

I'd like innocent-app-doing-little-write-or-fsync not being stalled by big-bad-app-doing-lots-of-dirtying.

Signature · 1 line
Jens Axboe
#22 lore raw
In reply to Andrew Morton
Quoted 30 lines · Andrew Morton
Quoted 12 lines · On Wed, 20 Jun 2007 10:35:36 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 3 lines · Andrew Morton

Anyway, this is all arse-about. What is the design? What algorithms do we need to implement to do this successfully? Answer me that, then we can decide upon these implementation details.

Building on the per BDI patches, how about integrating feedback from the full-ness of device queues. That is, when we are happily doing IO and we cannot possibly saturate the active devices (as measured by their queue never reaching 75%?) then we can safely increase the total dirty limit.

Quoted 5 lines

OTOH, when even with the per BDI dirty limit the device queue is constantly saturated (contended) we ought to lower the total dirty limit.

Lots of detail here to work out, but does this sound workable?

It's pretty easy to fill the queues - I'd expect that there are a lot of not-very-heavy workloads which cause the kernel to shove a lot of little writes into the queue when it visits the blockdev mapping: a shower of inodes, directory entries, indirect blocks, etc. With very little dirty memory associated with it.

But back away further.

What do we actually want the kernel to *do*? Stated in terms of "when the dirty memory state is A, do B" and "when userspace does C, the kernel should do D".

Quoted 6 lines

Top-level statement: "when userspace does anything, the kernel should not suck" ;) Some refinement is needed there.

I _think_ the problem is basically one of latency: a) writes starving reads and b) dirty memory causing page reclaim to stall and c) inter-device contention on the global memory limits.

well, I hope to have solved c)... :-)

Quoted 5 lines

Hard. If the device isn't doing anything else then we can shove data at it freely. If reads (or synchronous writes) come in then perhaps the VM should back off and permit dirty memory to go higher.

The anticipatory scheduler(s) are supposed to fix this.

I must plead ignorance here, I'll try to fill this hole in my knowledge :-/

Quoted 5 lines

Perhaps our queues are too long - if the VFS _does_ back off, it'll take some time for that to have an effect.

Perhaps the fact that the queue size knows nothing about the _size_ of the requests in the queue is a problem.

Yes this would be an issue. By not knowing that, the queue limit is basically useless. It doesn't limit very much.

Quoted 3 lines

Back away even further here.

What user-visible problem(s) are we attemping to fix?

Good point, the only report so far is DaveM saying git sucked on his sparc64 box...

Dave, do you have any idea what caused that? could it be this global fsync ext3 suffers from?

But the basic problem is balancing under-utilisation of disks vs. keeping too much in memory.

#14 lore raw
In reply to Jens Axboe
Quoted 9 lines · Jens Axboe
Quoted 5 lines · Andrew Morton

Perhaps our queues are too long - if the VFS _does_ back off, it'll take some time for that to have an effect.

Perhaps the fact that the queue size knows nothing about the _size_ of the requests in the queue is a problem.

It's complicated, the size may not matter a lot. 128 sequential 512kb IO may complete faster than 128 random 4kb IO's.

Yes, is there any way a queue could be limited to a certain amount of 'completion time' ?

Quoted 6 lines
Quoted 3 lines

Back away even further here.

What user-visible problem(s) are we attemping to fix?

I'd like innocent-app-doing-little-write-or-fsync not being stalled by big-bad-app-doing-lots-of-dirtying.

Could you please try this per BDI dirty limit -v7 patch series, the very last patch tries to address this by taking the per task dirty rate into account.

Although, on the fsync, ext3 seems to want to do a global fsync, which will still make the experience suck. :-(

#15 lore raw
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Jens Axboe
Quoted 3 lines · Andrew Morton

Perhaps our queues are too long - if the VFS _does_ back off, it'll take some time for that to have an effect.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Perhaps the fact that the queue size knows nothing about the _size_ of the requests in the queue is a problem.

It's complicated, the size may not matter a lot. 128 sequential 512kb IO may complete faster than 128 random 4kb IO's.

Yes, is there any way a queue could be limited to a certain amount of 'completion time' ?

Not easily, we'd need some sort of disk profile for that to be remotely reliable.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

Back away even further here.

What user-visible problem(s) are we attemping to fix?

I'd like innocent-app-doing-little-write-or-fsync not being stalled by big-bad-app-doing-lots-of-dirtying.

Could you please try this per BDI dirty limit -v7 patch series, the very last patch tries to address this by taking the per task dirty rate into account.

Yeah, I've been watching your patchset with interesting. Hope it'll get merged some time soon, I think it's a real problem.

Quoted 2 lines

Although, on the fsync, ext3 seems to want to do a global fsync, which will still make the experience suck. :-(

Yeah well, extX sucks on many levels :-)

Signature · 1 line
Jens Axboe
#16 lore raw
Quoted 9 lines · Jens Axboe
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Jens Axboe
Quoted 5 lines · Andrew Morton

Perhaps our queues are too long - if the VFS _does_ back off, it'll take some time for that to have an effect.

Perhaps the fact that the queue size knows nothing about the _size_ of the requests in the queue is a problem.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

It's complicated, the size may not matter a lot. 128 sequential 512kb IO may complete faster than 128 random 4kb IO's.

Yes, is there any way a queue could be limited to a certain amount of 'completion time' ?

Not easily, we'd need some sort of disk profile for that to be remotely reliable.

Yes, I see the problem, benching the device is hard; you don't want it to do it each time, nor for it to take too long. Also, write performance might be destructive, also not quite wanted :-/

/me sees this libtune doom on the horizon again.

Something adaptive would be best, something that inserts barriers and measures the time to complete and then solves the read speed, write speed and seek latency. All during normal operation.

That would entail storing a bunch of these sample points, solve the equation for sets of 3, and (time-) average the results.. ugh

#25 lore raw
In reply to Peter Zijlstra
Quoted 5 lines · Peter Zijlstra

Building on the per BDI patches, how about integrating feedback from the full-ness of device queues. That is, when we are happily doing IO and we cannot possibly saturate the active devices (as measured by their queue never reaching 75%?) then we can safely increase the total dirty limit.

The really annoying things are the one-off things. You've been happily working for a while (never even being _close_ to saturatign any IO queues), and then you untar a large tree.

If the kernel now let's you dirty lots of memory, you'll have a very unpleasant experience.

And with hot-pluggable devices (which is where most of the throughput problems tend to be!), the "one-off" thing is not a "just after reboot" kind of situation.

So you'd have to be pretty smart about it.

Linus

#26 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · Peter Zijlstra

Building on the per BDI patches, how about integrating feedback from the full-ness of device queues. That is, when we are happily doing IO and we cannot possibly saturate the active devices (as measured by their queue never reaching 75%?) then we can safely increase the total dirty limit.

Quoted 6 lines

The really annoying things are the one-off things. You've been happily working for a while (never even being _close_ to saturatign any IO queues), and then you untar a large tree.

If the kernel now let's you dirty lots of memory, you'll have a very unpleasant experience.

maybe that needs to be fixed? If you stopped dirtying after the initial bump.. is there a reason for the kernel to dump all that data to the disk in such a way that it disturbs interactive users?

so the question maybe is.. is the vm tunable the cause or the symptom of the bad experience?

#27 lore raw
Quoted 4 lines · Arjan van de Ven

maybe that needs to be fixed? If you stopped dirtying after the initial bump.. is there a reason for the kernel to dump all that data to the disk in such a way that it disturbs interactive users?

No. I would argue that the kernel should try to trickle things out, so that it doesn't disturb anything, and a "big dump" becomes a "steady trickle".

And that's what "vm_dirty_ratio" is all about.

Quoted 2 lines

so the question maybe is.. is the vm tunable the cause or the symptom of the bad experience?

No, the vm tunable is exactly what it's all about.

Do a big "untar", and what you *want* to see is not "instant dump, followed by long pause".

A much *smoother* behaviour is generally preferable, and most of the time that's true even if it may be lower throughput in the end!

Of course, "synchronous writes" are *really* smooth (you never allow any dumps at *all* to build up), so this is about a balance - not about "perfect smoothness" vs "best throughput", but about a heuristic that finds a reasonable middle ground.

There is no "perfect". There is only "stupid heuristics". Maybe the "vm_dirty_ratio" is a bit *too* stupid, but it definitely is needed in some form.

It can actually be more than just a "performance" vs "smoothness" issue:
the 40% thing was actually a *correctness* issue too, back when we coutned it as a percentage of total memory. A highmem machine would allow 40% of all memory free and it was all in low memory, and that literally caused lockups.

So the dirty_ratio is not *only* about smoothness, it's also simply about the fact that the kernel must not allow too much memory to be dirtied, because that leads to out-of-memory deadlocks and other nasty issues. So it's not *purely* a tunable.

Linus

#28 lore raw
In reply to Dave Jones
Quoted 15 lines · Dave Jones
Quoted 4 lines · Andrew Morton

Frankly, I find it very depressing that the kernel defaults matter. These things are trivially tunable and you'd think that after all these years, distro initscripts would be establishing the settings, based upon expected workload, amount of memory, number and bandwidth of attached devices, etc.

"This is hard, lets make it someone else's problem" shouldn't ever be the answer, especially if the end result is that we become even more dependant on bits of userspace running before the system becomes useful.

Quoted 4 lines

Heck, there should even be userspace daemons which observe ongoing system behaviour and which adaptively tune these things to the most appropriate level.

Quoted 8 lines
Quoted 1 line

But nope, nothing.

See the 'libtune' crack that people have been trying to get distros to adopt for a long time.
If we need some form of adaptive behaviour, the kernel needs to be doing this monitoring/adapting, not some userspace daemon that may not get scheduled before its too late.

I'm wondering whether AKT I proposed a couple of months ago wouldn't be more appropriate (provided that we find the perfect heuristics to tune the dirty_ratio ;-) ) see thread http://lkml.org/lkml/2007/1/16/16

Regards,
Nadia

#23 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

What do we actually want the kernel to *do*? Stated in terms of "when the dirty memory state is A, do B" and "when userspace does C, the kernel should do D".

When we have dirty pages awaiting write-out,
and the write-out device is completely idle,
then we should be writing them out.

That's the easy bit taken care of. ;)

#24 lore raw
Quoted 11 lines · Mark Lord
Quoted 4 lines · Andrew Morton

What do we actually want the kernel to *do*? Stated in terms of "when the dirty memory state is A, do B" and "when userspace does C, the kernel should do D".

When we have dirty pages awaiting write-out,
and the write-out device is completely idle,
then we should be writing them out.

That's the easy bit taken care of. ;)

Unless we're in laptop mode, in that case we want to procrastinate.. :-)

#17 lore raw
In reply to Jens Axboe
Quoted 9 lines · Jens Axboe
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Jens Axboe
Quoted 5 lines · Andrew Morton

Perhaps our queues are too long - if the VFS _does_ back off, it'll take some time for that to have an effect.

Perhaps the fact that the queue size knows nothing about the _size_ of the requests in the queue is a problem.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

It's complicated, the size may not matter a lot. 128 sequential 512kb IO may complete faster than 128 random 4kb IO's.

Yes, is there any way a queue could be limited to a certain amount of 'completion time' ?

Not easily, we'd need some sort of disk profile for that to be remotely reliable.

Perhaps we want to throw some sliding window algorithms at it. We can bound requests and total I/O and if requests get retired too slowly we can shrink the windows. Alternately, we can grow the window if we're retiring things within our desired timeframe.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#18 lore raw
Quoted 5 lines · Matt Mackall

Perhaps we want to throw some sliding window algorithms at it. We can bound requests and total I/O and if requests get retired too slowly we can shrink the windows. Alternately, we can grow the window if we're retiring things within our desired timeframe.

I suspect that would tend to be a good way to go. But it almost certainly has to be per-device, which implies that somebody would have to do some major coding/testing on this..

The vm_dirty_ratio thing is a global value, and I think we need that regardless (for the independent issue of memory deadlocks etc), but if we *additionally* had a per-device throttle that was based on some kind of adaptive thing, we could probably raise the global (hard) vm_dirty_ratio a lot.

Linus

#19 lore raw
Quoted 12 lines · Linus Torvalds
Quoted 5 lines · Matt Mackall

Perhaps we want to throw some sliding window algorithms at it. We can bound requests and total I/O and if requests get retired too slowly we can shrink the windows. Alternately, we can grow the window if we're retiring things within our desired timeframe.

I suspect that would tend to be a good way to go. But it almost certainly has to be per-device, which implies that somebody would have to do some major coding/testing on this..

Quoted 5 lines

The vm_dirty_ratio thing is a global value, and I think we need that regardless (for the independent issue of memory deadlocks etc), but if we *additionally* had a per-device throttle that was based on some kind of adaptive thing, we could probably raise the global (hard) vm_dirty_ratio a lot.

I just did quite a bit of that:

http://lkml.org/lkml/2007/6/14/437

#20 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Linus Torvalds

The vm_dirty_ratio thing is a global value, and I think we need that regardless (for the independent issue of memory deadlocks etc), but if we *additionally* had a per-device throttle that was based on some kind of adaptive thing, we could probably raise the global (hard) vm_dirty_ratio a lot.

I just did quite a bit of that:

http://lkml.org/lkml/2007/6/14/437

Ok, that does look interesting.

A few comments:

- Cosmetic: please please *please* don't do this:

	-	if (atomic_long_dec_return(&nfss->writeback) < NFS_CONGESTION_OFF_THRESH) {
	+	if (atomic_long_dec_return(&nfss->writeback) <
	+			NFS_CONGESTION_OFF_THRESH)

we had a discussion about this not that long ago, and it drives me wild to see people split lines like that. It used to be readable. Now it's not.

If it's the checkpatch.pl thing that caused you to split it like that, I think we should just change the value where we start complaining. Maybe set it at 95 characters per line or something.

- I appreciate the extensive comments on floating proportions, and the code looks really quite clean (apart from the cosmetic thing about) from a quick look-through.

HOWEVER. It does seem to be a bit of an overkill. Do we really need to be quite that clever, and do we really need to do 64-bit calculations for this? The 64-bit ops in particular seem quite iffy: if we ever actually pass in an amount that doesn't fit in 32 bits, we'll turn those per-cpu counters into totally synchronous global counters, which seems to defeat the whole point of them. So I'm a bit taken aback by that whole "mod64" thing

(I also hate the naming. I don't think "..._mod()" was a good name to begin with: "mod" means "modulus" to me, not "modify". Making it be called "mod64" just makes it even *worse*, since it's now _obviously_ about modulus - but isn't)

So I'd appreciate some more explanations, but I'd also appreciate some renaming of those functions. What used to be pretty bad naming just turned *really* bad, imnsho.

Linus

#21 lore raw
Quoted 25 lines · Linus Torvalds
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Linus Torvalds

The vm_dirty_ratio thing is a global value, and I think we need that regardless (for the independent issue of memory deadlocks etc), but if we *additionally* had a per-device throttle that was based on some kind of adaptive thing, we could probably raise the global (hard) vm_dirty_ratio a lot.

I just did quite a bit of that:

http://lkml.org/lkml/2007/6/14/437

Ok, that does look interesting.

A few comments:

- Cosmetic: please please *please* don't do this:

	-	if (atomic_long_dec_return(&nfss->writeback) < NFS_CONGESTION_OFF_THRESH) {
	+	if (atomic_long_dec_return(&nfss->writeback) <
	+			NFS_CONGESTION_OFF_THRESH)
Quoted 7 lines

we had a discussion about this not that long ago, and it drives me wild to see people split lines like that. It used to be readable. Now it's not.

If it's the checkpatch.pl thing that caused you to split it like that, I think we should just change the value where we start complaining. Maybe set it at 95 characters per line or something.

It was.

Quoted 6 lines

- I appreciate the extensive comments on floating proportions, and the code looks really quite clean (apart from the cosmetic thing about) from a quick look-through.

HOWEVER. It does seem to be a bit of an overkill. Do we really need to be quite that clever,

Hehe, it is the simplest thing I could come up with. It is deterministic, fast and has a nice physical model :-)

Quoted 2 lines

and do we really need to do 64-bit calculations
for this?

No we don't (well, on 64bit arches we do). I actually only use unsigned long, and even cast whatever comes out of the percpu_counter thing to unsigned long.

Quoted 5 lines

The 64-bit ops in particular seem quite iffy: if we ever
actually pass in an amount that doesn't fit in 32 bits, we'll turn those per-cpu counters into totally synchronous global counters, which seems to defeat the whole point of them. So I'm a bit taken aback by that whole "mod64" thing

That is only needed on 64bit arches, and even there, actually encountering such large values will be rare at best.

Also, this re-normalisation event that uses the call is low frequency. That is, that part will be used once every ~ total_dirty_limit/nr_bdis written out.

Quoted 4 lines

(I also hate the naming. I don't think "..._mod()" was a good name to begin with: "mod" means "modulus" to me, not "modify". Making it be called "mod64" just makes it even *worse*, since it's now _obviously_ about modulus - but isn't)

Agreed.

Quoted 3 lines

So I'd appreciate some more explanations, but I'd also appreciate some renaming of those functions. What used to be pretty bad naming just turned *really* bad, imnsho.

It all just stems from Andrew asking if I could please re-use something instread of duplication a lot of things. I picked percpu_counter because that was the closest to what was needed. An unsigned long based per-cpu counter would suit better.

There is another problem I have with this percpu_counter, it is rather space hungry. It does a node affine sizeof(s32) kalloc on each cpu. Which will end up using the smallest slab, and that is quite a bit bigger than needed. But should be about the size of a cacheline (otherwise we might still end up with false sharing).

I've been thinking of extending this per cpu allocator thing a bit to be a little smarter about these things. What would be needed is a strict per-cpu slab allocator. The current ones are node affine, which can still cause false sharing (unless - as should be the case - these objects are both cacheline aligned and of cacheline size). When we have that, we can start using smaller objects.