A unresponsive file system can hang all I/O in the system on linux-2.6.23-rc6 (dirty_thresh problem?)

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

There is plenty of HIGH MEMORY available in the system, but this process never progresses.

...

The problem seems to be in balance_dirty_pages, which calculates dirty_thresh based on only ZONE_NORMAL. The same scenario works fine in 2.4. The dd processes finishes in no time.
NFS file systems can go offline, due to multiple reasons, a failed switch, filer etc, but that should not effect other file systems in the machine.
Can this behavior be fenced?, can the buffer cache be tuned so that other processes do not see the effect?

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Quoted 3 lines · Andrew Morton

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Nasty problem, don't do that :-)

But yeah, with per BDI dirty limits we get stuck at whatever ratio that NFS server/mount (?) has - which could be 100%. Other processes will then work almost synchronously against their BDIs but it should work.

[ They will lower the NFS-BDI's ratio, but some fancy clipping code will
limit the other BDIs their dirty limit to not exceed the total limit. And with all these NFS pages stuck, that will still be nothing. ]

Quoted 8 lines · Peter Zijlstra

Quoted 3 lines · Andrew Morton

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Nasty problem, don't do that :-)

Quoted 8 lines

But yeah, with per BDI dirty limits we get stuck at whatever ratio that NFS server/mount (?) has - which could be 100%. Other processes will then work almost synchronously against their BDIs but it should work.

[ They will lower the NFS-BDI's ratio, but some fancy clipping code will
limit the other BDIs their dirty limit to not exceed the total limit. And with all these NFS pages stuck, that will still be nothing. ]

Thanks.

The BDI dirty limits sounds like a good idea.

Is there already a patch for this, which I could try?

I believe it works like this,

Each BDI, will have a limit. If the dirty_thresh exceeds the limit, all the I/O on the block device will be synchronous.

so, if I have sda & a NFS mount, the dirty limit can be different for each of them.

I can set dirty limit for
- sda to be 90% and
- NFS mount to be 50%.

So, if the dirty limit is greater than 50%, NFS does synchronously, but sda can work asynchronously, till dirty limit reaches 90%.

However other devices will at that moment try to maintain a limit of 0, which ends up being similar to a sync mount.

So they'll not get stuck, but they will be slow.

[ please don't top-post! ]

Quoted 17 lines · Chakri n

Quoted 8 lines · Peter Zijlstra

Quoted 3 lines · Andrew Morton

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Nasty problem, don't do that :-)

Quoted 8 lines

But yeah, with per BDI dirty limits we get stuck at whatever ratio that NFS server/mount (?) has - which could be 100%. Other processes will then work almost synchronously against their BDIs but it should work.

[ They will lower the NFS-BDI's ratio, but some fancy clipping code will
limit the other BDIs their dirty limit to not exceed the total limit. And with all these NFS pages stuck, that will still be nothing. ]

Quoted 5 lines

Thanks.

The BDI dirty limits sounds like a good idea.

Is there already a patch for this, which I could try?

v2.6.23-rc8-mm2

Quoted 8 lines

I believe it works like this,

Each BDI, will have a limit. If the dirty_thresh exceeds the limit, all the I/O on the block device will be synchronous.

so, if I have sda & a NFS mount, the dirty limit can be different for each of them.

Quoted 6 lines

I can set dirty limit for
- sda to be 90% and
- NFS mount to be 50%.

So, if the dirty limit is greater than 50%, NFS does synchronously, but sda can work asynchronously, till dirty limit reaches 90%.

Not quite, the system determines the limit itself in an adaptive fashion.

bdi_limit = total_limit * p_bdi

Where p is a faction [0,1], and is determined by the relative writeout speed of the current BDI vs all other BDIs.

So if you were to have 3 BDIs (sda, sdb and 1 nfs mount), and sda is idle, and the nfs mount gets twice as much traffic as sdb, the ratios will look like:

p_sda: 0
p_sdb: 1/3
p_nfs: 2/3

Once the traffic exceeds the write speed of the device we build up a backlog and stuff gets throttled, so these proportions converge to the relative write speed of the BDIs when saturated with data.

So what can happen in your case is that the NFS mount is the only one with traffic is will get a fraction of 1. If it then disconnects like in your case, it will still have all of the dirty limit pinned for NFS.

However other devices will at that moment try to maintain a limit of 0, which ends up being similar to a sync mount.

So they'll not get stuck, but they will be slow.

Thanks for explaining the adaptive logic.

Quoted 6 lines

However other devices will at that moment try to maintain a limit of 0, which ends up being similar to a sync mount.

So they'll not get stuck, but they will be slow.

Sync should be ok, when the situation is bad like this and some one hijacked all the buffers.

But, I see my simple dd to write 10blocks on local disk never completes even after 10 minutes.

[root@h46 ~]# dd if=/dev/zero of=/tmp/x count=10

I think the process is completely stuck and is not progressing at all.

Is something going wrong in the calculations where it does not fall back to sync mode.

Quoted 15 lines · Chakri n

Thanks for explaining the adaptive logic.

Quoted 6 lines

However other devices will at that moment try to maintain a limit of 0, which ends up being similar to a sync mount.

So they'll not get stuck, but they will be slow.

Sync should be ok, when the situation is bad like this and some one hijacked all the buffers.

But, I see my simple dd to write 10blocks on local disk never completes even after 10 minutes.

Quoted 6 lines

[root@h46 ~]# dd if=/dev/zero of=/tmp/x count=10

I think the process is completely stuck and is not progressing at all.

Is something going wrong in the calculations where it does not fall back to sync mode.

What kernel is that?

It's 2.6.23-rc6.

It's 2.6.23-rc6.

[ and one copy for the list too ]

Quoted 1 line · Chakri n

It's 2.6.23-rc6.

Could you try .23-rc8-mm2. It includes the per bdi stuff.

[ and one copy for the list too ]

Quoted 1 line · Chakri n

It's 2.6.23-rc6.

Could you try .23-rc8-mm2. It includes the per bdi stuff.

Quoted 15 lines · Chakri n

Thanks for explaining the adaptive logic.

Quoted 6 lines

However other devices will at that moment try to maintain a limit of 0, which ends up being similar to a sync mount.

So they'll not get stuck, but they will be slow.

Sync should be ok, when the situation is bad like this and some one hijacked all the buffers.

But, I see my simple dd to write 10blocks on local disk never completes even after 10 minutes.

Quoted 6 lines

[root@h46 ~]# dd if=/dev/zero of=/tmp/x count=10

I think the process is completely stuck and is not progressing at all.

Is something going wrong in the calculations where it does not fall back to sync mode.

What kernel is that?

Thanks for explaining the adaptive logic.

Quoted 6 lines

However other devices will at that moment try to maintain a limit of 0, which ends up being similar to a sync mount.

So they'll not get stuck, but they will be slow.

Sync should be ok, when the situation is bad like this and some one hijacked all the buffers.

But, I see my simple dd to write 10blocks on local disk never completes even after 10 minutes.

[root@h46 ~]# dd if=/dev/zero of=/tmp/x count=10

I think the process is completely stuck and is not progressing at all.

Is something going wrong in the calculations where it does not fall back to sync mode.

However other devices will at that moment try to maintain a limit of 0, which ends up being similar to a sync mount.

So they'll not get stuck, but they will be slow.

[ please don't top-post! ]

Quoted 17 lines · Chakri n

Quoted 8 lines · Peter Zijlstra

Quoted 3 lines · Andrew Morton

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Nasty problem, don't do that :-)

Quoted 8 lines

But yeah, with per BDI dirty limits we get stuck at whatever ratio that NFS server/mount (?) has - which could be 100%. Other processes will then work almost synchronously against their BDIs but it should work.

[ They will lower the NFS-BDI's ratio, but some fancy clipping code will
limit the other BDIs their dirty limit to not exceed the total limit. And with all these NFS pages stuck, that will still be nothing. ]

Quoted 5 lines

Thanks.

The BDI dirty limits sounds like a good idea.

Is there already a patch for this, which I could try?

v2.6.23-rc8-mm2

Quoted 8 lines

I believe it works like this,

Each BDI, will have a limit. If the dirty_thresh exceeds the limit, all the I/O on the block device will be synchronous.

so, if I have sda & a NFS mount, the dirty limit can be different for each of them.

Quoted 6 lines

I can set dirty limit for
- sda to be 90% and
- NFS mount to be 50%.

So, if the dirty limit is greater than 50%, NFS does synchronously, but sda can work asynchronously, till dirty limit reaches 90%.

Not quite, the system determines the limit itself in an adaptive fashion.

bdi_limit = total_limit * p_bdi

Where p is a faction [0,1], and is determined by the relative writeout speed of the current BDI vs all other BDIs.

So if you were to have 3 BDIs (sda, sdb and 1 nfs mount), and sda is idle, and the nfs mount gets twice as much traffic as sdb, the ratios will look like:

p_sda: 0
p_sdb: 1/3
p_nfs: 2/3

Once the traffic exceeds the write speed of the device we build up a backlog and stuff gets throttled, so these proportions converge to the relative write speed of the BDIs when saturated with data.

So what can happen in your case is that the NFS mount is the only one with traffic is will get a fraction of 1. If it then disconnects like in your case, it will still have all of the dirty limit pinned for NFS.

However other devices will at that moment try to maintain a limit of 0, which ends up being similar to a sync mount.

So they'll not get stuck, but they will be slow.

Quoted 8 lines · Peter Zijlstra

Quoted 3 lines · Andrew Morton

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Nasty problem, don't do that :-)

Quoted 8 lines

But yeah, with per BDI dirty limits we get stuck at whatever ratio that NFS server/mount (?) has - which could be 100%. Other processes will then work almost synchronously against their BDIs but it should work.

[ They will lower the NFS-BDI's ratio, but some fancy clipping code will
limit the other BDIs their dirty limit to not exceed the total limit. And with all these NFS pages stuck, that will still be nothing. ]

Thanks.

The BDI dirty limits sounds like a good idea.

Is there already a patch for this, which I could try?

I believe it works like this,

Each BDI, will have a limit. If the dirty_thresh exceeds the limit, all the I/O on the block device will be synchronous.

so, if I have sda & a NFS mount, the dirty limit can be different for each of them.

I can set dirty limit for
- sda to be 90% and
- NFS mount to be 50%.

So, if the dirty limit is greater than 50%, NFS does synchronously, but sda can work asynchronously, till dirty limit reaches 90%.

Quoted 3 lines · Andrew Morton

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Nasty problem, don't do that :-)

But yeah, with per BDI dirty limits we get stuck at whatever ratio that NFS server/mount (?) has - which could be 100%. Other processes will then work almost synchronously against their BDIs but it should work.

[ They will lower the NFS-BDI's ratio, but some fancy clipping code will
limit the other BDIs their dirty limit to not exceed the total limit. And with all these NFS pages stuck, that will still be nothing. ]

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

It's unrelated to the actual value of dirty_thresh: if the machine fills up with dirty (or unstable) NFS pages then eventually new writers will block until that condition clears.

2.4 doesn't have this problem at low levels of dirty data because 2.4 VFS/MM doesn't account for NFS pages at all.

It's unrelated to the actual value of dirty_thresh: if the machine fills up with dirty (or unstable) NFS pages then eventually new writers will block until that condition clears.

2.4 doesn't have this problem at low levels of dirty data because 2.4 VFS/MM doesn't account for NFS pages at all.

Quoted 6 lines · Andrew

It's unrelated to the actual value of dirty_thresh: if the machine fills up with dirty (or unstable) NFS pages then eventually new writers will block until that condition clears.

2.4 doesn't have this problem at low levels of dirty data because 2.4 VFS/MM doesn't account for NFS pages at all.

Is it really NFS-related? I was trying to back up my 2.6.23-rc8 system to an external USB drive the other day when something flaked and the drive fell off the bus. That, too, was sufficient to wedge the entire system, even though the only thing which needed the dead drive was one rsync process. It's kind of a bummer to have to hit the reset button after the failure of (what should be) a non-critical piece of hardware.

Not that I have a fix to propose...:)

Quoted 7 lines · Jonathan Corbet

Is it really NFS-related? I was trying to back up my 2.6.23-rc8 system to an external USB drive the other day when something flaked and the drive fell off the bus. That, too, was sufficient to wedge the entire system, even though the only thing which needed the dead drive was one rsync process. It's kind of a bummer to have to hit the reset button after the failure of (what should be) a non-critical piece of hardware.

Quoted 1 line

Not that I have a fix to propose...:)

the per bdi work in -mm should make the system not drop dead.

Still, would a remove,re-insert of the usb media end up with the same bdi? That is, would it recognise as the same and resume the transfer.

Quoted 7 lines · Jonathan Corbet

Is it really NFS-related? I was trying to back up my 2.6.23-rc8 system to an external USB drive the other day when something flaked and the drive fell off the bus. That, too, was sufficient to wedge the entire system, even though the only thing which needed the dead drive was one rsync process. It's kind of a bummer to have to hit the reset button after the failure of (what should be) a non-critical piece of hardware.

Quoted 1 line

Not that I have a fix to propose...:)

the per bdi work in -mm should make the system not drop dead.

Still, would a remove,re-insert of the usb media end up with the same bdi? That is, would it recognise as the same and resume the transfer.

,,,it would be grand (and dangerous) if we could provide for a button that would just kill off all outstanding pages against a dead device.

,,,it would be grand (and dangerous) if we could provide for a button that would just kill off all outstanding pages against a dead device.

Quoted 6 lines · Andrew

It's unrelated to the actual value of dirty_thresh: if the machine fills up with dirty (or unstable) NFS pages then eventually new writers will block until that condition clears.

2.4 doesn't have this problem at low levels of dirty data because 2.4 VFS/MM doesn't account for NFS pages at all.

Is it really NFS-related? I was trying to back up my 2.6.23-rc8 system to an external USB drive the other day when something flaked and the drive fell off the bus. That, too, was sufficient to wedge the entire system, even though the only thing which needed the dead drive was one rsync process. It's kind of a bummer to have to hit the reset button after the failure of (what should be) a non-critical piece of hardware.

Not that I have a fix to propose...:)

Quoted 6 lines · Andrew

It's unrelated to the actual value of dirty_thresh: if the machine fills up with dirty (or unstable) NFS pages then eventually new writers will block until that condition clears.

2.4 doesn't have this problem at low levels of dirty data because 2.4 VFS/MM doesn't account for NFS pages at all.

Is it really NFS-related? I was trying to back up my 2.6.23-rc8 system to an external USB drive the other day when something flaked and the drive fell off the bus. That, too, was sufficient to wedge the entire system, even though the only thing which needed the dead drive was one rsync process. It's kind of a bummer to have to hit the reset button after the failure of (what should be) a non-critical piece of hardware.

Not that I have a fix to propose...:)

Actually we perhaps could address this at the VFS level in another way. Processes which are writing to the dead NFS server will eventually block in balance_dirty_pages() once they've exceeded the memory limits and will remain blocked until the server wakes up - that's the behaviour we want.

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Quoted 5 lines · Andrew Morton

Actually we perhaps could address this at the VFS level in another way. Processes which are writing to the dead NFS server will eventually block in balance_dirty_pages() once they've exceeded the memory limits and will remain blocked until the server wakes up - that's the behaviour we want.

Quoted 3 lines

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Do these patches also cause the memory reclaimers to steer clear of devices that are congested (and stop waiting on a congested device if they see that it remains congested for a long period of time)? Most of the collateral blocking I see tends to happen in memory allocation...

Do you know where the stalls are occurring? throttle_vm_writeout(), or via direct calls to congestion_wait() from page_alloc.c and vmscan.c? (running sysrq-w five or ten times will probably be enough to determine this)

Quoted 3 lines · Andrew Morton

Do you know where the stalls are occurring? throttle_vm_writeout(), or via direct calls to congestion_wait() from page_alloc.c and vmscan.c? (running sysrq-w five or ten times will probably be enough to determine this)

would it make sense to instrument congestion_wait() callsites with vmstats?

Quoted 3 lines · Andrew Morton

Do you know where the stalls are occurring? throttle_vm_writeout(), or via direct calls to congestion_wait() from page_alloc.c and vmscan.c? (running sysrq-w five or ten times will probably be enough to determine this)

would it make sense to instrument congestion_wait() callsites with vmstats?

(Searches for the lockstat documentation)

Did we forget to do that?

Quoted 3 lines · Andrew Morton

(Searches for the lockstat documentation)

Did we forget to do that?

yeah,...

/me quickly whips up something

Signed-off-by Peter Zijlstra <[email protected]>

---
 Documentation/lockstat.txt |  119 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 119 insertions(+)

Index: linux-2.6/Documentation/lockstat.txt
===================================================================

Documentation/lockstat.txt new file +36 -0

@@ -0,0 +1,119 @@
+
+LOCK STATISTICS
+
+- WHAT
+
+As the name suggests, it provides statistics on locks.
+
+- WHY
+
+Because things like lock contention can severely impact performance.
+
+- HOW
+
+Lockdep already has hooks in the lock functions and maps lock instances to
+lock classes. We build on that. The graph below shows the relation between
+the lock functions and the various hooks therein.
+
+        __acquire
+            |
+           lock _____
+            |        \
+            |    __contended
+            |         |
+            |       <wait>
+            | _______/
+            |/
+            |
+       __acquired
+            |
+            .
+          <hold>
+            .
+            |
+       __release
+            |
+         unlock

Documentation/lockstat.txt new file +25 -0

+           max		- longest time we ever held the lock
+           total	- total time this lock was held
+
+From these number various other statistics can be derived, such as:
+
+ hold time average = hold time total / acquisitions
+
+These numbers are gathered per lock class, per read/write state (when
+applicable).
+
+It also tracks (4) contention points per class. A contention point is a call
+site that had to wait on lock acquisition.
+
+ - USAGE
+
+Look at the current lock statistics:
+
+(line numbers not part of actual output, done for clarity in the explanation below)
+
+# less /proc/lock_stat
+
+01 lock_stat version 0.2
+02 -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+03                               class name    con-bounces    contentions   waittime-min   waittime-max waittime-total    acq-bounces   acquisitions   holdtime-min   holdtime-max holdtime-total
+04 -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Thanks Randy!

update patch below.

---
Subject: lockstat: documentation

Provide some documentation for CONFIG_LOCK_STAT

Signed-off-by Peter Zijlstra <[email protected]>

Do you know where the stalls are occurring? throttle_vm_writeout(), or via direct calls to congestion_wait() from page_alloc.c and vmscan.c? (running sysrq-w five or ten times will probably be enough to determine this)

Quoted 5 lines · On Fri, 28 Sep 2007 13:00:53 -0400 Trond Myklebust…

Do these patches also cause the memory reclaimers to steer clear of devices that are congested (and stop waiting on a congested device if they see that it remains congested for a long period of time)? Most of the collateral blocking I see tends to happen in memory allocation...

No, they don't attempt to do that, but I suspect they put in place infrastructure which could be used to improve direct-reclaimer latency. In the throttle_vm_writeout() path, at least.

Do you know where the stalls are occurring? throttle_vm_writeout(), or via direct calls to congestion_wait() from page_alloc.c and vmscan.c? (running sysrq-w five or ten times will probably be enough to determine this)

Quoted 11 lines · Andrew Morton

Quoted 5 lines · On Fri, 28 Sep 2007 13:00:53 -0400 Trond Myklebust…

Do these patches also cause the memory reclaimers to steer clear of devices that are congested (and stop waiting on a congested device if they see that it remains congested for a long period of time)? Most of the collateral blocking I see tends to happen in memory allocation...

No, they don't attempt to do that, but I suspect they put in place infrastructure which could be used to improve direct-reclaimer latency. In the throttle_vm_writeout() path, at least.

Quoted 3 lines

Do you know where the stalls are occurring? throttle_vm_writeout(), or via direct calls to congestion_wait() from page_alloc.c and vmscan.c? (running sysrq-w five or ten times will probably be enough to determine this)

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 11 lines · Andrew Morton

Quoted 5 lines · On Fri, 28 Sep 2007 13:00:53 -0400 Trond Myklebust…

Do these patches also cause the memory reclaimers to steer clear of devices that are congested (and stop waiting on a congested device if they see that it remains congested for a long period of time)? Most of the collateral blocking I see tends to happen in memory allocation...

No, they don't attempt to do that, but I suspect they put in place infrastructure which could be used to improve direct-reclaimer latency. In the throttle_vm_writeout() path, at least.

Quoted 3 lines

Do you know where the stalls are occurring? throttle_vm_writeout(), or via direct calls to congestion_wait() from page_alloc.c and vmscan.c? (running sysrq-w five or ten times will probably be enough to determine this)

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

that one is nfs-on-loopback, which is a special case, isn't it?

NFS on loopback used to hang, but then we fixed it. It looks like we broke it again sometime in the intervening four years or so.

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

that one is nfs-on-loopback, which is a special case, isn't it?

NFS on loopback used to hang, but then we fixed it. It looks like we broke it again sometime in the intervening four years or so.

Quoted 5 lines · Andrew Morton

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 1 line

that one is nfs-on-loopback, which is a special case, isn't it?

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

Quoted 5 lines · Andrew Morton

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 1 line

that one is nfs-on-loopback, which is a special case, isn't it?

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

Quoted 6 lines · On Fri, 28 Sep 2007 15:52:28 -0400 Trond Myklebust…

Quoted 5 lines · Andrew Morton

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 8 lines

Quoted 1 line

that one is nfs-on-loopback, which is a special case, isn't it?

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

hm, so ext3 got stuck in nfs via __alloc_pages direct reclaim?

We should be able to fix that by marking the backing device as write-congested. That'll have small race windows, but it should be a 99.9% fix?

Quoted 6 lines · On Fri, 28 Sep 2007 15:52:28 -0400 Trond Myklebust…

Quoted 5 lines · Andrew Morton

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 8 lines

Quoted 1 line

that one is nfs-on-loopback, which is a special case, isn't it?

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

hm, so ext3 got stuck in nfs via __alloc_pages direct reclaim?

We should be able to fix that by marking the backing device as write-congested. That'll have small race windows, but it should be a 99.9% fix?

Quoted 18 lines · Andrew Morton

Quoted 6 lines · On Fri, 28 Sep 2007 15:52:28 -0400 Trond Myklebust…

Quoted 5 lines · Andrew Morton

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 8 lines

Quoted 1 line

that one is nfs-on-loopback, which is a special case, isn't it?

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

Quoted 5 lines

hm, so ext3 got stuck in nfs via __alloc_pages direct reclaim?

We should be able to fix that by marking the backing device as write-congested. That'll have small race windows, but it should be a 99.9% fix?

No. The problem would rather appear to be that we're doing per-backing_dev writeback (if I read sync_sb_inodes() correctly), but we're measuring variables which are global to the VM. The backing device that we are selecting may not be writing out any dirty pages, in which case, we're just spinning in balance_dirty_pages_ratelimited().

Should we therefore perhaps be looking at adding per-backing_dev stats too?

Quoted 25 lines · On Fri, 28 Sep 2007 16:32:18 -0400 Trond Myklebust…

Quoted 18 lines · Andrew Morton

Quoted 6 lines · On Fri, 28 Sep 2007 15:52:28 -0400 Trond Myklebust…

Quoted 5 lines · Andrew Morton

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 8 lines

Quoted 1 line

that one is nfs-on-loopback, which is a special case, isn't it?

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

Quoted 5 lines

hm, so ext3 got stuck in nfs via __alloc_pages direct reclaim?

We should be able to fix that by marking the backing device as write-congested. That'll have small race windows, but it should be a 99.9% fix?

Quoted 5 lines

No. The problem would rather appear to be that we're doing per-backing_dev writeback (if I read sync_sb_inodes() correctly), but we're measuring variables which are global to the VM. The backing device that we are selecting may not be writing out any dirty pages, in which case, we're just spinning in balance_dirty_pages_ratelimited().

OK, so it's unrelated to page reclaim.

Quoted 2 lines

Should we therefore perhaps be looking at adding per-backing_dev stats too?

That's what mm-per-device-dirty-threshold.patch and friends are doing. Whether it works adequately is not really known at this time. Unfortunately kernel developers don't test -mm much.

Here is a the snapshot of vmstats when the problem happened. I believe this could help a little.

crash> kmem -V

       NR_FREE_PAGES: 680853
         NR_INACTIVE: 95380
           NR_ACTIVE: 26891
       NR_ANON_PAGES: 2507
      NR_FILE_MAPPED: 1832
       NR_FILE_PAGES: 119779
       NR_FILE_DIRTY: 0
        NR_WRITEBACK: 18272
 NR_SLAB_RECLAIMABLE: 1305
NR_SLAB_UNRECLAIMABLE: 2085
        NR_PAGETABLE: 123
     NR_UNSTABLE_NFS: 0
           NR_BOUNCE: 0
     NR_VMSCAN_WRITE: 0

In my testing, I always saw the processes are waiting in balance_dirty_pages_ratelimited(), never in throttle_vm_writeout() path.

But this could be because I have about 4Gig of memory in the system and plenty of mem is still available around.

I will rerun the test limiting memory to 1024MB and lets see if it takes in any different path.

Thanks
--Chakri

Quoted 43 lines · Andrew Morton

Quoted 25 lines · On Fri, 28 Sep 2007 16:32:18 -0400 Trond Myklebust…

Quoted 18 lines · Andrew Morton

Quoted 6 lines · On Fri, 28 Sep 2007 15:52:28 -0400 Trond Myklebust…

Quoted 5 lines · Andrew Morton

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 8 lines

Quoted 1 line

that one is nfs-on-loopback, which is a special case, isn't it?

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

Quoted 5 lines

hm, so ext3 got stuck in nfs via __alloc_pages direct reclaim?

We should be able to fix that by marking the backing device as write-congested. That'll have small race windows, but it should be a 99.9% fix?

Quoted 5 lines

No. The problem would rather appear to be that we're doing per-backing_dev writeback (if I read sync_sb_inodes() correctly), but we're measuring variables which are global to the VM. The backing device that we are selecting may not be writing out any dirty pages, in which case, we're just spinning in balance_dirty_pages_ratelimited().

OK, so it's unrelated to page reclaim.

Quoted 2 lines

Should we therefore perhaps be looking at adding per-backing_dev stats too?

That's what mm-per-device-dirty-threshold.patch and friends are doing. Whether it works adequately is not really known at this time. Unfortunately kernel developers don't test -mm much.

Here is a the snapshot of vmstats when the problem happened. I believe this could help a little.

crash> kmem -V

       NR_FREE_PAGES: 680853
         NR_INACTIVE: 95380
           NR_ACTIVE: 26891
       NR_ANON_PAGES: 2507
      NR_FILE_MAPPED: 1832
       NR_FILE_PAGES: 119779
       NR_FILE_DIRTY: 0
        NR_WRITEBACK: 18272
 NR_SLAB_RECLAIMABLE: 1305
NR_SLAB_UNRECLAIMABLE: 2085
        NR_PAGETABLE: 123
     NR_UNSTABLE_NFS: 0
           NR_BOUNCE: 0
     NR_VMSCAN_WRITE: 0

In my testing, I always saw the processes are waiting in balance_dirty_pages_ratelimited(), never in throttle_vm_writeout() path.

But this could be because I have about 4Gig of memory in the system and plenty of mem is still available around.

I will rerun the test limiting memory to 1024MB and lets see if it takes in any different path.

Thanks
--Chakri

Quoted 43 lines · Andrew Morton

Quoted 25 lines · On Fri, 28 Sep 2007 16:32:18 -0400 Trond Myklebust…

Quoted 18 lines · Andrew Morton

Quoted 6 lines · On Fri, 28 Sep 2007 15:52:28 -0400 Trond Myklebust…

Quoted 5 lines · Andrew Morton

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 8 lines

Quoted 1 line

that one is nfs-on-loopback, which is a special case, isn't it?

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

Quoted 5 lines

hm, so ext3 got stuck in nfs via __alloc_pages direct reclaim?

We should be able to fix that by marking the backing device as write-congested. That'll have small race windows, but it should be a 99.9% fix?

Quoted 5 lines

No. The problem would rather appear to be that we're doing per-backing_dev writeback (if I read sync_sb_inodes() correctly), but we're measuring variables which are global to the VM. The backing device that we are selecting may not be writing out any dirty pages, in which case, we're just spinning in balance_dirty_pages_ratelimited().

OK, so it's unrelated to page reclaim.

Quoted 2 lines

Should we therefore perhaps be looking at adding per-backing_dev stats too?

That's what mm-per-device-dirty-threshold.patch and friends are doing. Whether it works adequately is not really known at this time. Unfortunately kernel developers don't test -mm much.

Quoted 25 lines · On Fri, 28 Sep 2007 16:32:18 -0400 Trond Myklebust…

Quoted 18 lines · Andrew Morton

Quoted 6 lines · On Fri, 28 Sep 2007 15:52:28 -0400 Trond Myklebust…

Quoted 5 lines · Andrew Morton

Quoted 3 lines · On Fri, 28 Sep 2007 15:16:11 -0400 Trond Myklebust…

Looking back, they were getting caught up in balance_dirty_pages_ratelimited() and friends. See the attached example...

Quoted 8 lines

Quoted 1 line

that one is nfs-on-loopback, which is a special case, isn't it?

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

Quoted 5 lines

hm, so ext3 got stuck in nfs via __alloc_pages direct reclaim?

We should be able to fix that by marking the backing device as write-congested. That'll have small race windows, but it should be a 99.9% fix?

Quoted 5 lines

No. The problem would rather appear to be that we're doing per-backing_dev writeback (if I read sync_sb_inodes() correctly), but we're measuring variables which are global to the VM. The backing device that we are selecting may not be writing out any dirty pages, in which case, we're just spinning in balance_dirty_pages_ratelimited().

OK, so it's unrelated to page reclaim.

Quoted 2 lines

Should we therefore perhaps be looking at adding per-backing_dev stats too?

That's what mm-per-device-dirty-threshold.patch and friends are doing. Whether it works adequately is not really known at this time. Unfortunately kernel developers don't test -mm much.

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

I'm not sure that the hang that is illustrated here is so special. It is an example of a bog-standard ext3 write, that ends up calling the NFS client, which is hanging. The fact that it happens to be hanging on the nfsd process is more or less irrelevant here: the same thing could happen to any other process in the case where we have an NFS server that is down.

crash> bt 3188

crash> bt 3188

Quoted 5 lines · On Fri, 28 Sep 2007 13:00:53 -0400 Trond Myklebust…

Do these patches also cause the memory reclaimers to steer clear of devices that are congested (and stop waiting on a congested device if they see that it remains congested for a long period of time)? Most of the collateral blocking I see tends to happen in memory allocation...

No, they don't attempt to do that, but I suspect they put in place infrastructure which could be used to improve direct-reclaimer latency. In the throttle_vm_writeout() path, at least.

Do you know where the stalls are occurring? throttle_vm_writeout(), or via direct calls to congestion_wait() from page_alloc.c and vmscan.c? (running sysrq-w five or ten times will probably be enough to determine this)

Quoted 5 lines · Andrew Morton

Actually we perhaps could address this at the VFS level in another way. Processes which are writing to the dead NFS server will eventually block in balance_dirty_pages() once they've exceeded the memory limits and will remain blocked until the server wakes up - that's the behaviour we want.

Quoted 3 lines

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Do these patches also cause the memory reclaimers to steer clear of devices that are congested (and stop waiting on a congested device if they see that it remains congested for a long period of time)? Most of the collateral blocking I see tends to happen in memory allocation...

Actually we perhaps could address this at the VFS level in another way. Processes which are writing to the dead NFS server will eventually block in balance_dirty_pages() once they've exceeded the memory limits and will remain blocked until the server wakes up - that's the behaviour we want.

What we _don't_ want to happen is for other processes which are writing to other, non-dead devices to get collaterally blocked. We have patches which might fix that queued for 2.6.24. Peter?

Actually we perhaps could address this at the VFS level in another way. Processes which are writing to the dead NFS server will eventually block in balance_dirty_pages() once they've exceeded the memory limits and will remain blocked until the server wakes up - that's the behaviour we want.

Actually we perhaps could address this at the VFS level in another way. Processes which are writing to the dead NFS server will eventually block in balance_dirty_pages() once they've exceeded the memory limits and will remain blocked until the server wakes up - that's the behaviour we want.

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

There is plenty of HIGH MEMORY available in the system, but this process never progresses.

...

The problem seems to be in balance_dirty_pages, which calculates dirty_thresh based on only ZONE_NORMAL. The same scenario works fine in 2.4. The dd processes finishes in no time.
NFS file systems can go offline, due to multiple reasons, a failed switch, filer etc, but that should not effect other file systems in the machine.
Can this behavior be fenced?, can the buffer cache be tuned so that other processes do not see the effect?

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Quoted 8 lines · Chakri n

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Quoted 6 lines

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Peter, do you think this patch will help?

It's works on .23-rc8-mm2 with out any problems.

"dd" process does not hang any more.

Thanks for all the help.

Cheers
--Chakri

===
writeback: avoid possible balance_dirty_pages() lockup on light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than enough dirty data.

The problem case:

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time as long as sda is overloaded

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    3 +++
 1 file changed, 3 insertions(+)

Quoted 8 lines · Chakri n

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Quoted 6 lines

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Peter, do you think this patch will help?

It's works on .23-rc8-mm2 with out any problems.

"dd" process does not hang any more.

Thanks for all the help.

Cheers
--Chakri

===
writeback: avoid possible balance_dirty_pages() lockup on light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than enough dirty data.

The problem case:

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time as long as sda is overloaded

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    3 +++
 1 file changed, 3 insertions(+)

Quoted 9 lines · Fengguang Wu

Quoted 8 lines · Chakri n

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Quoted 8 lines

Quoted 6 lines

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Peter, do you think this patch will help?

In another sub-thread:

Quoted 2 lines

It's works on .23-rc8-mm2 with out any problems.

Quoted 6 lines

"dd" process does not hang any more.

Thanks for all the help.

Cheers
--Chakri

So the per-bdi dirty patches that are in -mm already fix the problem.

Quoted 35 lines

===
writeback: avoid possible balance_dirty_pages() lockup on light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than enough dirty data.

The problem case:

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time as long as sda is overloaded

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    3 +++
 1 file changed, 3 insertions(+)

mm/page-writeback.c +3 -0

@@ -227,6 +227,9 @@ static void balance_dirty_pages(struct a
 		if (nr_reclaimable + global_page_state(NR_WRITEBACK) <=
 			dirty_thresh)
 				break;
+		if (list_empty(&mapping->host->i_sb->s_dirty) &&
+		    list_empty(&mapping->host->i_sb->s_io))
+			break;
 

mm/page-writeback.c static void balance_dirty_pages(struct a · line 235

Quoted 3 lines

mm/page-writeback.c +0 -0

 		if (!dirty_exceeded)
 			dirty_exceeded = 1;

On the patch itself, not sure if it would have been enough. As soon as there is a single dirty inode on the list one would get caught in the same problem as before.

That is, if NFS_dirty+NFS_unstable+NFS_writeback > dirty_limit this break won't fix it.

Quoted 9 lines · Fengguang Wu

Quoted 8 lines · Chakri n

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Quoted 8 lines

Quoted 6 lines

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Peter, do you think this patch will help?

In another sub-thread:

Quoted 2 lines

It's works on .23-rc8-mm2 with out any problems.

Quoted 6 lines

"dd" process does not hang any more.

Thanks for all the help.

Cheers
--Chakri

So the per-bdi dirty patches that are in -mm already fix the problem.

Quoted 35 lines

===
writeback: avoid possible balance_dirty_pages() lockup on light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than enough dirty data.

The problem case:

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time as long as sda is overloaded

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    3 +++
 1 file changed, 3 insertions(+)

mm/page-writeback.c +3 -0

@@ -227,6 +227,9 @@ static void balance_dirty_pages(struct a
 		if (nr_reclaimable + global_page_state(NR_WRITEBACK) <=
 			dirty_thresh)
 				break;
+		if (list_empty(&mapping->host->i_sb->s_dirty) &&
+		    list_empty(&mapping->host->i_sb->s_io))
+			break;
 

mm/page-writeback.c static void balance_dirty_pages(struct a · line 235

Quoted 3 lines

mm/page-writeback.c +0 -0

 		if (!dirty_exceeded)
 			dirty_exceeded = 1;

On the patch itself, not sure if it would have been enough. As soon as there is a single dirty inode on the list one would get caught in the same problem as before.

That is, if NFS_dirty+NFS_unstable+NFS_writeback > dirty_limit this break won't fix it.

Quoted 3 lines · Peter Zijlstra

On the patch itself, not sure if it would have been enough. As soon as there is a single dirty inode on the list one would get caught in the same problem as before.

That should not be a problem. Normally the few new dirty inodes will be all cleaned in one go and there are no more dirty inodes left(at least for a moment). Hmm, I guess the new 'break' should be moved immediately after writeback_inodes()...

Quoted 2 lines

That is, if NFS_dirty+NFS_unstable+NFS_writeback > dirty_limit this break won't fix it.

In fact this patch exactly targets at this condition.
When NFS* < dirty_limit, Chakri won't see the lockup at all. The problem was, there are only two 'break's in the loop, and neither one evaluates to true for his dd command.

Quoted 3 lines · Peter Zijlstra

On the patch itself, not sure if it would have been enough. As soon as there is a single dirty inode on the list one would get caught in the same problem as before.

That should not be a problem. Normally the few new dirty inodes will be all cleaned in one go and there are no more dirty inodes left(at least for a moment). Hmm, I guess the new 'break' should be moved immediately after writeback_inodes()...

Quoted 2 lines

That is, if NFS_dirty+NFS_unstable+NFS_writeback > dirty_limit this break won't fix it.

In fact this patch exactly targets at this condition.
When NFS* < dirty_limit, Chakri won't see the lockup at all. The problem was, there are only two 'break's in the loop, and neither one evaluates to true for his dd command.

Quoted 8 lines · Chakri n

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Quoted 6 lines

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Peter, do you think this patch will help?

===
writeback: avoid possible balance_dirty_pages() lockup on light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than enough dirty data.

The problem case:

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time as long as sda is overloaded

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    3 +++
 1 file changed, 3 insertions(+)

Quoted 8 lines · Chakri n

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Quoted 6 lines

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Peter, do you think this patch will help?

===
writeback: avoid possible balance_dirty_pages() lockup on light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than enough dirty data.

The problem case:

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time as long as sda is overloaded

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    3 +++
 1 file changed, 3 insertions(+)

Quoted 20 lines · Fengguang Wu

Quoted 8 lines · Chakri n

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Quoted 6 lines

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Peter, do you think this patch will help?

===
writeback: avoid possible balance_dirty_pages() lockup on light-load bdi

This looks better than the other candidate to fix the problem. Are we going to fix 2.6.23 before release? Multiple people have reported this problem now...

Quoted 20 lines · Fengguang Wu

Quoted 8 lines · Chakri n

Hi,

In my testing, a unresponsive file system can hang all I/O in the system. This is not seen in 2.4.

I started 20 threads doing I/O on a NFS share. They are just doing 4K writes in a loop.

Quoted 6 lines

Now I stop NFS server hosting the NFS share and start a "dd" process to write a file on local EXT3 file system.

# dd if=/dev/zero of=/tmp/x count=1000

This process never progresses.

Peter, do you think this patch will help?

===
writeback: avoid possible balance_dirty_pages() lockup on light-load bdi

This looks better than the other candidate to fix the problem. Are we going to fix 2.6.23 before release? Multiple people have reported this problem now...

writeback: avoid possible balance_dirty_pages() lockup on a light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than available dirty data.

The problem case:

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

But the newly introduced 'break' could make the nr_writeback drift away above the dirty limit. The workaround is to limit the error under 1MB.

Cc Chuck Ebbert <[email protected]>

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    5 +++++
 1 file changed, 5 insertions(+)

mm/page-writeback.c +0 -0

@@ -250,6 +250,11 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;

Quoted 7 lines · On Tue, 2 Oct 2007 10:00:40 +0800 Fengguang Wu <[email protected]> wrote:

writeback: avoid possible balance_dirty_pages() lockup on a light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than available dirty data.

The problem case:

Quoted 6 lines

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time

Please quantify loooong.

Quoted 6 lines

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

But the newly introduced 'break' could make the nr_writeback drift away above the dirty limit. The workaround is to limit the error under 1MB.

I'm still not sure that we fully understand this yet.

If the sdb writer is stuck in balance_dirty_pages() then all sda writers will be in balance_dirty_pages() too, madly writing stuff out to sda. And pdflush will be writing out sda as well. All this writeout to sda should release the sdb writer.

Why isn't this happening?

Quoted 17 lines

Cc Chuck Ebbert <[email protected]>

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    5 +++++
 1 file changed, 5 insertions(+)

mm/page-writeback.c +3 -0

@@ -250,6 +250,11 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;
 			if (pages_written >= write_chunk)
 				break;		/* We've done our duty */
+			if (list_empty(&mapping->host->i_sb->s_dirty) &&
+			    list_empty(&mapping->host->i_sb->s_io) &&
+			    nr_reclaimable + global_page_state(NR_WRITEBACK) <=

mm/page-writeback.c static void balance_dirty_pages(struct a · line 260

Quoted 5 lines

mm/page-writeback.c +2 -0

+				    dirty_thresh + (1 << (20-PAGE_CACHE_SHIFT)))
+				break;
 		}
 		congestion_wait(WRITE, HZ/10);
 	}

Well that has a nice safetly net. Perhaps it could fail a bit later on, but that depends on why it's failing.

How well tested was this?

If we merge this for 2.6.23 then I expect that we'll immediately unmerge it for 2.6.24 because Peter's stuff fixes this problem by other means.

Do we all agree with the above sentence?

Quoted 7 lines · On Tue, 2 Oct 2007 10:00:40 +0800 Fengguang Wu <[email protected]> wrote:

writeback: avoid possible balance_dirty_pages() lockup on a light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than available dirty data.

The problem case:

Quoted 6 lines

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time

Please quantify loooong.

Quoted 6 lines

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

But the newly introduced 'break' could make the nr_writeback drift away above the dirty limit. The workaround is to limit the error under 1MB.

I'm still not sure that we fully understand this yet.

If the sdb writer is stuck in balance_dirty_pages() then all sda writers will be in balance_dirty_pages() too, madly writing stuff out to sda. And pdflush will be writing out sda as well. All this writeout to sda should release the sdb writer.

Why isn't this happening?

Quoted 17 lines

Cc Chuck Ebbert <[email protected]>

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    5 +++++
 1 file changed, 5 insertions(+)

mm/page-writeback.c +3 -0

@@ -250,6 +250,11 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;
 			if (pages_written >= write_chunk)
 				break;		/* We've done our duty */
+			if (list_empty(&mapping->host->i_sb->s_dirty) &&
+			    list_empty(&mapping->host->i_sb->s_io) &&
+			    nr_reclaimable + global_page_state(NR_WRITEBACK) <=

mm/page-writeback.c static void balance_dirty_pages(struct a · line 260

Quoted 5 lines

mm/page-writeback.c +2 -0

+				    dirty_thresh + (1 << (20-PAGE_CACHE_SHIFT)))
+				break;
 		}
 		congestion_wait(WRITE, HZ/10);
 	}

Well that has a nice safetly net. Perhaps it could fail a bit later on, but that depends on why it's failing.

How well tested was this?

If we merge this for 2.6.23 then I expect that we'll immediately unmerge it for 2.6.24 because Peter's stuff fixes this problem by other means.

Do we all agree with the above sentence?

One more serious problem is, a busy writer could also drain all the dirty pages and make (nr_writeback == dirty_limit+1MB). In that case, I suspect the light-load sdb writer still have good chance to make progress(need confirmation).

One more serious problem is, a busy writer could also drain all the dirty pages and make (nr_writeback == dirty_limit+1MB). In that case, I suspect the light-load sdb writer still have good chance to make progress(need confirmation).

On Tue, Oct 02, 2007 at 08:13:27PM +0800, Fengguang Wu wrote:
[...]

Quoted 4 lines

One more serious problem is, a busy writer could also drain all the dirty pages and make (nr_writeback == dirty_limit+1MB). In that case, I suspect the light-load sdb writer still have good chance to make progress(need confirmation).

Well it seems to be a really tricky issue without knowing the per-bdi numbers. Maybe we could just encourage users to upgrade to 2.6.24...

Quoted 1 line · Chuck Ebbert

On 09/29/2007 07:04 AM, Fengguang Wu wrote:

(expecting real world confirmations...)

Here is a new safer version. It's more ugly though.

---
writeback: avoid possible balance_dirty_pages() lockup on a light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than available dirty data.

The problem case:

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

But the newly introduced 'break' could make the nr_writeback drift away above the dirty limit. The workaround is to limit the error under 1MB.

Cc Chuck Ebbert <[email protected]>

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    5 +++++
 1 file changed, 5 insertions(+)

mm/page-writeback.c +0 -0

@@ -250,6 +250,11 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;

Quoted 1 line · Chuck Ebbert

On 09/29/2007 07:04 AM, Fengguang Wu wrote:

(expecting real world confirmations...)

Here is a new safer version. It's more ugly though.

---
writeback: avoid possible balance_dirty_pages() lockup on a light-load bdi

On a busy-writing system, a writer could be hold up infinitely on a light-load device. It will be trying to sync more than available dirty data.

The problem case:

0. sda/nr_dirty >= dirty_limit;
   sdb/nr_dirty == 0

1. dd writes 32 pages on sdb
2. balance_dirty_pages() blocks dd, and tries to write 6MB.
3. it never gets there: there's only 128KB dirty data.
4. dd may be blocked for a loooong time

Fix it by returning on 'zero dirty inodes' in the current bdi.
(In fact there are slight differences between 'dirty inodes' and 'dirty pages'. But there is no available counters for 'dirty pages'.)

But the newly introduced 'break' could make the nr_writeback drift away above the dirty limit. The workaround is to limit the error under 1MB.

Cc Chuck Ebbert <[email protected]>

Cc Peter Zijlstra <[email protected]>

Signed-off-by Fengguang Wu <[email protected]>

---
 mm/page-writeback.c |    5 +++++
 1 file changed, 5 insertions(+)

mm/page-writeback.c +0 -0

@@ -250,6 +250,11 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;

Quoted 8 lines · Fengguang Wu

---
 mm/page-writeback.c |    5 +++++
 1 file changed, 5 insertions(+)

mm/page-writeback.c +0 -0

@@ -250,6 +250,11 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;

mm/page-writeback.c static void balance_dirty_pages(struct a · line 260

Quoted 10 lines

mm/page-writeback.c +5 -0

 			if (pages_written >= write_chunk)
 				break;		/* We've done our duty */
+			if (list_empty(&mapping->host->i_sb->s_dirty) &&
+			    list_empty(&mapping->host->i_sb->s_io) &&
+			    nr_reclaimable + global_page_state(NR_WRITEBACK) <=
+				    dirty_thresh + (1 << (20-PAGE_CACHE_SHIFT)))
+				break;
 		}
 		congestion_wait(WRITE, HZ/10);
 	}

I've been testing 2.6.23-rc9 + this patch all morning but have just seen a lockup. As usual it happened just after a large file copy finished and while nr_dirty is still large. I'm sorry to say I didn't have a serial console running so I don't have an other info. I will try again and see if I can capture some more data.

I did notice that at the beginning of my tests the dirty blocks are written back more quickly than usual

nr_dirty count after the copy finished and then 60 seconds later :-
after copy +60 seconds
73520 0
73533 0
68554 1

but after several iterations of my testcase & just before the lockup
68560 57165
71974 62896

which is about the same as a unpatched kernel.

Quoted 8 lines · Fengguang Wu

---
 mm/page-writeback.c |    5 +++++
 1 file changed, 5 insertions(+)

mm/page-writeback.c +0 -0

@@ -250,6 +250,11 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;

mm/page-writeback.c static void balance_dirty_pages(struct a · line 260

Quoted 10 lines

mm/page-writeback.c +5 -0

 			if (pages_written >= write_chunk)
 				break;		/* We've done our duty */
+			if (list_empty(&mapping->host->i_sb->s_dirty) &&
+			    list_empty(&mapping->host->i_sb->s_io) &&
+			    nr_reclaimable + global_page_state(NR_WRITEBACK) <=
+				    dirty_thresh + (1 << (20-PAGE_CACHE_SHIFT)))
+				break;
 		}
 		congestion_wait(WRITE, HZ/10);
 	}

I've been testing 2.6.23-rc9 + this patch all morning but have just seen a lockup. As usual it happened just after a large file copy finished and while nr_dirty is still large. I'm sorry to say I didn't have a serial console running so I don't have an other info. I will try again and see if I can capture some more data.

I did notice that at the beginning of my tests the dirty blocks are written back more quickly than usual

nr_dirty count after the copy finished and then 60 seconds later :-
after copy +60 seconds
73520 0
73533 0
68554 1

but after several iterations of my testcase & just before the lockup
68560 57165
71974 62896

which is about the same as a unpatched kernel.