Killing process in D state on mount to dead NFS server.

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

I want to kill the process and unmount.

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains:
umount2: Device or resource busy
umount.nfs: /mnt/foo: device is busy

kill -9 does not work on process.

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

Quoted 17 lines · Ben Greear [[email protected]] wrote:

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

I want to kill the process and unmount.

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains:
umount2: Device or resource busy
umount.nfs: /mnt/foo: device is busy

Quoted 6 lines

kill -9 does not work on process.

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

You don't need a fake NFS server, you just need a fake or real server with that IP address. A popular way is to alias that IP on the NFS client itself.

See the second popular answer below:
http://stackoverflow.com/questions/40317/force-unmount-of-nfs-mounted-directory

Regards, Malahal.

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

I want to kill the process and unmount.

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains:
umount2: Device or resource busy
umount.nfs: /mnt/foo: device is busy

kill -9 does not work on process.

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

Thanks,
Ben

Quoted 9 lines · On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear <[email protected]> wrote:

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

Quoted 6 lines

I want to kill the process and unmount.

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs: /mnt/foo:
device is busy

kill -9 does not work on process.

Kill -1 should work (since about 2.6.25 or so).

If it doesn't please report the kernel version and cat /proc/$PID/stack

for some processes that cannot be killed.

NeilBrown

Quoted 7 lines

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

Thanks, Ben

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 13 lines · NeilBrown

Quoted 11 lines · On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear <[email protected]> wrote:

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

I want to kill the process and unmount.

Quoted 6 lines

Quoted 4 lines

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs: /mnt/foo:
device is busy

kill -9 does not work on process.

Kill -1 should work (since about 2.6.25 or so).

That is -[ONE], right? Assuming so, it did not work for me.

Kernel is 3.14.4+, with some of extra patches, but probably nothing that influences this particular behaviour.

[root@lf1005-14010010 ~]# cat /proc/3805/stack

[<ffffffff811371ba>] sleep_on_page+0x9/0xd
[<ffffffff8113738e>] wait_on_page_bit+0x71/0x78
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d
[<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77
[<ffffffffa0f04734>] nfs_file_fsync+0x37/0x83 [nfs]
[<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b
[<ffffffff811a8d4b>] vfs_fsync+0x17/0x19
[<ffffffffa0f05305>] nfs_file_flush+0x6b/0x6f [nfs]
[<ffffffff81183e46>] filp_close+0x3f/0x71
[<ffffffff8119c8ae>] __close_fd+0x80/0x98
[<ffffffff81183de5>] SyS_close+0x1c/0x3e
[<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
[root@lf1005-14010010 ~]# kill -1 3805
[root@lf1005-14010010 ~]# cat /proc/3805/stack
[<ffffffff811371ba>] sleep_on_page+0x9/0xd
[<ffffffff8113738e>] wait_on_page_bit+0x71/0x78
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d
[<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77
[<ffffffffa0f04734>] nfs_file_fsync+0x37/0x83 [nfs]
[<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b
[<ffffffff811a8d4b>] vfs_fsync+0x17/0x19
[<ffffffffa0f05305>] nfs_file_flush+0x6b/0x6f [nfs]
[<ffffffff81183e46>] filp_close+0x3f/0x71
[<ffffffff8119c8ae>] __close_fd+0x80/0x98
[<ffffffff81183de5>] SyS_close+0x1c/0x3e
[<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Thanks,
Ben

Quoted 14 lines

If it doesn't please report the kernel version and cat /proc/$PID/stack

for some processes that cannot be killed.

NeilBrown

Quoted 7 lines

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

Thanks, Ben

- --
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJT2rLiAAoJELbHqkYeJT4OqPgH/0taKW6Be90c1mETZf9yeqZF YMLZk8XC2wloEd9nVz//mXREmiu18Hc+5p7Upd4Os21J2P4PBMGV6P/9DMxxehwH YX1HKha0EoAsbO5ILQhbLf83cRXAPEpvJPgYHrq6xjlKB8Q8OxxND37rY7kl19Zz sdAw6GiqHICF3Hq1ATa/jvixMluDnhER9Dln3wOdAGzmmuFYqpTsV4EwzbKKqInJ 6C15q+cq/9aYh6usN6z2qJhbHgqM9EWcPL6jOrCwX4PbC1XjKHekpFN0t9oKQClx qSPuweMQ7fP4IBd2Ke8L/QlyOVblAKSE7t+NdrjfzLmYPzyHTyfLABR/BI053to= =/9FJ
-----END PGP SIGNATURE-----

Quoted 17 lines · On Thu, 31 Jul 2014 14:20:07 -0700 Ben Greear…

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 13 lines · NeilBrown

Quoted 11 lines · On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear…

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

I want to kill the process and unmount.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs: /mnt/foo:
device is busy

kill -9 does not work on process.

Kill -1 should work (since about 2.6.25 or so).

That is -[ONE], right? Assuming so, it did not work for me.

No, it was "-9" .... sorry, I really shouldn't be let out without my proof reader.

However the 'stack' is sufficient to see what is going on.

The problem is that it is blocked inside the "VM" well away from NFS and there is no way for NFS to say "give up and go home".

I'd suggest that is a bug. I cannot see any justification for fsync to not
be killable.
It wouldn't be too hard to create a patch to make it so.
It would be a little harder to examine all call paths and create a convincing case that the patch was safe.
It might be herculean task to convince others that it was the right thing to do.... so let's start with that one.

Hi Linux-mm and fs-devel people. What do people think of making "fsync" and variants "KILLABLE" ??

Quoted 17 lines · On Thu, 31 Jul 2014 14:20:07 -0700 Ben Greear…

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 13 lines · NeilBrown

Quoted 11 lines · On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear…

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

I want to kill the process and unmount.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs: /mnt/foo:
device is busy

kill -9 does not work on process.

Kill -1 should work (since about 2.6.25 or so).

That is -[ONE], right? Assuming so, it did not work for me.

No, it was "-9" .... sorry, I really shouldn't be let out without my proof reader.

However the 'stack' is sufficient to see what is going on.

The problem is that it is blocked inside the "VM" well away from NFS and there is no way for NFS to say "give up and go home".

I'd suggest that is a bug. I cannot see any justification for fsync to not
be killable.
It wouldn't be too hard to create a patch to make it so.
It would be a little harder to examine all call paths and create a convincing case that the patch was safe.
It might be herculean task to convince others that it was the right thing to do.... so let's start with that one.

Hi Linux-mm and fs-devel people. What do people think of making "fsync" and variants "KILLABLE" ??

I probably only need a little bit of encouragement to write a patch....

Thanks,
NeilBrown

Quoted 69 lines

Kernel is 3.14.4+, with some of extra patches, but probably nothing that influences this particular behaviour.

[root@lf1005-14010010 ~]# cat /proc/3805/stack

[<ffffffff811371ba>] sleep_on_page+0x9/0xd
[<ffffffff8113738e>] wait_on_page_bit+0x71/0x78
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d
[<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77
[<ffffffffa0f04734>] nfs_file_fsync+0x37/0x83 [nfs]
[<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b
[<ffffffff811a8d4b>] vfs_fsync+0x17/0x19
[<ffffffffa0f05305>] nfs_file_flush+0x6b/0x6f [nfs]
[<ffffffff81183e46>] filp_close+0x3f/0x71
[<ffffffff8119c8ae>] __close_fd+0x80/0x98
[<ffffffff81183de5>] SyS_close+0x1c/0x3e
[<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
[root@lf1005-14010010 ~]# kill -1 3805
[root@lf1005-14010010 ~]# cat /proc/3805/stack
[<ffffffff811371ba>] sleep_on_page+0x9/0xd
[<ffffffff8113738e>] wait_on_page_bit+0x71/0x78
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d
[<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77
[<ffffffffa0f04734>] nfs_file_fsync+0x37/0x83 [nfs]
[<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b
[<ffffffff811a8d4b>] vfs_fsync+0x17/0x19
[<ffffffffa0f05305>] nfs_file_flush+0x6b/0x6f [nfs]
[<ffffffff81183e46>] filp_close+0x3f/0x71
[<ffffffff8119c8ae>] __close_fd+0x80/0x98
[<ffffffff81183de5>] SyS_close+0x1c/0x3e
[<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Thanks,
Ben

Quoted 14 lines

If it doesn't please report the kernel version and cat /proc/$PID/stack

for some processes that cannot be killed.

NeilBrown

Quoted 7 lines

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

Thanks, Ben

- --
Ben Greear <greearb-my8/[email protected]> Candela Technologies Inc http://www.candelatech.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJT2rLiAAoJELbHqkYeJT4OqPgH/0taKW6Be90c1mETZf9yeqZF YMLZk8XC2wloEd9nVz//mXREmiu18Hc+5p7Upd4Os21J2P4PBMGV6P/9DMxxehwH YX1HKha0EoAsbO5ILQhbLf83cRXAPEpvJPgYHrq6xjlKB8Q8OxxND37rY7kl19Zz sdAw6GiqHICF3Hq1ATa/jvixMluDnhER9Dln3wOdAGzmmuFYqpTsV4EwzbKKqInJ 6C15q+cq/9aYh6usN6z2qJhbHgqM9EWcPL6jOrCwX4PbC1XjKHekpFN0t9oKQClx qSPuweMQ7fP4IBd2Ke8L/QlyOVblAKSE7t+NdrjfzLmYPzyHTyfLABR/BI053to= =/9FJ
-----END PGP SIGNATURE-----

Quoted 44 lines · On Fri, 1 Aug 2014 07:50:53 +1000 NeilBrown <[email protected]> wrote:

Quoted 17 lines · On Thu, 31 Jul 2014 14:20:07 -0700 Ben Greear <[email protected]> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 13 lines · NeilBrown

Quoted 11 lines · On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear <[email protected]> wrote:

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

I want to kill the process and unmount.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs: /mnt/foo:
device is busy

kill -9 does not work on process.

Kill -1 should work (since about 2.6.25 or so).

That is -[ONE], right? Assuming so, it did not work for me.

No, it was "-9" .... sorry, I really shouldn't be let out without my proof reader.

However the 'stack' is sufficient to see what is going on.

The problem is that it is blocked inside the "VM" well away from NFS and there is no way for NFS to say "give up and go home".

I'd suggest that is a bug. I cannot see any justification for fsync to not
be killable.
It wouldn't be too hard to create a patch to make it so.
It would be a little harder to examine all call paths and create a convincing case that the patch was safe.
It might be herculean task to convince others that it was the right thing to do.... so let's start with that one.

Quoted 8 lines

Hi Linux-mm and fs-devel people. What do people think of making "fsync" and variants "KILLABLE" ??

I probably only need a little bit of encouragement to write a patch....

Thanks,
NeilBrown

It would be good to fix this in some fashion once and for all, and the wait_on_page_writeback wait is a major source of pain for a lot of people.

So to summarize...

The problem in a nutshell is that Ben has some cached writes to the NFS server, but the server has gone away (presumably forever). The question is -- how do we communicate to the kernel that that server isn't coming back and that those dirty pages should be invalidated so that we can umount the filesystem?

Allowing fsync/close to be killable sounds reasonable to me as at least a partial solution. Both close(2) and fsync(2) are allowed to return EINTR according to the POSIX spec. Allowing a kill -9 there seems like it should be fine, and maybe we ought to even consider letting it be susceptible to lesser signals.

That still leaves some open questions though...

Is that enough to fix it? You'd still have the dirty pages lingering around, right? Would a umount -f presumably work at that point?

Quoted 61 lines

Quoted 61 lines

Kernel is 3.14.4+, with some of extra patches, but probably nothing that influences this particular behaviour.

[root@lf1005-14010010 ~]# cat /proc/3805/stack

[<ffffffff811371ba>] sleep_on_page+0x9/0xd
[<ffffffff8113738e>] wait_on_page_bit+0x71/0x78
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d
[<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77
[<ffffffffa0f04734>] nfs_file_fsync+0x37/0x83 [nfs]
[<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b
[<ffffffff811a8d4b>] vfs_fsync+0x17/0x19
[<ffffffffa0f05305>] nfs_file_flush+0x6b/0x6f [nfs]
[<ffffffff81183e46>] filp_close+0x3f/0x71
[<ffffffff8119c8ae>] __close_fd+0x80/0x98
[<ffffffff81183de5>] SyS_close+0x1c/0x3e
[<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
[root@lf1005-14010010 ~]# kill -1 3805
[root@lf1005-14010010 ~]# cat /proc/3805/stack
[<ffffffff811371ba>] sleep_on_page+0x9/0xd
[<ffffffff8113738e>] wait_on_page_bit+0x71/0x78
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d
[<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77
[<ffffffffa0f04734>] nfs_file_fsync+0x37/0x83 [nfs]
[<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b
[<ffffffff811a8d4b>] vfs_fsync+0x17/0x19
[<ffffffffa0f05305>] nfs_file_flush+0x6b/0x6f [nfs]
[<ffffffff81183e46>] filp_close+0x3f/0x71
[<ffffffff8119c8ae>] __close_fd+0x80/0x98
[<ffffffff81183de5>] SyS_close+0x1c/0x3e
[<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Thanks,
Ben

Quoted 14 lines

If it doesn't please report the kernel version and cat /proc/$PID/stack

for some processes that cannot be killed.

NeilBrown

Quoted 7 lines

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

Thanks, Ben

- --
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

Quoted 9 lines

Quoted 8 lines

iQEcBAEBAgAGBQJT2rLiAAoJELbHqkYeJT4OqPgH/0taKW6Be90c1mETZf9yeqZF YMLZk8XC2wloEd9nVz//mXREmiu18Hc+5p7Upd4Os21J2P4PBMGV6P/9DMxxehwH YX1HKha0EoAsbO5ILQhbLf83cRXAPEpvJPgYHrq6xjlKB8Q8OxxND37rY7kl19Zz sdAw6GiqHICF3Hq1ATa/jvixMluDnhER9Dln3wOdAGzmmuFYqpTsV4EwzbKKqInJ 6C15q+cq/9aYh6usN6z2qJhbHgqM9EWcPL6jOrCwX4PbC1XjKHekpFN0t9oKQClx qSPuweMQ7fP4IBd2Ke8L/QlyOVblAKSE7t+NdrjfzLmYPzyHTyfLABR/BI053to= =/9FJ
-----END PGP SIGNATURE-----

--
Jeff Layton <[email protected]>

Doesn't NFS have an intr flag to allow kill -9 to work? Whenever I
have had that set it has appeared to work after about 30 seconds or so...without that kill -9 does not work when the nfs server is missing.

Quoted 154 lines

Quoted 44 lines · On Fri, 1 Aug 2014 07:50:53 +1000 NeilBrown <[email protected]> wrote:

Quoted 17 lines · On Thu, 31 Jul 2014 14:20:07 -0700 Ben Greear <[email protected]> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 13 lines · NeilBrown

Quoted 11 lines · On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear <[email protected]> wrote:

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

I want to kill the process and unmount.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs: /mnt/foo:
device is busy

kill -9 does not work on process.

Kill -1 should work (since about 2.6.25 or so).

That is -[ONE], right? Assuming so, it did not work for me.

No, it was "-9" .... sorry, I really shouldn't be let out without my proof reader.

However the 'stack' is sufficient to see what is going on.

The problem is that it is blocked inside the "VM" well away from NFS and there is no way for NFS to say "give up and go home".

I'd suggest that is a bug. I cannot see any justification for fsync to not
be killable.
It wouldn't be too hard to create a patch to make it so.
It would be a little harder to examine all call paths and create a convincing case that the patch was safe.
It might be herculean task to convince others that it was the right thing to do.... so let's start with that one.

Quoted 8 lines

Hi Linux-mm and fs-devel people. What do people think of making "fsync" and variants "KILLABLE" ??

I probably only need a little bit of encouragement to write a patch....

Thanks,
NeilBrown

It would be good to fix this in some fashion once and for all, and the wait_on_page_writeback wait is a major source of pain for a lot of people.

So to summarize...

The problem in a nutshell is that Ben has some cached writes to the NFS server, but the server has gone away (presumably forever). The question is -- how do we communicate to the kernel that that server isn't coming back and that those dirty pages should be invalidated so that we can umount the filesystem?

Allowing fsync/close to be killable sounds reasonable to me as at least a partial solution. Both close(2) and fsync(2) are allowed to return EINTR according to the POSIX spec. Allowing a kill -9 there seems like it should be fine, and maybe we ought to even consider letting it be susceptible to lesser signals.

That still leaves some open questions though...

Is that enough to fix it? You'd still have the dirty pages lingering around, right? Would a umount -f presumably work at that point?

Quoted 61 lines

Quoted 61 lines

Kernel is 3.14.4+, with some of extra patches, but probably nothing that influences this particular behaviour.

[root@lf1005-14010010 ~]# cat /proc/3805/stack

[<ffffffff811371ba>] sleep_on_page+0x9/0xd
[<ffffffff8113738e>] wait_on_page_bit+0x71/0x78
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d
[<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77
[<ffffffffa0f04734>] nfs_file_fsync+0x37/0x83 [nfs]
[<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b
[<ffffffff811a8d4b>] vfs_fsync+0x17/0x19
[<ffffffffa0f05305>] nfs_file_flush+0x6b/0x6f [nfs]
[<ffffffff81183e46>] filp_close+0x3f/0x71
[<ffffffff8119c8ae>] __close_fd+0x80/0x98
[<ffffffff81183de5>] SyS_close+0x1c/0x3e
[<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
[root@lf1005-14010010 ~]# kill -1 3805
[root@lf1005-14010010 ~]# cat /proc/3805/stack
[<ffffffff811371ba>] sleep_on_page+0x9/0xd
[<ffffffff8113738e>] wait_on_page_bit+0x71/0x78
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d
[<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77
[<ffffffffa0f04734>] nfs_file_fsync+0x37/0x83 [nfs]
[<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b
[<ffffffff811a8d4b>] vfs_fsync+0x17/0x19
[<ffffffffa0f05305>] nfs_file_flush+0x6b/0x6f [nfs]
[<ffffffff81183e46>] filp_close+0x3f/0x71
[<ffffffff8119c8ae>] __close_fd+0x80/0x98
[<ffffffff81183de5>] SyS_close+0x1c/0x3e
[<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Thanks,
Ben

Quoted 14 lines

If it doesn't please report the kernel version and cat /proc/$PID/stack

for some processes that cannot be killed.

NeilBrown

Quoted 7 lines

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

Thanks, Ben

- --
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

Quoted 9 lines

Quoted 8 lines

iQEcBAEBAgAGBQJT2rLiAAoJELbHqkYeJT4OqPgH/0taKW6Be90c1mETZf9yeqZF YMLZk8XC2wloEd9nVz//mXREmiu18Hc+5p7Upd4Os21J2P4PBMGV6P/9DMxxehwH YX1HKha0EoAsbO5ILQhbLf83cRXAPEpvJPgYHrq6xjlKB8Q8OxxND37rY7kl19Zz sdAw6GiqHICF3Hq1ATa/jvixMluDnhER9Dln3wOdAGzmmuFYqpTsV4EwzbKKqInJ 6C15q+cq/9aYh6usN6z2qJhbHgqM9EWcPL6jOrCwX4PbC1XjKHekpFN0t9oKQClx qSPuweMQ7fP4IBd2Ke8L/QlyOVblAKSE7t+NdrjfzLmYPzyHTyfLABR/BI053to= =/9FJ
-----END PGP SIGNATURE-----

--
Jeff Layton <[email protected]>

Quoted 44 lines · On Fri, 1 Aug 2014 07:50:53 +1000 NeilBrown <[email protected]> wrote:

Quoted 17 lines · On Thu, 31 Jul 2014 14:20:07 -0700 Ben Greear <[email protected]> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 13 lines · NeilBrown

Quoted 11 lines · On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear <[email protected]> wrote:

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

I want to kill the process and unmount.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs: /mnt/foo:
device is busy

kill -9 does not work on process.

Kill -1 should work (since about 2.6.25 or so).

That is -[ONE], right? Assuming so, it did not work for me.

No, it was "-9" .... sorry, I really shouldn't be let out without my proof reader.

However the 'stack' is sufficient to see what is going on.

The problem is that it is blocked inside the "VM" well away from NFS and there is no way for NFS to say "give up and go home".

I'd suggest that is a bug. I cannot see any justification for fsync to not
be killable.
It wouldn't be too hard to create a patch to make it so.
It would be a little harder to examine all call paths and create a convincing case that the patch was safe.
It might be herculean task to convince others that it was the right thing to do.... so let's start with that one.

Quoted 8 lines

Hi Linux-mm and fs-devel people. What do people think of making "fsync" and variants "KILLABLE" ??

I probably only need a little bit of encouragement to write a patch....

Thanks,
NeilBrown

It would be good to fix this in some fashion once and for all, and the wait_on_page_writeback wait is a major source of pain for a lot of people.

So to summarize...

The problem in a nutshell is that Ben has some cached writes to the NFS server, but the server has gone away (presumably forever). The question is -- how do we communicate to the kernel that that server isn't coming back and that those dirty pages should be invalidated so that we can umount the filesystem?

Allowing fsync/close to be killable sounds reasonable to me as at least a partial solution. Both close(2) and fsync(2) are allowed to return EINTR according to the POSIX spec. Allowing a kill -9 there seems like it should be fine, and maybe we ought to even consider letting it be susceptible to lesser signals.

That still leaves some open questions though...

Is that enough to fix it? You'd still have the dirty pages lingering around, right? Would a umount -f presumably work at that point?

Quoted 4 lines

That still leaves some open questions though...

Is that enough to fix it? You'd still have the dirty pages lingering around, right? Would a umount -f presumably work at that point?

'umount -f' will kill any outstanding RPC calls that are causing the mount to hang, but doesn't do anything to change page states or NFS file/lock states.

Quoted 8 lines · On Fri, 1 Aug 2014 22:55:42 -0400 Trond Myklebust <[email protected]> wrote:

Quoted 4 lines

That still leaves some open questions though...

Is that enough to fix it? You'd still have the dirty pages lingering around, right? Would a umount -f presumably work at that point?

'umount -f' will kill any outstanding RPC calls that are causing the mount to hang, but doesn't do anything to change page states or NFS file/lock states.

Should it though?

MNT_FORCE (since Linux 2.1.116)
Force unmount even if busy. This can cause data loss. (Only for NFS mounts.)

Given that data loss is explicitly permitted, I suspect it should.

Can we make MNT_FORCE on NFS not only abort outstanding RPC calls, but
fail all subsequent RPC calls? That might make it really useful. You
wouldn't even need to "kill -9" then.

Quoted 11 lines · NeilBrown

Quoted 9 lines · On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear <[email protected]> wrote:

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

Quoted 8 lines

Quoted 6 lines

I want to kill the process and unmount.

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs:
/mnt/foo: device is busy

kill -9 does not work on process.

Kill -1 should work (since about 2.6.25 or so).

That is -[ONE], right? Assuming so, it did not work for me.

Kernel is 3.14.4+, with some of extra patches, but probably nothing that influences this particular behaviour.

[root@lf1005-14010010 ~]# cat /proc/3805/stack [<ffffffff811371ba>] sleep_on_page+0x9/0xd [<ffffffff8113738e>] wait_on_page_bit+0x71/0x78

[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d [<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77 [<ffffffffa0f04734>]
nfs_file_fsync+0x37/0x83 [nfs] [<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b [<ffffffff811a8d4b>] vfs_fsync+0x17/0x19 [<ffffffffa0f05305>]
nfs_file_flush+0x6b/0x6f [nfs] [<ffffffff81183e46>] filp_close+0x3f/0x71 [<ffffffff8119c8ae>] __close_fd+0x80/0x98 [<ffffffff81183de5>]
SyS_close+0x1c/0x3e [<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b [<ffffffffffffffff>] 0xffffffffffffffff [root@lf1005-14010010 ~]# kill -1 3805 
[root@lf1005-14010010 ~]# cat /proc/3805/stack [<ffffffff811371ba>] sleep_on_page+0x9/0xd [<ffffffff8113738e>] wait_on_page_bit+0x71/0x78 
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d [<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77 [<ffffffffa0f04734>]
nfs_file_fsync+0x37/0x83 [nfs] [<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b [<ffffffff811a8d4b>] vfs_fsync+0x17/0x19 [<ffffffffa0f05305>]
nfs_file_flush+0x6b/0x6f [nfs] [<ffffffff81183e46>] filp_close+0x3f/0x71 [<ffffffff8119c8ae>] __close_fd+0x80/0x98 [<ffffffff81183de5>]
SyS_close+0x1c/0x3e [<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b [<ffffffffffffffff>] 0xffffffffffffffff

Thanks, Ben

Quoted 11 lines

If it doesn't please report the kernel version and cat /proc/$PID/stack

for some processes that cannot be killed.

NeilBrown

Quoted 5 lines

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

Quoted 2 lines

Quoted 2 lines

Thanks, Ben

-- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello!

Did you get a chance to look at the stacks below?

Thanks,
Ben

Quoted 49 lines · Ben Greear

Quoted 11 lines · NeilBrown

Quoted 9 lines · On Thu, 31 Jul 2014 11:00:35 -0700 Ben Greear <[email protected]> wrote:

So, this has been asked all over the interweb for years and years, but the best answer I can find is to reboot the system or create a fake NFS server somewhere with the same IP as the gone-away NFS server.

The problem is:

I have some mounts to an NFS server that no longer exists (crashed/powered down).

I have some processes stuck trying to write to files open on these mounts.

Quoted 8 lines

Quoted 6 lines

I want to kill the process and unmount.

umount -l will make the mount go a way, sort of. But process is still hung. umount -f complains: umount2: Device or resource busy umount.nfs:
/mnt/foo: device is busy

kill -9 does not work on process.

Kill -1 should work (since about 2.6.25 or so).

That is -[ONE], right? Assuming so, it did not work for me.

Kernel is 3.14.4+, with some of extra patches, but probably nothing that influences this particular behaviour.

[root@lf1005-14010010 ~]# cat /proc/3805/stack [<ffffffff811371ba>] sleep_on_page+0x9/0xd [<ffffffff8113738e>] wait_on_page_bit+0x71/0x78

[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d [<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77 [<ffffffffa0f04734>]
nfs_file_fsync+0x37/0x83 [nfs] [<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b [<ffffffff811a8d4b>] vfs_fsync+0x17/0x19 [<ffffffffa0f05305>]
nfs_file_flush+0x6b/0x6f [nfs] [<ffffffff81183e46>] filp_close+0x3f/0x71 [<ffffffff8119c8ae>] __close_fd+0x80/0x98 [<ffffffff81183de5>]
SyS_close+0x1c/0x3e [<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b [<ffffffffffffffff>] 0xffffffffffffffff [root@lf1005-14010010 ~]# kill -1 3805 
[root@lf1005-14010010 ~]# cat /proc/3805/stack [<ffffffff811371ba>] sleep_on_page+0x9/0xd [<ffffffff8113738e>] wait_on_page_bit+0x71/0x78 
[<ffffffff8113769a>] filemap_fdatawait_range+0xa2/0x16d [<ffffffff8113780e>] filemap_write_and_wait_range+0x3b/0x77 [<ffffffffa0f04734>]
nfs_file_fsync+0x37/0x83 [nfs] [<ffffffff811a8d32>] vfs_fsync_range+0x19/0x1b [<ffffffff811a8d4b>] vfs_fsync+0x17/0x19 [<ffffffffa0f05305>]
nfs_file_flush+0x6b/0x6f [nfs] [<ffffffff81183e46>] filp_close+0x3f/0x71 [<ffffffff8119c8ae>] __close_fd+0x80/0x98 [<ffffffff81183de5>]
SyS_close+0x1c/0x3e [<ffffffff815c55f9>] system_call_fastpath+0x16/0x1b [<ffffffffffffffff>] 0xffffffffffffffff

Thanks, Ben

Quoted 11 lines

If it doesn't please report the kernel version and cat /proc/$PID/stack

for some processes that cannot be killed.

NeilBrown

Quoted 5 lines

Aside from bringing a fake NFS server back up on the same IP, is there any other way to get these mounts unmounted and the processes killed without rebooting?

Quoted 8 lines

Quoted 2 lines

Quoted 2 lines

Thanks, Ben

-- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

- --
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJT64dqAAoJELbHqkYeJT4OHC0IAIRB2A8v5msRhXrdd+ybvkwD NcOSYOhSsxCHIS5BR5CNLg89zipRuocVCbdLRdtbse8nspMq8PAiQJt3YOkGwzos ifcsgxouMUKfmLcFHtJ0maIkWMPIrttPvHJuw67gt7LbHLPsFjlrdrKPv6aGa95m 7mCkY/bRniiJYCxrCqixzQpuWfIyVal6FPGtmpydTVh6lq0y05vDEVB8lP5xGyes w+I/vJkGf9ddTIDasYJbLwUXECbN3makJxmHNAZf4slQMB5FNNnpeTOqL17u62cY F/do8m/zxzztibTZqjKHIhHGDw/huTyQWfRsQ0AA9Exu8/RZKhJlL2EeYlFJWJQ= =hNGY
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

On Wed, 13 Aug 2014 08:42:34 -0700 Ben Greear <[email protected]> wrote:

Hello!

Did you get a chance to look at the stacks below?

Quoted 1 line

Yes I did, and I replied on Date: Fri, 1 Aug 2014 07:50:53 +1000

Quoted 2 lines

The problem is that "fsync" and related functions are not killable. I think it is generally agreed that this is a bug, and that a fix would probably be accepted. I started working on one the other day but haven't got very hard yet (lots of other things to work on).