Kernel NFSd CPU hog?

Hello,

we have a strange NFS problem with a newly setup Linux server, and I hope someone here can help.

The symptom is that, slowly over time (speaking of several days up to 2 weeks), the kernel nfsd processes/threads consume more and more CPU until the system finally becomes unresponsive. We recorded system activity with sar, which shows that CPU (system) usage slowly rises after reboot from about 1% to nearly 100% over the course of several days. Load averages stay around 0.1-0.3 until 100% are reached, up to this point the problem is almost not noticable from the clients. Then load averages climb up to 30.0; at this point the system becomes more or less unusable and has to be restarted. 'top' output shows the CPU usage evenly distributed across all nfsd threads.

The system is a fairly recent, though entry level server with a Core i3 and 4G RAM, hosting the home directories for about 15-20 clients. CPU activity does not drop at night, when no clients are connected. It is running Debian 6.0 with linux 3.2.0 (from the backports repository), with nfs-utils 1.2.5 (also from the backports repository). I suspect that these backports might be the culprit, but since we need this kernel for other purposes, and I cannot reboot that machine during office hours, I'd rather not try going back to the official Debian kernel without good reasons. If there are known problems, I'd give it a try.

Quoted 16 lines · On Mon, 02 Jul 2012 11:43:48 +0200 Andreas Heinlein <[email protected]> wrote:

Hello,

we have a strange NFS problem with a newly setup Linux server, and I hope someone here can help.

The symptom is that, slowly over time (speaking of several days up to 2 weeks), the kernel nfsd processes/threads consume more and more CPU until the system finally becomes unresponsive. We recorded system activity with sar, which shows that CPU (system) usage slowly rises after reboot from about 1% to nearly 100% over the course of several days. Load averages stay around 0.1-0.3 until 100% are reached, up to this point the problem is almost not noticable from the clients. Then load averages climb up to 30.0; at this point the system becomes more or less unusable and has to be restarted. 'top' output shows the CPU usage evenly distributed across all nfsd threads.

Quoted 10 lines

The system is a fairly recent, though entry level server with a Core i3 and 4G RAM, hosting the home directories for about 15-20 clients. CPU activity does not drop at night, when no clients are connected. It is running Debian 6.0 with linux 3.2.0 (from the backports repository), with nfs-utils 1.2.5 (also from the backports repository). I suspect that these backports might be the culprit, but since we need this kernel for other purposes, and I cannot reboot that machine during office hours, I'd rather not try going back to the official Debian kernel without good reasons. If there are known problems, I'd give it a try.

Find the pid of one of the nfsd threads that's spinning, then get a stack trace from it:

     # cat /proc/<pidofnfsd>/stack

...that should give us some idea of what it's doing.

--
Jeff Layton <[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 34 lines · Jeff Layton

Quoted 16 lines · On Mon, 02 Jul 2012 11:43:48 +0200 Andreas Heinlein <[email protected]> wrote:

Hello,

we have a strange NFS problem with a newly setup Linux server, and I hope someone here can help.

The symptom is that, slowly over time (speaking of several days up to 2 weeks), the kernel nfsd processes/threads consume more and more CPU until the system finally becomes unresponsive. We recorded system activity with sar, which shows that CPU (system) usage slowly rises after reboot from about 1% to nearly 100% over the course of several days. Load averages stay around 0.1-0.3 until 100% are reached, up to this point the problem is almost not noticable from the clients. Then load averages climb up to 30.0; at this point the system becomes more or less unusable and has to be restarted. 'top' output shows the CPU usage evenly distributed across all nfsd threads.

Quoted 10 lines

The system is a fairly recent, though entry level server with a Core i3 and 4G RAM, hosting the home directories for about 15-20 clients. CPU activity does not drop at night, when no clients are connected. It is running Debian 6.0 with linux 3.2.0 (from the backports repository), with nfs-utils 1.2.5 (also from the backports repository). I suspect that these backports might be the culprit, but since we need this kernel for other purposes, and I cannot reboot that machine during office hours, I'd rather not try going back to the official Debian kernel without good reasons. If there are known problems, I'd give it a try.

Find the pid of one of the nfsd threads that's spinning, then get a stack trace from it:

    # cat /proc/<pidofnfsd>/stack

Quoted 8 lines

...that should give us some idea of what it's doing.

--
Jeff Layton <[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello,

I've run into the problem again, and did a 'watch cat /proc/<pidofnfsd>/stack'. It actually seems to be doing something, because the stack trace changes every now and then, but mostly looks like

[<c1038be2>] try_to_wake_up+0x144/0x14d
[<c1045d26>] lock_timer_base+0x19/0x34
[<c10462fd>] __mod_timer+0x10c/0x116
[<c1045d41>] process_timeout+0x0/0x5
[<f858a243>] svc_recv+0x2e2/0x698 [sunrpc]
[<c1038beb>] default_wake_function+0x0/0x8
[<f8640748>] nfsd+0x90/0x108 [nfsd]
[<f86406b8>] nfsd+0x0/0x108 [nfsd]
[<c105176b>] kthread+0x63/0x68
[<c1051708>] kthread+0x0/0x68
[<c12dadbe>] kernel_thread_helper+0x6/0x10

[<ffffffff>] 0xffffffff

Meanwhile, I've found a quite recent thread on this list named "3.0+ NFS issues", and within two links to Ubuntu bug reports
(https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/879334 and
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1006446) and

again to a kernel bug
(https://bugzilla.kernel.org/show_bug.cgi?id=40912), all suggesting

that this is indeed a kernel 3.0 problem.

So I will try going back to 2.6.32 and hope this issue gets fixed soon.

Thanks for your help!

Quoted 34 lines · Jeff Layton

Quoted 16 lines · On Mon, 02 Jul 2012 11:43:48 +0200 Andreas Heinlein <[email protected]> wrote:

Hello,

we have a strange NFS problem with a newly setup Linux server, and I hope someone here can help.

The symptom is that, slowly over time (speaking of several days up to 2 weeks), the kernel nfsd processes/threads consume more and more CPU until the system finally becomes unresponsive. We recorded system activity with sar, which shows that CPU (system) usage slowly rises after reboot from about 1% to nearly 100% over the course of several days. Load averages stay around 0.1-0.3 until 100% are reached, up to this point the problem is almost not noticable from the clients. Then load averages climb up to 30.0; at this point the system becomes more or less unusable and has to be restarted. 'top' output shows the CPU usage evenly distributed across all nfsd threads.

Quoted 10 lines

The system is a fairly recent, though entry level server with a Core i3 and 4G RAM, hosting the home directories for about 15-20 clients. CPU activity does not drop at night, when no clients are connected. It is running Debian 6.0 with linux 3.2.0 (from the backports repository), with nfs-utils 1.2.5 (also from the backports repository). I suspect that these backports might be the culprit, but since we need this kernel for other purposes, and I cannot reboot that machine during office hours, I'd rather not try going back to the official Debian kernel without good reasons. If there are known problems, I'd give it a try.

Find the pid of one of the nfsd threads that's spinning, then get a stack trace from it:

    # cat /proc/<pidofnfsd>/stack

Quoted 8 lines

...that should give us some idea of what it's doing.

--
Jeff Layton <[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello,

I've run into the problem again, and did a 'watch cat /proc/<pidofnfsd>/stack'. It actually seems to be doing something, because the stack trace changes every now and then, but mostly looks like

[<c1038be2>] try_to_wake_up+0x144/0x14d
[<c1045d26>] lock_timer_base+0x19/0x34
[<c10462fd>] __mod_timer+0x10c/0x116
[<c1045d41>] process_timeout+0x0/0x5
[<f858a243>] svc_recv+0x2e2/0x698 [sunrpc]
[<c1038beb>] default_wake_function+0x0/0x8
[<f8640748>] nfsd+0x90/0x108 [nfsd]
[<f86406b8>] nfsd+0x0/0x108 [nfsd]
[<c105176b>] kthread+0x63/0x68
[<c1051708>] kthread+0x0/0x68
[<c12dadbe>] kernel_thread_helper+0x6/0x10

[<ffffffff>] 0xffffffff

Meanwhile, I've found a quite recent thread on this list named "3.0+ NFS issues", and within two links to Ubuntu bug reports
(https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/879334 and
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1006446) and again to a kernel bug
(https://bugzilla.kernel.org/show_bug.cgi?id=40912), all suggesting that this is indeed a kernel 3.0 problem.

So I will try going back to 2.6.32 and hope this issue gets fixed soon.

Thanks for your help!

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html