rpc.mountd at 99% cpu
Subsystems (2)
I've got an nfs server (quad cpu amd64, 36G mem, failover fiberchannel) running ubuntu, and I'm getting the following behavior that I haven't been able to track down yet:
* rpc.mountd spikes to 99% cpu usage when a client machine mounts, causing a temporary disruption in service to all client systems
Google, NFS Howtos, NFS Perf Tuning docs, IRC (can't find an NFS specific IRC channel), local sysadmins all turn up nothing so far. Here are potentially relavant data:
* exportfs -v -o rw,secure,sync,no_root_squash 10.10.0.0/255.255.0.0:/export/home
* nfs server has all clients info in /etc/hosts
* /etc/hosts is first in nsswitch.conf
* kernel is ubuntu's: linux-image-2.6.10-5-amd64-k8-smp
* nfs performance is fine once a filesystem is mounted
* unless, someone else is mounting a filesystem, in which case already mounted filesystems
* only 8 clients
* home directories mounted by autofs on clients as server:/export/home/bob /home/bob
* 32 nfsd threads
* % netstat -in
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP
TX-OVR Flg
eth0 1500 0 247854230 0 0 0324570001 0
0 0 BMRU
eth1 1500 0 1022171197 0 0 0258643880
0 0 0 BMR U
lo 16436 0 419024 0 0 0 419024 0
0 0 LRU
* % cat /proc/net/rpc/nfsd | grep ^th
th 32 8229 13759.142 6537.314 1919.479 4.212 129.977 52.636 11.780 10.231 0.000 101.321
* caching bind9 installed on server, and it points to itself for first nameserver entry
Any one know what's up here? Or how I can tell what's making rpc.mountd take so much time?
Signature · 14 lines
Brian Elliott Finley Linux Strategist, CIS Desk: 630.252.4742 Cell: 630.631.6621 ------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Brian,
is there a file system mounted in /proc/nfs/nfsd - then nfsd is working in a new mode? You can try *not* to mount it in the start/stop-script of the NFS server. This way you will force nsfd to operate in a legacy mode. If you followed my posts last week, this solved a problem with stale file handles for me.
Cheers - Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 50 lines
I've got an nfs server (quad cpu amd64, 36G mem, failover fiberchannel) running ubuntu, and I'm getting the following behavior that I haven't been able to track down yet:
* rpc.mountd spikes to 99% cpu usage when a client machine mounts, causing a temporary disruption in service to all client systems
Google, NFS Howtos, NFS Perf Tuning docs, IRC (can't find an NFS specific IRC channel), local sysadmins all turn up nothing so far. Here are potentially relavant data:
* exportfs -v -o rw,secure,sync,no_root_squash
10.10.0.0/255.255.0.0:/export/home * nfs server has all clients info in /etc/hosts * /etc/hosts is first in nsswitch.conf * kernel is ubuntu's: linux-image-2.6.10-5-amd64-k8-smp * nfs performance is fine once a filesystem is mounted * unless, someone else is mounting a filesystem, in which case already mounted filesystems * only 8 clients * home directories mounted by autofs on clients as server:/export/home/bob /home/bob * 32 nfsd threads * % netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 247854230 0 0 0324570001 0 0 0 BMRU eth1 1500 0 1022171197 0 0 0258643880 0 0 0 BMR U lo 16436 0 419024 0 0 0 419024 0 0 0 LRU * % cat /proc/net/rpc/nfsd | grep ^th th 32 8229 13759.142 6537.314 1919.479 4.212 129.977 52.636 11.780 10.231 0.000 101.321* caching bind9 installed on server, and it points to itself for first nameserver entry
Any one know what's up here? Or how I can tell what's making rpc.mountd take so much time?
--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621
Quoted 8 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Reuti,
Thanks for your reply, but I'm afraid /proc/nfs/nfsd is *not* currently mounted.
Cheers, -Brian
Quoted 87 lines · Reuti
Brian,
is there a file system mounted in
/proc/nfs/nfsd- then nfsd is working in a new
mode? You can try *not* to mount it in the start/stop-script of the NFS server.
This way you will force nsfd to operate in a legacy mode. If you followed my
posts last week, this solved a problem with stale file handles for me.Cheers - Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 50 lines
I've got an nfs server (quad cpu amd64, 36G mem, failover fiberchannel) running ubuntu, and I'm getting the following behavior that I haven't been able to track down yet:
* rpc.mountd spikes to 99% cpu usage when a client machine mounts, causing a temporary disruption in service to all client systems
Google, NFS Howtos, NFS Perf Tuning docs, IRC (can't find an NFS specific IRC channel), local sysadmins all turn up nothing so far. Here are potentially relavant data:
* exportfs -v -o rw,secure,sync,no_root_squash
10.10.0.0/255.255.0.0:/export/home * nfs server has all clients info in /etc/hosts * /etc/hosts is first in nsswitch.conf * kernel is ubuntu's: linux-image-2.6.10-5-amd64-k8-smp * nfs performance is fine once a filesystem is mounted * unless, someone else is mounting a filesystem, in which case already mounted filesystems * only 8 clients * home directories mounted by autofs on clients as server:/export/home/bob /home/bob * 32 nfsd threads * % netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 247854230 0 0 0324570001 0 0 0 BMRU eth1 1500 0 1022171197 0 0 0258643880 0 0 0 BMR U lo 16436 0 419024 0 0 0 419024 0 0 0 LRU * % cat /proc/net/rpc/nfsd | grep ^th th 32 8229 13759.142 6537.314 1919.479 4.212 129.977 52.636 11.780 10.231 0.000 101.321* caching bind9 installed on server, and it points to itself for first nameserver entry
Any one know what's up here? Or how I can tell what's making rpc.mountd take so much time?
--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 3 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 5 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Signature · 14 lines
Brian Elliott Finley Linux Strategist, CIS Desk: 630.252.4742 Cell: 630.631.6621 ------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Typo: the one I meant was /proc/fs/nfsd - is there anything in this directory?
-- Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 106 lines
Reuti,
Thanks for your reply, but I'm afraid
/proc/nfs/nfsdis *not* currently mounted.Cheers, -Brian
Quoted 79 lines · Reuti
Brian,
is there a file system mounted in
/proc/nfs/nfsd- then nfsd is working in a new
mode? You can try *not* to mount it in the start/stop-script of the NFS server.
This way you will force nsfd to operate in a legacy mode. If you followed my
posts last week, this solved a problem with stale file handles for me.Cheers - Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 50 lines
I've got an nfs server (quad cpu amd64, 36G mem, failover fiberchannel) running ubuntu, and I'm getting the following behavior that I haven't been able to track down yet:
* rpc.mountd spikes to 99% cpu usage when a client machine mounts, causing a temporary disruption in service to all client systems
Google, NFS Howtos, NFS Perf Tuning docs, IRC (can't find an NFS specific IRC channel), local sysadmins all turn up nothing so far. Here are potentially relavant data:
* exportfs -v -o rw,secure,sync,no_root_squash
10.10.0.0/255.255.0.0:/export/home * nfs server has all clients info in /etc/hosts * /etc/hosts is first in nsswitch.conf * kernel is ubuntu's: linux-image-2.6.10-5-amd64-k8-smp * nfs performance is fine once a filesystem is mounted * unless, someone else is mounting a filesystem, in which case already mounted filesystems * only 8 clients * home directories mounted by autofs on clients as server:/export/home/bob /home/bob * 32 nfsd threads * % netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 247854230 0 0 0324570001 0 0 0 BMRU eth1 1500 0 1022171197 0 0 0258643880 0 0 0 BMR U lo 16436 0 419024 0 0 0 419024 0 0 0 LRU * % cat /proc/net/rpc/nfsd | grep ^th th 32 8229 13759.142 6537.314 1919.479 4.212 129.977 52.636 11.780 10.231 0.000 101.321* caching bind9 installed on server, and it points to itself for first nameserver entry
Any one know what's up here? Or how I can tell what's making rpc.mountd take so much time?
--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 3 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 5 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 8 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621
Quoted 8 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Reuti,
/proc/fs/nfsd/ does exist. I'll go back and have a look at your posts.
-Brian
Quoted 129 lines · Reuti
Typo: the one I meant was
/proc/fs/nfsd- is there anything in this directory?-- Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 26 lines
Reuti,
Thanks for your reply, but I'm afraid
/proc/nfs/nfsdis *not* currently mounted.Cheers, -Brian
Quoted 16 lines · Reuti
Brian,
is there a file system mounted in
/proc/nfs/nfsd- then nfsd is working in a new
mode? You can try *not* to mount it in the start/stop-script of the NFS server.
This way you will force nsfd to operate in a legacy mode. If you followed my
posts last week, this solved a problem with stale file handles for me.Cheers - Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 1 line
I've got an nfs server (quad cpu amd64, 36G mem, failover
fiberchannel)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
running ubuntu, and I'm getting the following behavior that I haven't been able to track down yet:
* rpc.mountd spikes to 99% cpu usage when a client machine mounts, causing a temporary disruption in service to all client systems
Google, NFS Howtos, NFS Perf Tuning docs, IRC (can't find an NFS specific IRC channel), local sysadmins all turn up nothing so
far. Here
Quoted 17 lines
Quoted 17 lines
Quoted 4 lines
are potentially relavant data:
* exportfs -v -o rw,secure,sync,no_root_squash
10.10.0.0/255.255.0.0:/export/homeQuoted 13 lines
* nfs server has all clients info in
/etc/hosts
*/etc/hostsis first in nsswitch.conf
* kernel is ubuntu's: linux-image-2.6.10-5-amd64-k8-smp
* nfs performance is fine once a filesystem is mounted
* unless, someone else is mounting a filesystem, in which case already mounted filesystems
* only 8 clients
* home directories mounted by autofs on clients as server:/export/home/bob/home/bob
* 32 nfsd threads
* % netstat -in
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERRTX-DRP
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
TX-OVR Flg
eth0 1500 0 247854230 0 0 0324570001 0
0 0 BMRU
eth1 1500 0 1022171197 0 0 0258643880
0 0 0 BMR U
lo 16436 0 419024 0 0 0 419024 0
0 0 LRU
* % cat/proc/net/rpc/nfsd| grep ^thth 32 8229 13759.142 6537.314 1919.479 4.212 129.977 52.63611.780
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
10.231 0.000 101.321* caching bind9 installed on server, and it points to itself for first nameserver entry
Any one know what's up here? Or how I can tell what's making
rpc.mountd
Quoted 24 lines
Quoted 24 lines
Quoted 10 lines
take so much time?
--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 3 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 5 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 3 lines
Quoted 3 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 14 lines
Quoted 5 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 3 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 5 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Signature · 14 lines
Brian Elliott Finley Linux Strategist, CIS Desk: 630.252.4742 Cell: 630.631.6621 ------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
I've taken a quick look at your posts. My situation is a bit different:
* Using nfs v3 only
* /proc/fs/nfsd does exist, but it is not a mounted filesystem
* stale filehandles and errors don't seem to be an issue, only rpc.mountd monopolizing CPU during client mounting
Cheers, -Brian
Quoted 184 lines · Brian Elliott Finley
Reuti,
/proc/fs/nfsd/does exist. I'll go back and have a look at your posts.-Brian
Quoted 169 lines · Reuti
Typo: the one I meant was
/proc/fs/nfsd- is there anything in this directory?-- Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 32 lines
Reuti,
Thanks for your reply, but I'm afraid
/proc/nfs/nfsdis *not* currently mounted.Cheers, -Brian
Quoted 20 lines · Reuti
Brian,
is there a file system mounted in
/proc/nfs/nfsd- then nfsd is working in a new
mode? You can try *not* to mount it in the start/stop-script of the NFS server.
This way you will force nsfd to operate in a legacy mode. If you followed my
posts last week, this solved a problem with stale file handles for me.Cheers - Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 3 lines
I've got an nfs server (quad cpu amd64, 36G mem, failover
fiberchannel)
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
running ubuntu, and I'm getting the following behavior that I haven't been able to track down yet:
* rpc.mountd spikes to 99% cpu usage when a client machine mounts, causing a temporary disruption in service to all client systems
Google, NFS Howtos, NFS Perf Tuning docs, IRC (can't find an NFS specific IRC channel), local sysadmins all turn up nothing so
far. Here
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
are potentially relavant data:
* exportfs -v -o rw,secure,sync,no_root_squash
10.10.0.0/255.255.0.0:/export/home* nfs server has all clients info in
/etc/hosts
*/etc/hostsis first in nsswitch.conf
* kernel is ubuntu's: linux-image-2.6.10-5-amd64-k8-smp
* nfs performance is fine once a filesystem is mounted
* unless, someone else is mounting a filesystem, in which case already mounted filesystems
* only 8 clientsQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
* home directories mounted by autofs on clients as server:/export/home/bob
/home/bob
* 32 nfsd threads
* % netstat -in
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERRTX-DRP
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
TX-OVR Flg
eth0 1500 0 247854230 0 0 0324570001 0
0 0 BMRU
eth1 1500 0 1022171197 0 0 0258643880
0 0 0 BMR U
lo 16436 0 419024 0 0 0 419024 0
0 0 LRU
* % cat/proc/net/rpc/nfsd| grep ^thth 32 8229 13759.142 6537.314 1919.479 4.212 129.977 52.63611.780
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
10.231 0.000 101.321* caching bind9 installed on server, and it points to itself for first nameserver entry
Any one know what's up here? Or how I can tell what's making
rpc.mountd
Quoted 28 lines
Quoted 28 lines
Quoted 10 lines
take so much time?
--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 5 lines
Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 15 lines
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Signature · 14 lines
Brian Elliott Finley Linux Strategist, CIS Desk: 630.252.4742 Cell: 630.631.6621 ------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Yes, I als use v3. And there is no entry in "df" for this filesystem or so; if it is mounted in any of the scripts, to me it looks like you can check this only if there is anything inside /proc/fs/nfsd - then it's mounted.
But anyway, if you modify the script just to test the other mode of nsfd (have a look at "man 8 exportfs" - and mountd is involved there), you could check whether there is any difference.
-- Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 205 lines
I've taken a quick look at your posts. My situation is a bit different:
* Using nfs v3 only
*/proc/fs/nfsddoes exist, but it is not a mounted filesystem
* stale filehandles and errors don't seem to be an issue, only rpc.mountd monopolizing CPU during client mountingCheers, -Brian
Quoted 184 lines · Brian Elliott Finley
Reuti,
/proc/fs/nfsd/does exist. I'll go back and have a look at your posts.-Brian
Quoted 169 lines · Reuti
Typo: the one I meant was
/proc/fs/nfsd- is there anything in this directory?-- Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 32 lines
Reuti,
Thanks for your reply, but I'm afraid
/proc/nfs/nfsdis *not* currently mounted.Cheers, -Brian
Quoted 20 lines · Reuti
Brian,
is there a file system mounted in
/proc/nfs/nfsd- then nfsd is working in a new
mode? You can try *not* to mount it in the start/stop-script of the NFS server.
This way you will force nsfd to operate in a legacy mode. If you followed my
posts last week, this solved a problem with stale file handles for me.Cheers - Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 3 lines
I've got an nfs server (quad cpu amd64, 36G mem, failover
fiberchannel)
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
running ubuntu, and I'm getting the following behavior that I haven't been able to track down yet:
* rpc.mountd spikes to 99% cpu usage when a client machine mounts, causing a temporary disruption in service to all client systems
Google, NFS Howtos, NFS Perf Tuning docs, IRC (can't find an NFS specific IRC channel), local sysadmins all turn up nothing so
far. Here
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
are potentially relavant data:
* exportfs -v -o rw,secure,sync,no_root_squash
10.10.0.0/255.255.0.0:/export/home* nfs server has all clients info in
/etc/hosts
*/etc/hostsis first in nsswitch.conf
* kernel is ubuntu's: linux-image-2.6.10-5-amd64-k8-smp
* nfs performance is fine once a filesystem is mounted
* unless, someone else is mounting a filesystem, in which case already mounted filesystems
* only 8 clientsQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
* home directories mounted by autofs on clients as server:/export/home/bob
/home/bob
* 32 nfsd threads
* % netstat -in
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERRTX-DRP
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
TX-OVR Flg
eth0 1500 0 247854230 0 0 0324570001 0
0 0 BMRU
eth1 1500 0 1022171197 0 0 0258643880
0 0 0 BMR U
lo 16436 0 419024 0 0 0 419024 0
0 0 LRU
* % cat/proc/net/rpc/nfsd| grep ^thth 32 8229 13759.142 6537.314 1919.479 4.212 129.977 52.63611.780
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
10.231 0.000 101.321* caching bind9 installed on server, and it points to itself for first nameserver entry
Any one know what's up here? Or how I can tell what's making
rpc.mountd
Quoted 28 lines
Quoted 28 lines
Quoted 10 lines
take so much time?
--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 5 lines
Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 15 lines
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621
Quoted 8 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
I've tried mounting it, and see that rpc.mountd performance is just over 3 times better.
I notice that "df" doesn't show it, but "mount" does. Now that I've got it mounted, I get this:
% mount | grep nfsd
nfsd on /proc/fs/nfsd type nfsd (rw)
Now examining an strace of rpc.mountd.
-Brian
Quoted 237 lines · Reuti
Yes, I als use v3. And there is no entry in "df" for this filesystem or so; if
it is mounted in any of the scripts, to me it looks like you can check this
only if there is anything inside/proc/fs/nfsd- then it's mounted.But anyway, if you modify the script just to test the other mode of nsfd (have a
look at "man 8 exportfs" - and mountd is involved there), you could check whether there is any difference.-- Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 7 lines
I've taken a quick look at your posts. My situation is a bit different:
* Using nfs v3 only
*/proc/fs/nfsddoes exist, but it is not a mounted filesystem
* stale filehandles and errors don't seem to be an issue, only rpc.mountd monopolizing CPU during client mountingQuoted 29 lines
Cheers, -Brian
Quoted 24 lines · Brian Elliott Finley
Reuti,
/proc/fs/nfsd/does exist. I'll go back and have a look at your posts.-Brian
Quoted 12 lines · Reuti
Typo: the one I meant was
/proc/fs/nfsd- is there anything in this directory?-- Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 3 lines
Reuti,
Thanks for your reply, but I'm afraid
/proc/nfs/nfsdis *not*currently
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
mounted.
Cheers, -Brian
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines · Reuti
Brian,
is there a file system mounted in
/proc/nfs/nfsd- then nfsd isworking
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
in a new
mode? You can try *not* to mount it in the start/stop-script ofthe NFS
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
server.
This way you will force nsfd to operate in a legacy mode. If you followed my
posts last week, this solved a problem with stale file handlesfor me.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Cheers - Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I've got an nfs server (quad cpu amd64, 36G mem, failover
fiberchannel)
Quoted 1 line
Quoted 1 line
Quoted 1 line
running ubuntu, and I'm getting the following behavior that I
haven't
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
been able to track down yet:
* rpc.mountd spikes to 99% cpu usage when a client machine
mounts,
Quoted 133 lines
Quoted 114 lines
Quoted 114 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
causing a temporary disruption in service to all client systems
Google, NFS Howtos, NFS Perf Tuning docs, IRC (can't find an NFS specific IRC channel), local sysadmins all turn up nothing so
far. Here
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
are potentially relavant data:
* exportfs -v -o rw,secure,sync,no_root_squash
10.10.0.0/255.255.0.0:/export/home* nfs server has all clients info in
/etc/hosts
*/etc/hostsis first in nsswitch.conf
* kernel is ubuntu's: linux-image-2.6.10-5-amd64-k8-smp
* nfs performance is fine once a filesystem is mounted
* unless, someone else is mounting a filesystem, in which case already mounted filesystems
* only 8 clientsQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
* home directories mounted by autofs on clients as server:/export/home/bob
/home/bob
* 32 nfsd threads
* % netstat -in
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERRTX-DRP
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
TX-OVR Flg
eth0 1500 0 247854230 0 0 0324570001 0
0 0 BMRU
eth1 1500 0 1022171197 0 0 0258643880
0 0 0 BMR U
lo 16436 0 419024 0 0 0 419024 0
0 0 LRU
* % cat/proc/net/rpc/nfsd| grep ^thth 32 8229 13759.142 6537.314 1919.479 4.212 129.977 52.63611.780
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
10.231 0.000 101.321* caching bind9 installed on server, and it points to itself for first nameserver entry
Any one know what's up here? Or how I can tell what's making
rpc.mountd
Quoted 28 lines
Quoted 28 lines
Quoted 10 lines
take so much time?
--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 5 lines
Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 15 lines
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 10 lines
Quoted 7 lines
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 3 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 5 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Signature · 14 lines
Brian Elliott Finley Linux Strategist, CIS Desk: 630.252.4742 Cell: 630.631.6621 ------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Reuti, Dan,
I've resolved my problem, and I think I've solved it too.
The following details my process and result, in the hope that it may benefit others. Thanks to Dan and Reuti for their help.
Cheers, -Brian
<<<<<<<<<<>>>>>>>>>>
NFS Server Performance Troubleshooting
--------------------------------------
#
# 2005.09.30 Brian Elliott Finley
#
Problem:
- Poor performance in general
- Sometimes home directories not mounted on clients
- Bursty performance -- sometimes good, sometimes bad
- Performance got so bad at one point that nfs server service had to be restarted.
Initial Observations:
rpc.mountd and/or nfsd sometimes taking 99% of one CPU
Initial Tuning:
/etc/default/nfs-kernel-server: s/RPCNFSDCOUNT=8/RPCNFSDCOUNT=32/
See: http://nfs.sourceforge.net/nfs-howto/performance.html
(section 5.6)Further Observations:
rpc.mountd, nfsd, and ypserv (sometimes) tend to spike when a mount
request happens.
Timeouts in auto.master are set to 10s for 3 maps, and 160s for one
map, so mount requests happen very often. The autofs default timeout is 5m (300).
Some entries in /etc/exports use hostnames, some use IP addresses
(herring or no?)
client1 using these ext3 mount options: noatime,data=writeback. nfs01
not using these.
client1 using async nfs exports (nfs-utils-0.3.1-14.72). The default
export behavior for both NFS Version 2 and Version 3 protocols, used by exportfs in nfs-utils versions prior to nfs-utils-1.0.1 is "asynchronous". See:
http://nfs.sourceforge.net/nfs-howto/performance.html (section 5.9). nfs01 using synchronous Writes (nfs-kernel-server 1.0.6-3.1ubuntu1).
Second Level Tuning:
1.) Modify ext3 mount options to include noatime, and mount -o remount
each file system. No casually noticeable improvement.
2.) Try s/sync/async/ nfs01:/etc/exports for /export/home filesystem,
then re-export. No casually noticeable improvement. Changed back to sync. async is technically non-NFS compliant.
3.) Change all hostnames to IP addresses in /etc/exports, then
re-export. No casually noticeable imrovement.
4.) Some hostnames, some IPs show up in /var/lib/nfs/etab, despite
having only IPs listed in /etc/exports. Tried this:
"exportfs -v -o \
rw,secure,sync,no_root_squash \
10.10.0.0/255.255.0.0:/export/home"
Then unexported everything else. No improvement.
5.) Used Ethereal to sniff the net -- no obvious issues there. A
re-transmit of a mount request, but that is likely due to slow response time of rpc.mountd rather than a network problem.
6.) rpcinfo -p nfs01-priv.net.anl.gov from both nfs01, and from client
side look fine.
7.) No entries in /etc/hosts.{allow,deny} to get in the way.
8.) All client IP/Hostname combos in /etc/hosts on nfs01, so lookup
timeouts shouldn't be an issue, but tried commenting out nis on the hosts line in /etc/nsswitch.conf just in case. No improvement.
9.) Installed a bind9 daemon in a cacheing only config on nfs01, and
modified /etc/resolv.conf to look to itself first when performing a DNS resolution. No improvement.
10.) Modified auto.master umount timeout, and upped it from 10s to just
over 1h. This significantly decreased the frequency of the disruptions, but didn't address the underlying problem.
11.) Did this: "mount -t nfsd nfsd /proc/fs/nfsd" to try the "new
style" NFS for 2.6 series kernels. Without this, NFS behaves like it's on a 2.4 series kernel. There was no specific reason why this should cause an improvement, but it cut the following ls command (which forces an automount of the three filesystems in question) down from 30s to 7s.
sudo umount /home/{user1,user2,user3}
time ls -d /home/{user1,user2,user3}/.
12.) strace -p $(pidof rpc.mountd) didn't reveal anything particularly
odd, but it was done after /proc/fs/nfsd was mounted, and it did try to read /proc/fs/nfsd/filehandle. I take this as further indication that mounting /proc/fs/nfsd/ was key.
13.) client1 reboot determined necessary. I took this opportunity to do
a complete stop and restart of the NFS daemons and the portmapper. Immediately upon a clean restart of these two (no changes had actually been made to anything that should affect portmap, but restarted it anyway just to get a perfectly clean slate) mount performance improved from the 7s to execute "time ls -d /home/{user1,user2,user3}/." down to 0.030s.
Bingo!
14.) I modified /etc/fstab to mount /proc/fs/nfsd on future boots.
Conclusion:
Prior to Second Level Tuning step 13, the NFS daemons hadn't been restarted since the Initial Tuning, which only increased the number of nfsd threads. This means that Second Level Tuning steps 1 - 12 were performed without a restart of the nfs daemons.
Which step was key? Was it a combination of them? Well, I can't say with 100% certainty without backstepping and restarting the nfs daemons each time, which obviously causes an service outage, but seeing as how step number 11 was the only step that provoked any kind of improvement prior to restarting the daemons, and was the only part of the configuration that was clearly expected to change behavior in a significant way (2.4 series kernel style behavior vs. 2.6 series kernel style behavior), my current theory is this:
The mounting of /proc/fs/nfsd was key. And while the mounting of it had an immediate effect for certain nfs daemon functions, it didn't have an immediate effect for others. Once the nfs
daemons were completely flushed and restarted cleanly,
it had an effect on all relevant nfs daemon functions.
Quoted 275 lines · Brian Elliott Finley
I've tried mounting it, and see that rpc.mountd performance is just over 3 times better.
I notice that "df" doesn't show it, but "mount" does. Now that I've got it mounted, I get this:
% mount | grep nfsd
nfsd on/proc/fs/nfsdtype nfsd (rw)Now examining an strace of rpc.mountd.
-Brian
Quoted 9 lines · Reuti
Yes, I als use v3. And there is no entry in "df" for this filesystem or so; if
it is mounted in any of the scripts, to me it looks like you can check this
only if there is anything inside/proc/fs/nfsd- then it's mounted.But anyway, if you modify the script just to test the other mode of nsfd (have a
look at "man 8 exportfs" - and mountd is involved there), you couldcheck
Quoted 8 lines
whether there is any difference.
-- Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 1 line
I've taken a quick look at your posts. My situation is a bit
different:
Quoted 6 lines
Quoted 6 lines
* Using nfs v3 only
*/proc/fs/nfsddoes exist, but it is not a mounted filesystem
* stale filehandles and errors don't seem to be an issue, only rpc.mountd monopolizing CPU during client mountingQuoted 8 lines
Quoted 8 lines
Cheers, -Brian
Quoted 3 lines · Brian Elliott Finley
Reuti,
/proc/fs/nfsd/does exist. I'll go back and have a look at yourposts.
Quoted 57 lines
Quoted 14 lines
Quoted 14 lines
-Brian
Quoted 5 lines · Reuti
Typo: the one I meant was
/proc/fs/nfsd- is there anything in this directory?-- Reuti
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 3 lines
Reuti,
Thanks for your reply, but I'm afraid
/proc/nfs/nfsdis *not*currently
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
mounted.
Cheers, -Brian
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines · Reuti
Brian,
is there a file system mounted in
/proc/nfs/nfsd- then nfsd isworking
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
in a new
mode? You can try *not* to mount it in the start/stop-script ofthe NFS
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
server.
This way you will force nsfd to operate in a legacy mode. If you followed my
posts last week, this solved a problem with stale file handlesfor me.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines
Quoted 11 lines
Cheers - Reuti
Zitat von Brian Elliott Finley <[email protected]>:
Quoted 3 lines
I've got an nfs server (quad cpu amd64, 36G mem, failover
fiberchannel)
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
running ubuntu, and I'm getting the following behavior that I
haven't
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
been able to track down yet:
* rpc.mountd spikes to 99% cpu usage when a client machine
mounts,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
causing a temporary disruption in service to all client
systems
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Google, NFS Howtos, NFS Perf Tuning docs, IRC (can't find an NFS specific IRC channel), local sysadmins all turn up nothing so
far. Here
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
are potentially relavant data:
* exportfs -v -o rw,secure,sync,no_root_squash
10.10.0.0/255.255.0.0:/export/home* nfs server has all clients info in
/etc/hosts
*/etc/hostsis first in nsswitch.conf
* kernel is ubuntu's: linux-image-2.6.10-5-amd64-k8-smp
* nfs performance is fine once a filesystem is mounted
* unless, someone else is mounting a filesystem, in which case already mounted filesystems
* only 8 clientsQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
* home directories mounted by autofs on clients as server:/export/home/bob
/home/bob
* 32 nfsd threads
* % netstat -in
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERRTX-DRP
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
TX-OVR Flg
eth0 1500 0 247854230 0 0 0324570001 0
0 0 BMRU
eth1 1500 0 1022171197 0 0 0258643880
0 0 0 BMR U
lo 16436 0 419024 0 0 0 419024 0
0 0 LRU
* % cat/proc/net/rpc/nfsd| grep ^thth 32 8229 13759.142 6537.314 1919.479 4.212 129.977 52.636Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
11.780
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
10.231 0.000 101.321* caching bind9 installed on server, and it points to
itself for
Quoted 86 lines
Quoted 86 lines
Quoted 67 lines
Quoted 67 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
first nameserver entry
Any one know what's up here? Or how I can tell what's making
rpc.mountd
Quoted 28 lines
Quoted 28 lines
Quoted 10 lines
take so much time?
--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 5 lines
Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 15 lines
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 5 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 10 lines
Quoted 7 lines
Quoted 7 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621Quoted 10 lines
Quoted 3 lines
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads,discussions,
Quoted 5 lines
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs--
Brian Elliott Finley
Linux Strategist, CIS
Desk: 630.252.4742
Cell: 630.631.6621-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Signature · 14 lines
Brian Elliott Finley Linux Strategist, CIS Desk: 630.252.4742 Cell: 630.631.6621 ------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs