nfs locking problems maybe solved
Subsystems (2)
Hi all,
I may have solved the locking problems I experience every once in a while on my machines. It seems the nfs server needs access to the portmapper on the clients. So, I granted access in hosts.allow and that seems to have solved a lot of problems.
I experienced two kinds of locking problems:
1) sometimes rrdtool could not lock rrd files
rrdtool runs every 5 minutes on all clients, this problem always solved itself
2) sometimes it was impossible to lock any file on an entire filesystem and it would be impossible to release those locks even by unmounting the filesystem and mounting it again
This morning I started receiving mails from all client machines complaining they couldn't lock rrd files. The nfs server was spewing the following lines in its log:
May 21 09:25:02 ananke kernel: lockd: server voyager not responding, timed out May 21 09:25:02 ananke kernel: lockd: couldn't create RPC handle for
voyager
Meanwhile the client said:
May 21 09:09:25 voyager portmap[13799]: connect from 192.168.200.45 to
getport(nlockmgr): request from unauthorized host
So I granted 192.168.200.45 access to the portmapper on voyager, and most clients were unstuck. I updated hosts.allow on most clients to grant access to the portmapper now.
Now I still have about 4 clients which can't lock rrd files. Umounting the filesystem doesn't help. Umounting the filesystem for about 10 minutes doesn't help. So, I remounted the fs nolock on these machines.
So, now I've got problem #2 on these clients on the filesystem containing rrd files. Never had that before.
Problem #2 was always on the mail spool. Not sure if the problem is still there, however currently the clients do seem to be able to lock mailboxes.
I think problem #1 is gone now, I haven't received any mails of servers not being able to lock rrd files anymore, except for the 4 clients which are never ever able to lock those files.
So, in short: locking is still unreliable for me, but the situation has improved.
Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
Met vriendelijke groet,
Erik Hensema
Signature · 18 lines
HostingXS eXcellent Service Telefoon: 024 - 324 91 77 E-mail: hensema-yolSOB3tinz/[email protected] Website: http://www.hostingxs.nl/ ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead. http://vger.kernel.org/vger-lists.html#linux-nfs
A message that is not in the archive (sent off-list or not yet received): 200805211320.35610.hensema-yolSOB3tinz/[email protected]
Quoted 5 lines · Erik Hensema / HostingXS Internet Services
So I granted 192.168.200.45 access to the portmapper on voyager, and most clients were unstuck. I updated hosts.allow on most clients to grant access to the portmapper now.
Now I still have about 4 clients which can't lock rrd files.
...
Quoted 3 lines
Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
You need to grant access to the client's portmap, nlm and statd ports in order for the server to call back with locking operations. Unfortunately, only portmap is at a well-known port (111). The nlm and statd ports are dynamically selected, and receive random high numbered port values.
They are then advertised in portmap so if you are concerned about opening up an entire port range to the servers, you can see these with "rpcinfo -p". This command will show them under the names "nlockmgr" and "status" respectively. Remember these ports may change at every boot.
Also be aware that "statd" is a kernel process on opensuse. Most other distros have rpc.statd as a user process.
Tom.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
A message that is not in the archive (sent off-list or not yet received): RTPCLUEXC1-PRDqKwRP0000013f-rtwIt2gI0FxT+ZUat5FNkAK/[email protected]
Sorry - to be clear, the "rpcinfo -p" command must be run on the *client*, or directed to the client with "rpcinfo -p <clienthostname>", in order to show the client's current ports.
Tom.
Quoted 27 lines · Thomas
Quoted 5 lines · Erik Hensema / HostingXS Internet Services
So I granted 192.168.200.45 access to the portmapper on voyager, and most clients were unstuck. I updated hosts.allow on most clients to grant access to the portmapper now.
Now I still have about 4 clients which can't lock rrd files.
...
Quoted 3 lines
Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
You need to grant access to the client's portmap, nlm and statd ports in order for the server to call back with locking operations. Unfortunately, only portmap is at a well-known port (111). The nlm and statd ports are dynamically selected, and receive random high numbered port values.
They are then advertised in portmap so if you are concerned about opening up an entire port range to the servers, you can see these with "rpcinfo -p". This command will show them under the names "nlockmgr" and "status" respectively. Remember these ports may change at every boot.
Also be aware that "statd" is a kernel process on opensuse. Most other distros have rpc.statd as a user process.
Tom.
Quoted 11 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 17 lines · Thomas
Quoted 5 lines · Erik Hensema / HostingXS Internet Services
So I granted 192.168.200.45 access to the portmapper on voyager, and most clients were unstuck. I updated hosts.allow on most clients to grant access to the portmapper now.
Now I still have about 4 clients which can't lock rrd files.
...
Quoted 3 lines
Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
You need to grant access to the client's portmap, nlm and statd ports in order for the server to call back with locking operations. Unfortunately, only portmap is at a well-known port (111). The nlm and statd ports are dynamically selected, and receive random high numbered port values.
Quoted 8 lines
They are then advertised in portmap so if you are concerned about opening up an entire port range to the servers, you can see these with "rpcinfo -p". This command will show them under the names "nlockmgr" and "status" respectively. Remember these ports may change at every boot.
Also be aware that "statd" is a kernel process on opensuse. Most other distros have rpc.statd as a user process.
It's also possible to statically set ports for each of these services (though I'm not certain about SUSE's in-kernel statd). This is what you want to be doing if you're going to have firewalls between NFS servers and clients. With RH/Fedora distros, this is generally set up in /etc/sysconfig/nfs, but I'm not sure how SUSE does it...
Signature · 13 lines
Jeff Layton <[email protected]> ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead. http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 1 line · Thomas
At 07:20 AM 5/21/2008, Erik Hensema / HostingXS Internet Services
Quoted 8 lines · wrote:
Quoted 5 lines
So I granted 192.168.200.45 access to the portmapper on voyager, and most clients were unstuck. I updated hosts.allow on most clients to grant access to the portmapper now.
Now I still have about 4 clients which can't lock rrd files.
...
Quoted 8 lines
Quoted 2 lines
Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
You need to grant access to the client's portmap, nlm and statd ports in order for the server to call back with locking operations. Unfortunately, only portmap is at a well-known port (111). The nlm and statd ports are dynamically selected, and receive random high numbered port values.
That's no problem to me, I don't firewall ports on the internal network. I do have an "ALL : ALL" line in hosts.deny, so I explicitely have to open services using tcp_wrapper.
Met vriendelijke groet,
Erik Hensema
Signature · 18 lines
HostingXS eXcellent Service Telefoon: 024 - 324 91 77 E-mail: hensema-yolSOB3tinz/[email protected] Website: http://www.hostingxs.nl/ ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead. http://vger.kernel.org/vger-lists.html#linux-nfs
A message that is not in the archive (sent off-list or not yet received): 200805211527.10783.hensema-yolSOB3tinz/[email protected]
Quoted 3 lines · Erik Hensema / HostingXS Internet Services
That's no problem to me, I don't firewall ports on the internal network. I do have an "ALL : ALL" line in hosts.deny, so I explicitely have to open services using
tcp_wrapper.
Your followup reminds me that some kernels use local RPC to communicate between nlm and statd on the client as well. So, if you haven't included the local client host in your hosts.allow for at least these services, that may be advisable.
Tom.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 15 lines · Thomas
Quoted 5 lines · Erik Hensema / HostingXS Internet Services
So I granted 192.168.200.45 access to the portmapper on voyager, and most clients were unstuck. I updated hosts.allow on most clients to grant access to the portmapper now.
Now I still have about 4 clients which can't lock rrd files.
...
Quoted 2 lines
Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
You need to grant access to the client's portmap, nlm and statd ports in order for the server to call back with locking operations. Unfortunately, only portmap is at a well-known port (111). The nlm and statd ports are dynamically selected, and receive random high numbered port values.
Quoted 8 lines
They are then advertised in portmap so if you are concerned about opening up an entire port range to the servers, you can see these with "rpcinfo -p". This command will show them under the names "nlockmgr" and "status" respectively. Remember these ports may change at every boot.
Also be aware that "statd" is a kernel process on opensuse. Most other distros have rpc.statd as a user process.
"statd" is in-kernel only till OpenSUSE 10.2 and not in OpenSUSE 10.3. It's a userspace process in 10.3 and make sure it's running.
Thanks,
Signature · 13 lines
Suresh Jayaraman ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead. http://vger.kernel.org/vger-lists.html#linux-nfs