klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine
Subsystems (2)
Hi,
I suffer from a problem, where my diskless environment doesn't work anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 (2.6.27.21).
The diskless environment gets set up with an customized initrd via pxelinux. The initrd uses klibc's nfsmount, and I've tried both protocols: udp and tcp. It's not an iptables issue, since I can mount these directories from other (fully booted) systems just fine.
Using the debug version of klibc's nfsmount prints these messages:
short read: 0 < 28
Port for 100003/3[udp]: 0
short read: 0 < 28
Port for 100005/3[udp]: 0
NFS params:
server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp, mount_port = 627, nfs_port = 2049, flags = 00010282, rsize = 0, wsize = 0, timeo = 7, retrans = 3,
acreg (min,max) = (3,60), acdir (min,max) = (30,60), soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0
tcpdumping the communication shows, that for whatever reason, the client sends the mount request from source port 798 to destination port 627, but the server doesn't listen on 627:
# rpcinfo -p
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100005 1 udp 41656 mountd
100005 1 tcp 49063 mountd
100005 2 udp 41656 mountd
100005 2 tcp 49063 mountd
100005 3 udp 41656 mountd
100005 3 tcp 49063 mountd
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100021 1 udp 33144 nlockmgr
100021 3 udp 33144 nlockmgr
100021 4 udp 33144 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 tcp 57268 nlockmgr
100021 3 tcp 57268 nlockmgr
100021 4 tcp 57268 nlockmgr
100024 1 udp 39508 status
100024 1 tcp 45430 statusI've attached the relevant tcpdump section.
I would be very grateful, if some kind soul could enlight me here, as I'm fully puzzled..
Pete
Quoted 11 lines · Hans-Peter Jansen
Hi,
I suffer from a problem, where my diskless environment doesn't work anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 (2.6.27.21).
The diskless environment gets set up with an customized initrd via pxelinux. The initrd uses klibc's nfsmount, and I've tried both protocols: udp and tcp. It's not an iptables issue, since I can mount these directories from other (fully booted) systems just fine.
Quoted 8 lines
Using the debug version of klibc's nfsmount prints these messages:
short read: 0 < 28
Port for 100003/3[udp]: 0
short read: 0 < 28
Port for 100005/3[udp]: 0
NFS params:
server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp,mount_port= 627,nfs_port= 2049, flags = 00010282,
^^^^^^^^^^^^^^^^
Quoted 7 lines
rsize = 0, wsize = 0, timeo = 7, retrans = 3, acreg (min,max) = (3,60), acdir (min,max) = (30,60), soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0
tcpdumping the communication shows, that for whatever reason, the client sends the mount request from source port 798 to destination port 627, but the server doesn't listen on 627:
The question is why is klibc specifying mount port = 627?
Trond
Am Mittwoch, 15. April 2009 schrieb Trond Myklebust:
Quoted 27 lines · Hans-Peter Jansen
Quoted 11 lines
Hi,
I suffer from a problem, where my diskless environment doesn't work anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 (2.6.27.21).
The diskless environment gets set up with an customized initrd via pxelinux. The initrd uses klibc's nfsmount, and I've tried both
protocols: udp and tcp. It's not an iptables issue, since I can mount these directories from other (fully booted) systems just fine.Quoted 8 lines
Using the debug version of klibc's nfsmount prints these messages:
short read: 0 < 28
Port for 100003/3[udp]: 0
short read: 0 < 28
Port for 100005/3[udp]: 0
NFS params:
server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp,mount_port= 627,nfs_port= 2049, flags = 00010282,^^^^^^^^^^^^^^^^Quoted 4 lines
rsize = 0, wsize = 0, timeo = 7, retrans = 3, acreg (min,max) = (3,60), acdir (min,max) = (30,60), soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0
Quoted 5 lines
Quoted 3 lines
tcpdumping the communication shows, that for whatever reason, the client sends the mount request from source port 798 to destination port 627, but the server doesn't listen on 627:
The question is why is klibc specifying mount port = 627?
Well, this is exactly my problem. That's the mount port, but this one cannot be set to a manual value, thus the rpc stuff infront of it must have gone wrong.
[Update]
Issue solved. It was that silly rpcbind package (0.1.6+git20080930-5.2), that has gone wrong. Replacing it with portmap (6.0+git20070716-31.37) did it. Looks like this deserves a new bugzilla record.
Sorry for the churn and thanks for your care, Trond.
Pete
Quoted 33 lines · Hans-Peter Jansen
Am Mittwoch, 15. April 2009 schrieb Trond Myklebust:
Quoted 32 lines · Hans-Peter Jansen
Quoted 11 lines
Hi,
I suffer from a problem, where my diskless environment doesn't work anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 (2.6.27.21).
The diskless environment gets set up with an customized initrd via pxelinux. The initrd uses klibc's nfsmount, and I've tried both
protocols: udp and tcp. It's not an iptables issue, since I can mount these directories from other (fully booted) systems just fine.Quoted 8 lines
Using the debug version of klibc's nfsmount prints these messages:
short read: 0 < 28
Port for 100003/3[udp]: 0
short read: 0 < 28
Port for 100005/3[udp]: 0
NFS params:
server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp,mount_port= 627,nfs_port= 2049, flags = 00010282,^^^^^^^^^^^^^^^^Quoted 7 lines
rsize = 0, wsize = 0, timeo = 7, retrans = 3, acreg (min,max) = (3,60), acdir (min,max) = (30,60), soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0
tcpdumping the communication shows, that for whatever reason, the client sends the mount request from source port 798 to destination port 627, but the server doesn't listen on 627:
The question is why is klibc specifying mount port = 627?
nfs_root_ports() (in fs/nfs/nfsroot.c) sets the default mountd port to 627, I think only if the rpcbind doesn't work.
Quoted 8 lines
Well, this is exactly my problem. That's the mount port, but this one cannot be set to a manual value, thus the rpc stuff infront of it must have gone wrong.
[Update]
Issue solved. It was that silly rpcbind package (0.1.6+git20080930-5.2), that has gone wrong. Replacing it with portmap (6.0+git20070716-31.37) did it. Looks like this deserves a new bugzilla record.
When using rpcbind instead of portmapper, what does the output of "rpcinfo" look like on the server?
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
Quoted 3 lines
When using rpcbind instead of portmapper, what does the output of "rpcinfo" look like on the server?
It's dumped in the mail starting this thread.
Thanks,
Pete
Quoted 6 lines · Hans-Peter Jansen
Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
Quoted 3 lines
When using rpcbind instead of portmapper, what does the output of "rpcinfo" look like on the server?
It's dumped in the mail starting this thread.
I don't see why the client's rpcbind attempt for the server's mountd service should have failed.
Would it be possible for you to capture a packet trace of the client's attempt to mount it's root file system? (You will likely need to do this for a bugzilla report, anyway).
On the server you can use "tcpdump -s0 -w /tmp/raw host " followed by the hostname of your client. It might be useful to capture both a failed session (with rpcbind) and a working session (with portmapper). We want the raw (binary) trace dumps, not text output.
Also let us know what's running on your clients (distribution, kernel version, etc).
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
Quoted 9 lines · Hans-Peter Jansen
Quoted 5 lines
Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
Quoted 2 lines
When using rpcbind instead of portmapper, what does the output of "rpcinfo" look like on the server?
It's dumped in the mail starting this thread.
I don't see why the client's rpcbind attempt for the server's mountd service should have failed.
For completeness, here's the current rpcinfo view with portmap:
# rpcinfo -p 172.16.23.110
Program Vers Proto Port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100005 1 udp 54838 mountd
100005 1 tcp 32772 mountd
100005 2 udp 54838 mountd
100005 2 tcp 32772 mountd
100005 3 udp 54838 mountd
100005 3 tcp 32772 mountd
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100021 1 udp 35501 nlockmgr
100021 3 udp 35501 nlockmgr
100021 4 udp 35501 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 tcp 54766 nlockmgr
100021 3 tcp 54766 nlockmgr
100021 4 tcp 54766 nlockmgr
100024 1 udp 44650 status
100024 1 tcp 39765 statusQuoted 3 lines
Would it be possible for you to capture a packet trace of the client's attempt to mount it's root file system? (You will likely need to do this for a bugzilla report, anyway).
Ähem, Chuck, may I ask you to look into the initial mail again. The failing case is attached there. Here I've attached the good one. Since I couldn't locate any mount attempt in the dump, I've left a few more nfs transactions.
Quoted 2 lines
Also let us know what's running on your clients (distribution, kernel version, etc).
Hmm, sure. The client setup is a legacy SuSE 9.3 diskless environment (unfortunately Novell didn't manage to create a distribution with similar stability since then..., being a rpm junke, I will soon check Cent-OS (again)).
Client (relevant) versions:
Kernel: 2.6.11.4
Udev (nfsmount): 053
Let me know, what more I can provide, please.
THanks,
Pete