Skip to contentSkip to conversation

klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

DISCUSSION Hans-Peter Jansen 3 participants 7 messages Apr 15, 2009 4 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I suffer from a problem, where my diskless environment doesn't work anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 (2.6.27.21).

The diskless environment gets set up with an customized initrd via pxelinux. The initrd uses klibc's nfsmount, and I've tried both protocols: udp and tcp. It's not an iptables issue, since I can mount these directories from other (fully booted) systems just fine.

Using the debug version of klibc's nfsmount prints these messages:
short read: 0 < 28
Port for 100003/3[udp]: 0
short read: 0 < 28
Port for 100005/3[udp]: 0
NFS params:
server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp, mount_port = 627, nfs_port = 2049, flags = 00010282, rsize = 0, wsize = 0, timeo = 7, retrans = 3,
acreg (min,max) = (3,60), acdir (min,max) = (30,60), soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0

tcpdumping the communication shows, that for whatever reason, the client sends the mount request from source port 798 to destination port 627, but the server doesn't listen on 627:

# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  41656  mountd
    100005    1   tcp  49063  mountd
    100005    2   udp  41656  mountd
    100005    2   tcp  49063  mountd
    100005    3   udp  41656  mountd
    100005    3   tcp  49063  mountd
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100021    1   udp  33144  nlockmgr
    100021    3   udp  33144  nlockmgr
    100021    4   udp  33144  nlockmgr
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   tcp  57268  nlockmgr
    100021    3   tcp  57268  nlockmgr
    100021    4   tcp  57268  nlockmgr
    100024    1   udp  39508  status
    100024    1   tcp  45430  status

I've attached the relevant tcpdump section.

I would be very grateful, if some kind soul could enlight me here, as I'm fully puzzled..

Pete

#2 lore raw
In reply to a message not in the archive
Quoted 11 lines · Hans-Peter Jansen

Hi,

I suffer from a problem, where my diskless environment doesn't work anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 (2.6.27.21).

The diskless environment gets set up with an customized initrd via pxelinux. The initrd uses klibc's nfsmount, and I've tried both protocols: udp and tcp. It's not an iptables issue, since I can mount these directories from other (fully booted) systems just fine.

Quoted 8 lines

Using the debug version of klibc's nfsmount prints these messages:
short read: 0 < 28
Port for 100003/3[udp]: 0
short read: 0 < 28
Port for 100005/3[udp]: 0
NFS params:
server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp, mount_port = 627, nfs_port = 2049, flags = 00010282,

    ^^^^^^^^^^^^^^^^
Quoted 7 lines

rsize = 0, wsize = 0, timeo = 7, retrans = 3, acreg (min,max) = (3,60), acdir (min,max) = (30,60), soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0

tcpdumping the communication shows, that for whatever reason, the client sends the mount request from source port 798 to destination port 627, but the server doesn't listen on 627:

The question is why is klibc specifying mount port = 627?

Trond

#4 lore raw
In reply to a message not in the archive

Am Mittwoch, 15. April 2009 schrieb Trond Myklebust:

Quoted 27 lines · Hans-Peter Jansen
Quoted 11 lines

Hi,

I suffer from a problem, where my diskless environment doesn't work anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 (2.6.27.21).

The diskless environment gets set up with an customized initrd via pxelinux. The initrd uses klibc's nfsmount, and I've tried both
protocols: udp and tcp. It's not an iptables issue, since I can mount these directories from other (fully booted) systems just fine.

Quoted 8 lines

Using the debug version of klibc's nfsmount prints these messages:
short read: 0 < 28
Port for 100003/3[udp]: 0
short read: 0 < 28
Port for 100005/3[udp]: 0
NFS params:
server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp, mount_port = 627, nfs_port = 2049, flags = 00010282,

    ^^^^^^^^^^^^^^^^
Quoted 4 lines

rsize = 0, wsize = 0, timeo = 7, retrans = 3, acreg (min,max) = (3,60), acdir (min,max) = (30,60), soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0

Quoted 5 lines
Quoted 3 lines

tcpdumping the communication shows, that for whatever reason, the client sends the mount request from source port 798 to destination port 627, but the server doesn't listen on 627:

The question is why is klibc specifying mount port = 627?

Well, this is exactly my problem. That's the mount port, but this one cannot be set to a manual value, thus the rpc stuff infront of it must have gone wrong.

[Update]
Issue solved. It was that silly rpcbind package (0.1.6+git20080930-5.2), that has gone wrong. Replacing it with portmap (6.0+git20070716-31.37) did it. Looks like this deserves a new bugzilla record.

Sorry for the churn and thanks for your care, Trond.

Pete

#6 lore raw
In reply to a message not in the archive
Quoted 33 lines · Hans-Peter Jansen

Am Mittwoch, 15. April 2009 schrieb Trond Myklebust:

Quoted 32 lines · Hans-Peter Jansen
Quoted 11 lines

Hi,

I suffer from a problem, where my diskless environment doesn't work anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 (2.6.27.21).

The diskless environment gets set up with an customized initrd via pxelinux. The initrd uses klibc's nfsmount, and I've tried both
protocols: udp and tcp. It's not an iptables issue, since I can mount these directories from other (fully booted) systems just fine.

Quoted 8 lines

Using the debug version of klibc's nfsmount prints these messages:
short read: 0 < 28
Port for 100003/3[udp]: 0
short read: 0 < 28
Port for 100005/3[udp]: 0
NFS params:
server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp, mount_port = 627, nfs_port = 2049, flags = 00010282,

   ^^^^^^^^^^^^^^^^
Quoted 7 lines

rsize = 0, wsize = 0, timeo = 7, retrans = 3, acreg (min,max) = (3,60), acdir (min,max) = (30,60), soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0

tcpdumping the communication shows, that for whatever reason, the client sends the mount request from source port 798 to destination port 627, but the server doesn't listen on 627:

The question is why is klibc specifying mount port = 627?

nfs_root_ports() (in fs/nfs/nfsroot.c) sets the default mountd port to 627, I think only if the rpcbind doesn't work.

Quoted 8 lines

Well, this is exactly my problem. That's the mount port, but this one cannot be set to a manual value, thus the rpc stuff infront of it must have gone wrong.

[Update]
Issue solved. It was that silly rpcbind package (0.1.6+git20080930-5.2), that has gone wrong. Replacing it with portmap (6.0+git20070716-31.37) did it. Looks like this deserves a new bugzilla record.

When using rpcbind instead of portmapper, what does the output of "rpcinfo" look like on the server?

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#7 lore raw

Am Mittwoch, 15. April 2009 schrieb Chuck Lever:

Quoted 3 lines

When using rpcbind instead of portmapper, what does the output of "rpcinfo" look like on the server?

It's dumped in the mail starting this thread.

Thanks,
Pete

#9 lore raw
In reply to a message not in the archive
Quoted 6 lines · Hans-Peter Jansen

Am Mittwoch, 15. April 2009 schrieb Chuck Lever:

Quoted 3 lines

When using rpcbind instead of portmapper, what does the output of "rpcinfo" look like on the server?

It's dumped in the mail starting this thread.

I don't see why the client's rpcbind attempt for the server's mountd service should have failed.

Would it be possible for you to capture a packet trace of the client's attempt to mount it's root file system? (You will likely need to do this for a bugzilla report, anyway).

On the server you can use "tcpdump -s0 -w /tmp/raw host " followed by the hostname of your client. It might be useful to capture both a failed session (with rpcbind) and a working session (with portmapper). We want the raw (binary) trace dumps, not text output.

Also let us know what's running on your clients (distribution, kernel version, etc).

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#10 lore raw

Am Mittwoch, 15. April 2009 schrieb Chuck Lever:

Quoted 9 lines · Hans-Peter Jansen
Quoted 5 lines

Am Mittwoch, 15. April 2009 schrieb Chuck Lever:

Quoted 2 lines

When using rpcbind instead of portmapper, what does the output of "rpcinfo" look like on the server?

It's dumped in the mail starting this thread.

I don't see why the client's rpcbind attempt for the server's mountd service should have failed.

For completeness, here's the current rpcinfo view with portmap:

# rpcinfo -p 172.16.23.110
   Program Vers Proto   Port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  54838  mountd
    100005    1   tcp  32772  mountd
    100005    2   udp  54838  mountd
    100005    2   tcp  32772  mountd
    100005    3   udp  54838  mountd
    100005    3   tcp  32772  mountd
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100021    1   udp  35501  nlockmgr
    100021    3   udp  35501  nlockmgr
    100021    4   udp  35501  nlockmgr
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   tcp  54766  nlockmgr
    100021    3   tcp  54766  nlockmgr
    100021    4   tcp  54766  nlockmgr
    100024    1   udp  44650  status
    100024    1   tcp  39765  status
Quoted 3 lines

Would it be possible for you to capture a packet trace of the client's attempt to mount it's root file system? (You will likely need to do this for a bugzilla report, anyway).

Ähem, Chuck, may I ask you to look into the initial mail again. The failing case is attached there. Here I've attached the good one. Since I couldn't locate any mount attempt in the dump, I've left a few more nfs transactions.

Quoted 2 lines

Also let us know what's running on your clients (distribution, kernel version, etc).

Hmm, sure. The client setup is a legacy SuSE 9.3 diskless environment (unfortunately Novell didn't manage to create a distribution with similar stability since then..., being a rpm junke, I will soon check Cent-OS (again)).

Client (relevant) versions:
Kernel: 2.6.11.4
Udev (nfsmount): 053

Let me know, what more I can provide, please.

THanks,
Pete