mountd gives "illegal port"
Subsystems (2)
Hi all,
I have discovered the following problem in Linux NFS: When you mount many filesystems from the same server, mountd sometimes refuses the request with "illegal port" message in syslog. To reproduce this, I did the following:
On the NFS server:
# for i in `seq 1 2048`; do mkdir /tmp/nfs/$i; echo "$i" >/tmp/nfs/$i/index-$i.txt; echo "/tmp/nfs/$i *.fi.muni.cz(ro)"; done >/etc/exports # exportfs -a
On the NFS client;
# for i in `seq 1 2048`; do mkdir /tmp/nfs/$i; done # for i in `seq 1 100`; do echo $i; mount -t nfs -r nfs-server:/tmp/nfs/$i /tmp/nfs/$i || break; done
It fails after random number of iterations (seen 6 to 57) with
mount: nfs-server:/tmp/nfs/6 failed, reason given by server: Permission denied
And in NFS server the following is added to syslog:
Oct 27 14:28:48 nfs-server rpc.mountd: refused mount request from nfs-client.fi.muni.cz for /tmp/nfs/6 (/tmp/nfs/6): illegal port 54071
The situation seems a bit better when you add "sleep 5" to the mount loop. Sleep 2 does not help, though.
-Yenya
Signature · 3 lines
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |Quoted 2 lines
Whatever the Java applications and desktop dances may lead to, Unix will <
still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · Jan Kasprzak
I have discovered the following problem in Linux NFS: When you mount many filesystems from the same server, mountd sometimes refuses the request with "illegal port" message in syslog. To reproduce this, I did the following:
Just add the "insecure" export option to the server's export.
--Bruce Fields
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
J. Bruce Fields wrote:
: On Wed, Oct 27, 2004 at 02:42:28PM +0200, Jan Kasprzak wrote:
: > I have discovered the following problem in Linux NFS: When you mount : > many filesystems from the same server, mountd sometimes refuses the : > request with "illegal port" message in syslog. To reproduce this, I did : > the following:
:
: Just add the "insecure" export option to the server's export. :
Of course, but I don't want to do this. The problem is that client
is querying portmap over TCP and from privileged port. By changing this we can use twice as much mounts in the TIME_WAIT period. We may also force mount to query mountd over UDP, but I don't know how to do this. -o udp is probably for nfsd traffic, isn't it?
-Yenya
Signature · 3 lines
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |Quoted 2 lines
Whatever the Java applications and desktop dances may lead to, Unix will <
still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 10 lines · Jan Kasprzak
J. Bruce Fields wrote:
: On Wed, Oct 27, 2004 at 02:42:28PM +0200, Jan Kasprzak wrote:
: > I have discovered the following problem in Linux NFS: When you mount : > many filesystems from the same server, mountd sometimes refuses the : > request with "illegal port" message in syslog. To reproduce this, I did : > the following:
:
: Just add the "insecure" export option to the server's export. :
Of course, but I don't want to do this.
Why not?
Quoted 3 lines
The problem is that client is querying portmap over TCP and from privileged port. By changing this we can use twice as much mounts in the
TIME_WAITperiod.
OK. It seems like kind of an obscure thing to care about....
Quoted 2 lines
We may also force mount to query mountd over UDP, but I don't know how to do this. -o udp is probably for nfsd traffic, isn't it?
That's what I'd assume, but I don't know.--b.
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
J. Bruce Fields wrote:
: > :
: > : Just add the "insecure" export option to the server's export. : > :
: > Of course, but I don't want to do this.
:
: Why not?
:
NFS client is a host where ordinary (=evil :-) users have accounts.
-Y.
Signature · 3 lines
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |Quoted 2 lines
Whatever the Java applications and desktop dances may lead to, Unix will <
still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 5 lines · Jan Kasprzak
Of course, but I don't want to do this. The problem is that client
is querying portmap over TCP and from privileged port. By changing this we can use twice as much mounts in theTIME_WAITperiod. We may also force mount to query mountd over UDP, but I don't know how to do this. -o udp is probably for nfsd traffic, isn't it?
As far as I know, this has never been documented. This is somewhat related to the Fedora bug 128100: Even if you specify -o tcp, port= and mountport=, the mount command tries to "autodetect" if it should use TCP or UDP for the MOUNT protocol by contacting the portmapper (which fails in many cases, if you use SSH tunnels, for example).
Most people nowadays have realized that it was a mistake to separate the NFS and the MOUNT protocol. It's much more convenient to run them on the same port, using the same transport. With the current mountd/knfsd implementation it might be a little hard to use the same port number, but with unfsd, for example, it's easy to use the same port. Using the same transport type, though, is easy.
So, my opinion is that you should either use TCP for both the NFS and MOUNT protocol, or UDP for both.
Perhaps this "illegal port" problem can be solved by making the mount command use SO_REUSEADDR?
Signature · 15 lines
Peter Åstrand Chief Developer Cendio www.thinlinc.com Teknikringen 3 www.cendio.se 583 30 Linköping Phone: +46-13-21 46 00 ------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Jan Kasprzak wrote:
: I have discovered the following problem in Linux NFS: When you mount : many filesystems from the same server, mountd sometimes refuses the : request with "illegal port" message in syslog.
Well, the mountd is probably OK, and it is the client who runs
out of privileged ports. netstat -ta on the client looks like this:
tcp 0 0 nfs-client.fi.muni.cz:880 nfs-server.fi.muni.c:sunrpc TIME_WAIT tcp 0 0 nfs-client.fi.muni.cz:883 nfs-server.fi.muni.c:sunrpc TIME_WAIT tcp 0 0 nfs-client.fi.muni.cz:884 nfs-server.fi.muni.c:sunrpc TIME_WAIT tcp 0 0 nfs-client.fi.muni.cz:887 nfs-server.fi.muni.c:sunrpc TIME_WAIT tcp 0 0 nfs-client.fi.muni.cz:888 nfs-server.fi.muni.c:sunrpc TIME_WAIT tcp 0 0 nfs-client.fi.muni.cz:891 nfs-server.fi.muni.c:sunrpc TIME_WAIT
[...]
The solution might be to query portmap over UDP from the NFS client
instead of TCP. Or to query portmap from an unprivileged port (why we send query to portmap from privileged port anyway?).
I have even tried echo 2 > /proc/sys/net/ipv4/tcp_fin_timeout,
but this did not solve the problem - connections to server's portmap remain for 60 seconds in FIN_WAIT.
-Yenya
Signature · 3 lines
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |Quoted 2 lines
Whatever the Java applications and desktop dances may lead to, Unix will <
still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-----BEGIN PGP SIGNED MESSAGE-----
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Jan" == Jan Kasprzak <[email protected]> writes:
Jan> I have discovered the following problem in Linux NFS: When you
What version are you using?
On client? on server?
Jan> And in NFS server the following is added to syslog:
Jan> Oct 27 14:28:48 nfs-server rpc.mountd: refused mount request Jan> from nfs-client.fi.muni.cz for /tmp/nfs/6 (/tmp/nfs/6): illegal Jan> port 54071
Does your server expect to see reserved ports only (<1024). Can you do a tcpdump of the mounts?
To do this, on the server do:
rpcinfo -p localhost
note the port number of the mountd service. Let's say it is X.
tcpdump -s 1600 -i eth0 -n -p -w somefile.pcap udp port X or tcp port X
tcpdump -r somefile.pcap -T rpc -n
- --
] "Elmo went to the wrong fundraiser" - The Simpson | firewalls [ ] Michael Richardson, Xelerance Corporation, Ottawa, ON |net architect[ ] [email protected] http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys
iQCVAwUBQYANXoqHRg3pndX9AQGncAP/bj6qf9FcHKdu0I37JGPzisrdeiI9VBV7 wYjwONqRjfyhub+STaM5Upid8uwpzSw3rs1ixZJeXX5PklSmDz5XW0Gik/oauwWL OfPik8NEUsZNjSnS6sjaLCRRUC7PKAB3wTap6IMq/cskLjBTACJ6Gzj2vNYu9sSD 96BPplHUrcs=
=liY1
-----END PGP SIGNATURE-----
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Michael Richardson wrote:
: What version are you using?
: On client? on server?
:
Kernel 2.6.9 on both hosts (server is x86-64, client i386),
client nfs-utils-1.0.6-22 on Fedora Core 2, server nfs-utils-1.0.6-34 on Fedora Core 3 test 3.
: Jan> And in NFS server the following is added to syslog:
:
: Jan> Oct 27 14:28:48 nfs-server rpc.mountd: refused mount request : Jan> from nfs-client.fi.muni.cz for /tmp/nfs/6 (/tmp/nfs/6): illegal : Jan> port 54071
:
: Does your server expect to see reserved ports only (<1024).
Yes (there is no "insecure" option in /etc/exports).
: Can you do a tcpdump of the mounts?
:
Yes - I did. The failing mount request uses unprivileged source port
unlike the previous ones.
: tcpdump -s 1600 -i eth0 -n -p -w somefile.pcap udp port X or tcp port X
:
:
: tcpdump -r somefile.pcap -T rpc -n
I will do this on Monday - now I am not nowhere near these servers.
But as I said, the problem is that the client runs out of privileged
ports, because both portmap request and mountd request are done over TCP.
-Yenya
Signature · 3 lines
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |Quoted 2 lines
Whatever the Java applications and desktop dances may lead to, Unix will <
still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-----BEGIN PGP SIGNED MESSAGE-----
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Jan" == Jan Kasprzak <[email protected]> writes:
Jan> : Does your server expect to see reserved ports only (<1024).
Jan> Yes (there is no "insecure" option in /etc/exports).
Yes, the rest of the thread arrived.
Clearly, you are running out of reserved ports.
- --
] "Elmo went to the wrong fundraiser" - The Simpson | firewalls [ ] Michael Richardson, Xelerance Corporation, Ottawa, ON |net architect[ ] [email protected] http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys
iQCVAwUBQYAe1IqHRg3pndX9AQHbNwP9FAC+myGfExU3Cjl4phvRW48fpOUkOYci 80blUbA/Br+qOQvX+MMX52rSN0z7laQRV00LK7i1QXkk/uVUuwfIrxPVuZYIFm0S OSXHXA8jp6bH9ypcBivd9B4t1FvyLFb199VsyoMkav0JeJ1tornCQBKDK0TECfm5 yX6BwrP579k=
=FsWx
-----END PGP SIGNATURE-----
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs