Skip to contentSkip to conversation

NFS over a Win2k Server NAT Router with "DMZ": nfs: server not responding, still trying

DISCUSSION Michael Lackner 2 participants 7 messages Apr 27, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello!

Despite being new, i already have an interesting problem to solve (or i at least think so).

I work at a university, and we decided to place a router/firewall before 25 WinXP/Linux computers used by students. The router would also serve as software distribution server (MS IntelliMirror), so we decided to use MS Windows 2000 Server.

The server is routing with RRAS, and has 25 public IPs reserved and mapped to the private IPs of those 25 machines, so that a few people can directly connect to those PCs (in the future, the firewall on the router will decide who to let through and who to block). For outgoing connections from those 25 machines to the outside world, standard RRAS NAT is used.

Everything works so far, clients can connect to the outside world, and the Win2k Server
"DMZ" configuration works fine in letting our people through into the private pool using SSH. They just connect to the external IP (like 193.170.71.233) and are then routed to the corresponding private IP (like 10.20.15.233).

And here comes the problem:

Users' home directories are mounted via AutoFS/NFS at login time. Both users and AutoFS information are stored in an OpenLDAP database. The NFS server runs on the same machine as the LDAP server does, let's call it the "fileserver". This machine is a RedHat 9 Linux box with the stock NFS and OpenLDAP servers, Kernel 2.4.20. NFS
is v2 and v3, UDP only. rpcinfo -p <ip of fileserver> shows the services correctly,
no matter if i connect from the same network segment the NFS server is in, or from
my private network over the NAT router. OpenLDAP replies ok from within the private network too (tested with "id", reading user data out of the LDAP database and with "ldapsearch").

Now, i can mount directories from the fileserver, and i can cd into them. I can even cd
into subdirectories of a mounted directory! However, as soon as i try to "ls"/"dir" or
open a file, it's stuck. The last thing the NAT router shows is an outgoing translated
connection to the NFS server, port 2049. rpcinfo shows, that "nfs" is running on that port on the fileserver. But nfs seems to not respond to that translated connection.

The server is accessible from within its own network segment, totally ok, NFS mounts work. However, the described situation occurs, as soon as i try to mount and read NFS shares from within my "private" pool. All 25 machines show the same behavior.

dmesg and /var/log/messages say:
nfs: server <ip> not responding, still trying

Now you might think it's just the Win2000 Server NAT doing something wrong, so i set up another NFS server quickly, lets call him "testserver". This time i used a more modern Fedora Core 4 distribution, NFS is v2 and v3, UDP and TCP. This server is standing in
the outside world too.

I tried to mount an exported directory on the testserver from within my private network with the following parameters:
nfsver=2,udp
nfsver=3,udp
nfsver=2,tcp
nfsver=3,tcp

Works every time! I also tried those parameters on the fileserver. TCP of course didn't work at all, wouldn't let me mount anything, but with all the other parameters, i still got stuck when trying to ls within a mounted directory from the fileserver.

So, my best guest would have been to just update NFS on the fileserver, because a newer version seems to understand and correctly reply to my NAT'ed packages, that fly over the Win2000 Server Router.

But how can i update the server, without recompiling the kernel? As far as i understand it, the NFS support is compiled directly into the kernel, and there are no binary kernel updates for RedHat 9 anymore. Upgrading the whole machine to a newer distribution is also not acceptable, as this would mean breaking the license servers currently running on that machine as well as our backup solution.

To sum this up:

1.) Private IP Pool: 10.20.15.0/16, can connect to the internet (http, ftp etc.) through Win2k
router.
2.) Public IP Pool (changed for security reasons, but equivalent): 193.170.71.0/24
3.) Win2000 NAT Router with Interfaces 10.20.15.1/16 and 193.170.71.229/24
4.) Win2000 NAT Router reserves 25 addresses (193.170.71.230-193.170.71.254)
and maps those to 10.20.15.2 - 10.20.15.26, works for incoming SSH and ping, etc.
5.) not-working fileserver with NFS/LDAP is 193.179.71.208.
6.) working testserver with newer Linux and NFS is 193.171.71.207.
7.) For testing purposes, all firewalls and SELinux configurations have been disabled
on all machines!
/etc/exports and /etc/hosts.allow as well as /etc/hosts.deny have been checked on the fileserver to ensure it doesn't just drop my connections. 8.) Even serving a directory within the private IP Pool via NFS and mounting it from a machine outside WORKS... (tested this just because i didn't know what else to do anymore)

Is there a way to better analyze / fix this, without replacing the Microsoft router and without having to recompile the whole kernel (+additional nfs software?) on the fileserver or having to completely reinstall the fileserver?

What i really do not understand is, why the mounting works, why i can even cd in every subdirectory of the mounted nfs directory on the fileserver, but i can't read files or dirlistings.

I'm surely no NFS professional, so help for better understanding and probably solving this problem would be greatly appreciated! Frankly, i don't dare to touch the fileserver too much, as this is one hell of a critical machine...

Thanks in advance and sorry for writing that much.

Best Regards
Michael Lackner
University of Leoben
Chair for Information Technology, IT Administration

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#1 lore raw

Update: NFS over a Win2k Server NAT Router with "DMZ": nfs: server not responding, still trying

Greetings again.

A little update to the Situation as described in the mail before...

I now tried a "touch" on my NFS mounts, and to my surprise it worked. So i can even write to NFS mounted directories from within the private network. But i still cannot read...

When mounting soft, i get input/output errors when trying ls, no matter what values i set timeo and retrans to.. I also tried different values for rsize, down to 1024. Doesn't work either..

Thanks again Best Regards
Michael Lackner
University of Leoben
Chair for Information Technology, IT Administration

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw

Re: Update: NFS over a Win2k Server NAT Router with "DMZ": nfs: server not responding, still trying

Quoted 11 lines · Michael Lackner

Greetings again.

A little update to the Situation as described in the mail before...

I now tried a "touch" on my NFS mounts, and to my surprise it worked. So i can even write to NFS mounted directories from within the private network. But i still cannot read...

When mounting soft, i get input/output errors when trying ls, no matter what values i set timeo and retrans to.. I also tried different values for rsize, down to 1024. Doesn't work either..

If the problem is that READ and READDIR requests are getting lost, then it might be due to your firewall nixing the UDP fragments (recall that fragments don't have information about port numbers etc).

I'd suggest trying TCP instead.

Cheers,
Trond

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw

Re: Update: NFS over a Win2k Server NAT Router with "DMZ":nfs: server not responding, still trying

Hello!

Quoted 13 lines · Michael Lackner
Quoted 11 lines

Greetings again.

A little update to the Situation as described in the mail before...

I now tried a "touch" on my NFS mounts, and to my surprise it worked. So i can even write to NFS mounted directories from within the private network. But i still cannot read...

When mounting soft, i get input/output errors when trying ls, no matter what values i set timeo and retrans to.. I also tried different values for rsize, down to 1024. Doesn't work either..

Quoted 5 lines

If the problem is that READ and READDIR requests are getting lost, then it might be due to your firewall nixing the UDP fragments (recall that fragments don't have information about port numbers etc).

I'd suggest trying TCP instead.

Strangely, i can read/write directories from the newer "testserver" without problems even when using nfsver=2 and udp options.

I'm afraid, the nfs server on "fileserver" doesn't even support tcp. There should be
experimental tcp support for nfs in kernel 2.4.20 already, but i don't know if this feature is activated in RH9. How can i check if my nfs server supports tcp and how can i activate tcp, if so? By default, nfsd runs only on udp on that machine.

Thanks very much.

Best Regards
Michael Lackner
University of Leoben
Chair for Information Technology, IT Administration

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#4 lore raw

Re: Update: NFS over a Win2k Server NAT Router with "DMZ":nfs: server not responding, still trying

Quoted 4 lines · Michael Lackner

I'm afraid, the nfs server on "fileserver" doesn't even support tcp. There should be experimental tcp support for nfs in kernel 2.4.20 already, but i don't know if this feature is activated in RH9. How can i check if my nfs server supports tcp and how can i activate tcp, if so? By default, nfsd runs only on udp on that machine.

If TCP support has been compiled into the kernel, then it should be running, in which case 'rpcinfo -t <server> nfs' should confirm it.

Cheers,
Trond

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#5 lore raw

Re: Update: NFS over a Win2k Server NAT Router with "DMZ":nfs: server not responding, still trying

In reply to Trond Myklebust

Hello.

Maybe that might help resolving the issue? I did some tcpdump on client and fileserver. As expected, after the nfs readdir, there is no response coming from the fileserver.

On the fileserver itself i found this:
===
20:23:57.283012 CLIENTBEHINDNAT.3938904570 > FILESERVER.nfs: 164 readdirplus [|nfs] (DF) (ttl 63, id 9683, len 192)
20:23:57.283241 FILESERVER > CLIENTBEHINDNAT: udp (frag 9306:268@2960) (ttl 64, len 288)
20:23:57.283245 FILESERVER > CLIENTBEHINDNAT: udp (frag 9306:1480@1480+) (ttl 64, len 1500)
20:23:57.283249 FILESERVER.nfs > CLIENTBEHINDNAT.3938904570: reply ok 1472 readdirplus POST: DIR 40711 ids 505/505 [|nfs] (frag 9306:1480@0+) (ttl 64, len 1500) 20:23:57.284227 CLIENTBEHINDNAT > FILESERVER: icmp: CLIENTBEHINDNAT unreachable - need to frag (mtu 1500) (ttl 128, id 4037, len 56)
===

What's that ICMP doing there? I don't quite understand the meaning of that output, but it seems as if nfs is relying on that ICMP packets reaching their target to work? Or maybe i'm wrong and the router just drops the returning packages (thats what you meant?)

Quoted 2 lines

If TCP support has been compiled into the kernel, then it should be running, in which case 'rpcinfo -t <server> nfs' should confirm it.

rpcinfo -t <fileserver> nfs says:
==
rpcinfo: RPC: Program not registered
program 100003 is not available
==

However...

rpcinfo -p <fileserver> | grep nfs says:
==

   100003    2   udp   2049  nfs
   100003    3   udp   2049  nfs

==

(Server is definitly up, other machines from the same network segment as the server are mounting and reading/writing ok.)

Thanks.

Best Regards
Michael Lackner
University of Leoben
Chair for Information Technology, IT Administration

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#6 lore raw

Re: Update: NFS over a Win2k Server NAT Router with "DMZ":nfs: server not responding, still trying

Quoted 16 lines · Michael Lackner

Hello.

Maybe that might help resolving the issue? I did some tcpdump on client and fileserver. As expected, after the nfs readdir, there is no response coming from the fileserver.

On the fileserver itself i found this:
===
20:23:57.283012 CLIENTBEHINDNAT.3938904570 > FILESERVER.nfs: 164 readdirplus [|nfs] (DF) (ttl 63, id 9683, len 192) 20:23:57.283241 FILESERVER > CLIENTBEHINDNAT: udp (frag 9306:268@2960) (ttl 64, len 288)
20:23:57.283245 FILESERVER > CLIENTBEHINDNAT: udp (frag 9306:1480@1480+) (ttl 64, len 1500)
20:23:57.283249 FILESERVER.nfs > CLIENTBEHINDNAT.3938904570: reply ok 1472 readdirplus POST: DIR 40711 ids 505/505 [|nfs] (frag 9306:1480@0+) (ttl 64, len 1500)
20:23:57.284227 CLIENTBEHINDNAT > FILESERVER: icmp: CLIENTBEHINDNAT unreachable - need to frag (mtu 1500) (ttl 128, id 4037, len 56)
===

Quoted 4 lines

What's that ICMP doing there? I don't quite understand the meaning of that output, but it seems as if nfs is relying on that ICMP packets reaching their target to work? Or maybe i'm wrong and the router just drops the returning packages (thats what you meant?)

The ICMP message is telling the server that the fragments are not reaching the client.

If you can't use TCP, then I suggest you try mounting using rsize=1024. That means that the client will issue smaller read/readdir requests which should normally pass through the firewall without requiring fragmentation. You will see a significant performance penalty compared to clients that are behind the firewall, though...

Cheers,
Trond

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs