Skip to contentSkip to conversation

nfs-over-tcp still needs udp ports? (SLES 11)

NFS Frank Steiner 7 participants 20 messages May 7, 2009 – May 22, 2009 13 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I'm fighting with my firewall to get nfs-over-tcp through.

The server is outside the firewall, the client is inside. The firewall allows all tcp back-connections (without syn), no UDPs. Mount on the client side worked fine with kernel 2.6.16 in SLES 10.

Now when the NFS client is running SLES 11 with its kernel 2.6.27, the NFS server tries to make UDP connections from its ports 111 and 700 to different ports on the client.

If the client is running SLES 10 with 2.6.16, those connections are not tried from the server, no matter if the server runs 2.6.16 or 2.6.27.

So I've two questions:
1) Should nfs-over-tcp still use any UDP ports at all?
2) What has been changed in the client code between 2.6.16 and 2.6.27 that could cause this behaviour?

Is there a way to prevent those UDP connects?

cu,
Frank

Signature · 19 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw
Quoted 16 lines · Frank Steiner

Hi,

I'm fighting with my firewall to get nfs-over-tcp through.

The server is outside the firewall, the client is inside. The firewall allows all tcp back-connections (without syn), no UDPs. Mount on the client side worked fine with kernel 2.6.16 in SLES 10.

Now when the NFS client is running SLES 11 with its kernel 2.6.27, the NFS server tries to make UDP connections from its ports 111 and 700 to different ports on the client.

If the client is running SLES 10 with 2.6.16, those connections are not tried from the server, no matter if the server runs 2.6.16 or 2.6.27.

Quoted 6 lines

So I've two questions:
1) Should nfs-over-tcp still use any UDP ports at all?
2) What has been changed in the client code between 2.6.16 and 2.6.27 that could cause this behaviour?

Is there a way to prevent those UDP connects?

The default behaviour is to always try to use UDP to talk to mountd and the portmapper in order to minimize the number of ports that get left in the TIME_WAIT state. If you only want to use TCP, then you might try using '-omountproto=tcp'

Cheers
Trond

PS: Note that [email protected] is deprecated due to poor anti-spam filtering. You should rather send posts directly to [email protected]

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): 1241704326.4884.10.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#4 lore raw
Quoted 3 lines · Trond Myklebust

PS: Note that [email protected] is deprecated due to poor anti-spam filtering. You should rather send posts directly to [email protected]

Note that if you are trying to subscribe to [email protected], you'll get this message:

"This list is closed to new subscriptions. Please subscribe to linux-nfs-u79uwXL29TaiAVqoAR/[email protected] Send mail to [email protected] to subscribe."

[email protected] is misspelled; should be majordomo-u79uwXL29TaiAVqoAR/[email protected]

Best regards, ---
Peter Åstrand ThinLinc Chief Developer
Cendio AB http://www.cendio.com
Wallenbergs gata 4
583 30 Linköping Phone: +46-13-21 46 00

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#5 lore raw
In reply to Frank Steiner
Quoted 4 lines

Now when the NFS client is running SLES 11 with its kernel 2.6.27, the NFS server tries to make UDP connections from its ports 111 and 700 to different ports on the client.

Quoted 8 lines

If the client is running SLES 10 with 2.6.16, those connections are not tried from the server, no matter if the server runs 2.6.16 or 2.6.27.

So I've two questions:
1) Should nfs-over-tcp still use any UDP ports at all?
2) What has been changed in the client code between 2.6.16 and 2.6.27 that could cause this behaviour?

By default it is now using UDP to connect to mountd.

Quoted 1 line

Is there a way to prevent those UDP connects?

Yes, add "mountproto=tcp" to your mount options.

Regards,
Leonardo

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): c2d0b6ec0905070634p6888226cx5b2c8abae51cd1be-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#7 lore raw

Leonardo Chiquitto wrote

Quoted 3 lines
Quoted 1 line

Is there a way to prevent those UDP connects?

Yes, add "mountproto=tcp" to your mount options.

I was only aware of "proto=tcp" and indeed missed the mountproto option.

Thanks both Leonardo and Trond for your answers!

cu,
Frank

Signature · 19 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 10 lines · Frank Steiner

Leonardo Chiquitto wrote

Quoted 3 lines
Quoted 1 line

Is there a way to prevent those UDP connects?

Yes, add "mountproto=tcp" to your mount options.

I was only aware of "proto=tcp" and indeed missed the mountproto option.

Thanks both Leonardo and Trond for your answers!

There is one small caveat to using mountproto=tcp through firewalls:
while the mount will succeed, there are some side protocol exchanges which may not.

In particular, if you do NLM file locking, there is a server callback (NLM "granted") which the server may choose to issue via UDP. If this callback is not seen by the client due to firewall blocking, there may be a 30-second pause before a client retry unblocks the caller.

Also, the NSM (status monitor) exchanges are often performed via UDP. Again, if you are using NLM and the server reboots, the client may not become aware of this promptly, and lock reclaim will be affected.

OTOH, if your applications don't use locking on the NFS mounts, you'll probably be fine.

Tom.

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): 4a02ffdf.1ac1f10a.637d.ffffbc3a-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org

#11 lore raw
Quoted 5 lines · Tom Talpey

There is one small caveat to using mountproto=tcp through firewalls:
while the mount will succeed, there are some side protocol exchanges which may not.

Quoted 8 lines

In particular, if you do NLM file locking, there is a server callback (NLM "granted") which the server may choose to issue via UDP. If this callback is not seen by the client due to firewall blocking, there may be a 30-second pause before a client retry unblocks the caller.

Also, the NSM (status monitor) exchanges are often performed via UDP. Again, if you are using NLM and the server reboots, the client may not become aware of this promptly, and lock reclaim will be affected.

Sorry for the slight threadjack, but a question along those lines...

My understanding is that currently portmap will not bind any UDP NLM listeners unless there are UDP mounts on the machine. I know there are servers out there that will always speak NLM over UDP (netapp/ontap being the prominent one), and as a result there can be problems without firewalls. If servers are out there that will speak NLM over UDP regardless of the mount itself, shouldn't we be binding NLM/UDP regardless of the mount transport?

(Or did I miss this change being reverted a while back?)

-Aaron

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): e7ca40f70905070908p595c1d23gef745a122ae09caa-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#13 lore raw
Quoted 20 lines · Aaron Wiebe
Quoted 13 lines · Tom Talpey

There is one small caveat to using mountproto=tcp through firewalls:
while the mount will succeed, there are some side protocol exchanges which may not.

In particular, if you do NLM file locking, there is a server callback (NLM
"granted") which the server may choose to issue via UDP. If this callback
is not seen by the client due to firewall blocking, there may be a 30-second
pause before a client retry unblocks the caller.

Quoted 4 lines

Also, the NSM (status monitor) exchanges are often performed via UDP. Again, if you are using NLM and the server reboots, the client may not
become aware of this promptly, and lock reclaim will be affected.

Quoted 4 lines

Sorry for the slight threadjack, but a question along those lines...

My understanding is that currently portmap will not bind any UDP NLM listeners unless there are UDP mounts on the machine.

It's not portmap... lockd decides which listeners to start.

Quoted 8 lines

I know there
are servers out there that will always speak NLM over UDP (netapp/ontap being the prominent one), and as a result there can be problems without firewalls. If servers are out there that will speak NLM over UDP regardless of the mount itself, shouldn't we be binding NLM/UDP regardless of the mount transport?

(Or did I miss this change being reverted a while back?)

This change was reverted upstream; see commit 8c3916f4.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#14 lore raw
Quoted 21 lines · Chuck Lever
Quoted 20 lines · Aaron Wiebe
Quoted 17 lines · Tom Talpey

There is one small caveat to using mountproto=tcp through firewalls:
while the mount will succeed, there are some side protocol exchanges which may not.

In particular, if you do NLM file locking, there is a server callback (NLM
"granted") which the server may choose to issue via UDP. If this callback
is not seen by the client due to firewall blocking, there may be a 30-second
pause before a client retry unblocks the caller.

Also, the NSM (status monitor) exchanges are often performed via UDP. Again, if you are using NLM and the server reboots, the client may not
become aware of this promptly, and lock reclaim will be affected.

Quoted 6 lines
Quoted 4 lines

Sorry for the slight threadjack, but a question along those lines...

My understanding is that currently portmap will not bind any UDP NLM listeners unless there are UDP mounts on the machine.

It's not portmap... lockd decides which listeners to start.

Also, there is an important distinction between NLM and NLM *callbacks*. The NLM client follows the NFS protocol selection in many client kernels, i.e. if you mount with proto=tcp, you get both NFS and NLM over TCP.

The issue is NLM callbacks, which are used only in specific cases where clients take blocking locks, which actually need to block due to being already held by another client. The server replies over NLM (e.g. TCP) with an indication that a callback will arive later. But when the other lock is released, the callback comes on a second connection, initiated from the server back to the client and not on the original NLM channel. To make matters worse, some servers only ever perform the callback on UDP in order to simplify and reduce the overhead required.

If this callback doesn't arrive at the client, or arrives in such a way that it's not recognized (e.g traverses a NAT and therefore changes source IP), then the client only wakes up on a timer. The long pauses can be a real problem, and one which only arises occasionally - i.e. very hard to trace down.

Just something to be aware of... it's a day-one defect in the NLM protocol, actually.

Tom.

Quoted 8 lines
Quoted 8 lines

I know there
are servers out there that will always speak NLM over UDP (netapp/ontap being the prominent one), and as a result there can be problems without firewalls. If servers are out there that will speak NLM over UDP regardless of the mount itself, shouldn't we be binding NLM/UDP regardless of the mount transport?

Quoted 8 lines
Quoted 1 line

(Or did I miss this change being reverted a while back?)

This change was reverted upstream; see commit 8c3916f4.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): 4a031594.1c1d640a.6d45.5fed-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org

#16 lore raw
Quoted 2 lines · Tom Talpey
Quoted 1 line · Chuck Lever

It's not portmap... lockd decides which listeners to start.

Thanks Chuck and Tom - that clarifies it.

Take care,
-Aaron

Quoted 1 line

Also, there is an important distinction between NLM and NLM *callback=

s*.

Quoted 1 line

The NLM client follows the NFS protocol selection in many client kern=

els,

Quoted 1 line

i.e. if you mount with proto=3Dtcp, you get both NFS and NLM over TCP=

=2E

Quoted 2 lines

The issue is NLM callbacks, which are used only in specific cases whe=

re

Quoted 1 line

clients take blocking locks, which actually need to block due to bein=

g already

Quoted 1 line

held by another client. The server replies over NLM (e.g. TCP) with a=

n indication

Quoted 1 line

that a callback will arive later. But when the other lock is released=

, the callback

Quoted 1 line

comes on a second connection, initiated from the server back to the c=

lient and

Quoted 1 line

not on the original NLM channel. To make matters worse, some servers =

only

Quoted 1 line

ever perform the callback on UDP in order to simplify and reduce the =

overhead

Quoted 3 lines

required.

If this callback doesn't arrive at the client, or arrives in such a w=

ay that

Quoted 1 line

it's not recognized (e.g traverses a NAT and therefore changes source=

IP),

Quoted 1 line

then the client only wakes up on a timer. The long pauses can be a re=

al

Quoted 1 line

problem, and one which only arises occasionally - i.e. very hard to t=

race down.

Quoted 2 lines

Just something to be aware of... it's a day-one defect in the NLM pro=

tocol,

Quoted 8 lines

actually.

Tom.

Quoted 4 lines
Quoted 4 lines

=A0I know there
are servers out there that will always speak NLM over UDP (netapp/ontap being the prominent one), and as a result there can b=

e

Quoted 1 line
Quoted 1 line
Quoted 1 line

problems without firewalls. =A0If servers are out there that will s=

peak

Quoted 1 line
Quoted 1 line
Quoted 1 line

NLM over UDP regardless of the mount itself, shouldn't we be bindin=

g

Quoted 12 lines
Quoted 10 lines
Quoted 3 lines

NLM/UDP regardless of the mount transport?

(Or did I miss this change being reverted a while back?)

This change was reverted upstream; see commit 8c3916f4.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

-----------------------------------------------------------------------=
-------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! You= r
production scanning environment may not be a perfect world - but thanks=
to
Kodak, there's a perfect scanner to get the job done! With the NEW KODA= K i700
Series Scanner you'll get full speed at 300 dpi even with all image=20 processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#17 lore raw
In reply to a message not in the archive

Tom Talpey wrote

Quoted 11 lines

In particular, if you do NLM file locking, there is a server callback (NLM "granted") which the server may choose to issue via UDP. If this callback is not seen by the client due to firewall blocking, there may be a 30-second pause before a client retry unblocks the caller.

Also, the NSM (status monitor) exchanges are often performed via UDP. Again, if you are using NLM and the server reboots, the client may not become aware of this promptly, and lock reclaim will be affected.

OTOH, if your applications don't use locking on the NFS mounts, you'll probably be fine.

We do use locking on nfs mounts, so I wonder what that would mean for the firewall. Currently I see connections from the NFS server *from* port 700 and 111 (we've fixed mountd port to 700) to (it seems) arbitrary udp ports on the NFS clients.

Would that be enough to allow those? Or could the source ports be arbitrary with NLM, too? I.e., would we have to open all udp traffic from the NFS servers to all the NFS clients?

cu,
Frank

Signature · 19 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): [email protected]

#19 lore raw
Quoted 15 lines · Frank Steiner

Tom Talpey wrote

Quoted 11 lines

In particular, if you do NLM file locking, there is a server callback (NLM "granted") which the server may choose to issue via UDP. If this callback is not seen by the client due to firewall blocking, there may be a 30-second pause before a client retry unblocks the caller.

Also, the NSM (status monitor) exchanges are often performed via UDP. Again, if you are using NLM and the server reboots, the client may not become aware of this promptly, and lock reclaim will be affected.

OTOH, if your applications don't use locking on the NFS mounts, you'll probably be fine.

Quoted 8 lines

We do use locking on nfs mounts, so I wonder what that would mean for the firewall. Currently I see connections from the NFS server *from* port 700 and 111 (we've fixed mountd port to 700) to (it seems) arbitrary udp ports on the NFS clients.

Would that be enough to allow those? Or could the source ports be arbitrary with NLM, too? I.e., would we have to open all udp traffic from the NFS servers to all the NFS clients?

Most NFS servers allow you to pin the ports used by the lockd service. In Linux, the kernel boot parameters lockd.nlm_tcpport and lockd.nlm_udpport will suffice to do it for you (see linux/Documentation/kernel-parameters.txt).

Trond

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#20 lore raw
In reply to a message not in the archive
Quoted 15 lines · Frank Steiner

Tom Talpey wrote

Quoted 11 lines

In particular, if you do NLM file locking, there is a server callback (NLM "granted") which the server may choose to issue via UDP. If this callback is not seen by the client due to firewall blocking, there may be a 30-second pause before a client retry unblocks the caller.

Also, the NSM (status monitor) exchanges are often performed via UDP. Again, if you are using NLM and the server reboots, the client may not become aware of this promptly, and lock reclaim will be affected.

OTOH, if your applications don't use locking on the NFS mounts, you'll probably be fine.

Quoted 8 lines

We do use locking on nfs mounts, so I wonder what that would mean for the firewall. Currently I see connections from the NFS server *from* port 700 and 111 (we've fixed mountd port to 700) to (it seems) arbitrary udp ports on the NFS clients.

Would that be enough to allow those? Or could the source ports be arbitrary with NLM, too? I.e., would we have to open all udp traffic from the NFS servers to all the NFS clients?

The calls from the server to client port 111 are portmapper lookups, very likely while setting up the NSM callback that the server performs when it boots up. If the portmap call succeeds, then a second NSM call will occur, to whatever port the client NSM is using (700 sounds reasonable, but it's not guaranteed).

NLM callbacks on blocking lock collision will be similar, but will happen any time not just at server reboot. The server will resolve the client NLM daemon using port 111, then perform a call to the result on whatever port is returned.

There are very likely many ways to make these calls work through a firewall, but unfortunately not very many ways to guarantee it. The big problem is that the callbacks will be nearly silent on failure, so success is hard to detect!

Depending on how criticial the application is, I would make different recommendations. But the safest option from a reliability standpoint might be to, yes, open up all UDP traffic. Ouch. There are other ways, but they're likely to be more fragile, and not all server/client combinations might work with a single solution.

You might want to think through the DNS issues of this firewall, too. Are all the clients using hostnames in the same DNS domain as the server? And, do any of them use DHCP or other dynamic assignment? You want to be sure to use FQDNs as the client hostname(2), to ensure the server can always resolve them properly.

I made a presentation on some of the NSM issues a few years back for Connectathon, if you're brave :-) you can check out:
http://www.connectathon.org/talks06/talpey-cthon06-nsm.pdf

The very best solution, by the way, would be to use NFSv4. It has no side protocols, and therefore no UDP issue. It does have a callback connection from the server to the client, but is done with TCP and is configurable.

Tom.

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): 4a083d44.85c2f10a.4cf7.ffff85fb-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org

#22 lore raw
Quoted 57 lines · Tom Talpey
Quoted 27 lines · Frank Steiner

Tom Talpey wrote

Quoted 14 lines

In particular, if you do NLM file locking, there is a server callback (NLM
"granted") which the server may choose to issue via UDP. If this callback
is not seen by the client due to firewall blocking, there may be a 30-second
pause before a client retry unblocks the caller.

Also, the NSM (status monitor) exchanges are often performed via UDP.
Again, if you are using NLM and the server reboots, the client may not
become aware of this promptly, and lock reclaim will be affected.

Quoted 3 lines

OTOH, if your applications don't use locking on the NFS mounts, you'll
probably be fine.

We do use locking on nfs mounts, so I wonder what that would mean for the
firewall. Currently I see connections from the NFS server *from* port 700
and 111 (we've fixed mountd port to 700) to (it seems) arbitrary udp ports on the NFS clients.

Quoted 5 lines

Would that be enough to allow those? Or could the source ports be arbitrary
with NLM, too? I.e., would we have to open all udp traffic from the NFS
servers to all the NFS clients?

The calls from the server to client port 111 are portmapper lookups, very likely
while setting up the NSM callback that the server performs when it boots up.
If the portmap call succeeds, then a second NSM call will occur, to whatever
port the client NSM is using (700 sounds reasonable, but it's not guaranteed).

NLM callbacks on blocking lock collision will be similar, but will happen any time
not just at server reboot. The server will resolve the client NLM daemon using
port 111, then perform a call to the result on whatever port is returned.

There are very likely many ways to make these calls work through a firewall,
but unfortunately not very many ways to guarantee it. The big problem is
that the callbacks will be nearly silent on failure, so success is hard to detect!

Quoted 8 lines

Depending on how criticial the application is, I would make different recommendations. But the safest option from a reliability standpoint might be
to, yes, open up all UDP traffic. Ouch. There are other ways, but they're
likely to be more fragile, and not all server/client combinations might work
with a single solution.

You can specify a fixed listener port for both lockd and statd on your client. And of course rpcbind uses a well-known port number. So you should be able to open just those ports in your firewall.

Quoted 7 lines

You might want to think through the DNS issues of this firewall, too. Are
all the clients using hostnames in the same DNS domain as the server? And,
do any of them use DHCP or other dynamic assignment? You want to be sure to use FQDNs as the client hostname(2), to ensure the server can always resolve them properly.

It depends, though, on if the client and server are sending IP addresses or hostnames in their SM_NOTIFY requests.

The server will record the client's IP address in NLM requests and use that when sending NLM_GRANT requests. The server will look up the client's caller_name string and try to use that to call the client back for SM_NOTIFY. The mon_name the server uses in the SM_NOTIFY request will either be its IP address or a hostname. Usually this will match the hostname that the client used to contact it on the mount command.

If the client is behind a firewall and uses a private DNS service, the server will probably not be able to resolve the client's caller_name. I suppose one way to work around this is to provide a reasonable IP address to hostname binding for the client in the server's /etc/ hosts. Another way might be to ensure the server's kernel sends IP addresses and not hostnames to its statd. See /proc/sys/fs/nfs/ nsm_use_hostnames.

Quoted 29 lines

I made a presentation on some of the NSM issues a few years back for Connectathon, if you're brave :-) you can check out:
http://www.connectathon.org/talks06/talpey-cthon06-nsm.pdf

The very best solution, by the way, would be to use NFSv4. It has no side protocols, and therefore no UDP issue. It does have a callback connection from the server to the client, but is done with TCP and is configurable.

Tom.

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#23 lore raw
In reply to a message not in the archive

Tom, thanks for all your explanations! I've got to go through them carefully to make sure I understand every kind of side-effect with all the different protocols and callback-stuff etc.

NFSv4 could indeed be an option for us, so I will check that also, and I'm also going to read your talk.

Thanks a lot for caring!

cu,
Frank

Signature · 19 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs
#24 lore raw
In reply to a message not in the archive

Tom Talpey wrote

Quoted 4 lines

The very best solution, by the way, would be to use NFSv4. It has no side protocols, and therefore no UDP issue. It does have a callback connection from the server to the client, but is done with TCP and is configurable.

I've indeed switched our through-firewall-nfsservers to NFSv4 and the problems are gone. Thanks a lot for pointing me there! I only open port 2049/tcp and everything works.

However, I still see blocked connections on the firewall, coming from the NFS client to the NFS server:
...PROTO=TCP SPT=55598 DPT=111...
rpcinfo tells me the portmapper is running at port 111 (udp and tcp).

I didn't find a clear statement when googling if that should happen with NFSv4 or not. It doesn't seem to block the NFS share in any way, at least as far as I can see.

I wouldn't mind to open tcp port 111 to the NFS server. I'm just curios if that behaviour is correct or not with NFSv4.

cu,
Frank

Signature · 20 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment. 
http://p.sf.net/sfu/businessobjects
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): [email protected]

#26 lore raw
Quoted 10 lines · Frank Steiner

Tom Talpey wrote

Quoted 4 lines

The very best solution, by the way, would be to use NFSv4. It has no side protocols, and therefore no UDP issue. It does have a callback connection from the server to the client, but is done with TCP and is configurable.

I've indeed switched our through-firewall-nfsservers to NFSv4 and the problems are gone. Thanks a lot for pointing me there! I only open port 2049/tcp and everything works.

Great! If you want the full NFSv4 benefit, you'll also need to enable delegation callbacks, which requires a TCP port in the other direction. This port is chosen by the client, but the server makes the connection, so you'll need to configure it in both the client and the firewall.

The port is set by an NFS sysctl parameter named "nfs_callback_tcpport", which by default is 0 (any). You'll need to set it to some value with

sysctl -w fs.nfs.nfs_callback_tcpport = <value>

and also in your firewall from the server->client. The range is 0 to 65535, you can choose any convenient unused value (e.g. 2050).

BTW when NFSv4.1 is in use, this reverse connection will no longer occur. But that's a ways off in the distribution kernels.

Quoted 8 lines

However, I still see blocked connections on the firewall, coming from the NFS client to the NFS server:
...PROTO=TCP SPT=55598 DPT=111...
rpcinfo tells me the portmapper is running at port 111 (udp and tcp).

I didn't find a clear statement when googling if that should happen with NFSv4 or not. It doesn't seem to block the NFS share in any way, at least as far as I can see.

It could be any number of things, but probably harmless and properly ignored for NFSv4. I will guess it's something to do with the SLES11 client and other daemons.

Can you capture a trace of these messages with wireshark? The contents of the portmapper request the client is sending will tell us exactly what services it's trying to resolve.

Alternatively you could turn on kernel portmap debug and watch the log. I'm not sure if SLES11 has the "rpcdebug" command installed, but if so, on the client you could try

rpcdebug -m rpc -s bind

then watch the syslog.

Quoted 2 lines

I wouldn't mind to open tcp port 111 to the NFS server. I'm just curios if that behaviour is correct or not with NFSv4.

It's not part of NFSv4. But it could be a side effect of the client NFS implementation, which also supports v2 and v3 plus NSM, NLM, etc etc.

Tom.

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): 4a0d72a6.c5c2f10a.368f.5cc5-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org

#28 lore raw

Tom Talpey wrote

Quoted 5 lines

Great! If you want the full NFSv4 benefit, you'll also need to enable delegation callbacks, which requires a TCP port in the other direction. This port is chosen by the client, but the server makes the connection, so you'll need to configure it in both the client and the firewall.

Quoted 7 lines

The port is set by an NFS sysctl parameter named "nfs_callback_tcpport", which by default is 0 (any). You'll need to set it to some value with

sysctl -w fs.nfs.nfs_callback_tcpport = <value>

and also in your firewall from the server->client. The range is 0 to 65535, you can choose any convenient unused value (e.g. 2050).

Your mails really help :-) I've read many notes about that problem and that the firewall must be opened etc., but none stated how to fix those ports!

Quoted 3 lines

It could be any number of things, but probably harmless and properly ignored for NFSv4. I will guess it's something to do with the SLES11 client and other daemons.

Ok, good to know.

Quoted 5 lines

Can you capture a trace of these messages with wireshark? The contents of the portmapper request the client is sending will tell us exactly what services it's trying to resolve.

Quoted 7 lines

Alternatively you could turn on kernel portmap debug and watch the log. I'm not sure if SLES11 has the "rpcdebug" command installed, but if so, on the client you could try

rpcdebug -m rpc -s bind

then watch the syslog.

The server and clients in charge are still running SLES 10 SP2, and it doesn't have the rpcdebug command. However, SLES 11 has it, so I will check when all the servers and clients have been upgraded.

So far, thanks a lot again :-)
cu,
Frank

Signature · 19 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment. 
http://p.sf.net/sfu/businessobjects
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): [email protected]

#30 lore raw
Quoted 1 line · Frank Steiner

Tom Talpey wrote

...

Quoted 11 lines
Quoted 7 lines

Alternatively you could turn on kernel portmap debug and watch the log. I'm not sure if SLES11 has the "rpcdebug" command installed, but if so, on the client you could try

rpcdebug -m rpc -s bind

then watch the syslog.

The server and clients in charge are still running SLES 10 SP2, and it doesn't have the rpcdebug command. However, SLES 11 has it, so I will check when all the servers and clients have been upgraded.

If the rpcdebug command isn't available, you can alternatively set set the RPCDBG_BIND bit with

echo 32 >/proc/sys/sunrpc/rpc_debug

and clear it by echoing 0. Many other bits are accessible btw, defined in the kernel include/sunrpc/debug.h file.

Tom.

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): 4a115a4c.47c1f10a.53d0.ffff96cc-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org

#32 lore raw

Tom Talpey wrote

Quoted 7 lines

If the rpcdebug command isn't available, you can alternatively set set the RPCDBG_BIND bit with

echo 32 >/proc/sys/sunrpc/rpc_debug

and clear it by echoing 0. Many other bits are accessible btw, defined in the kernel include/sunrpc/debug.h file.

Thanks :-) Looking at the debug output I found it was a different program but NFS that was trying to contact the portmapper :-)

cu,
Frank

Signature · 19 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & 
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs