Skip to contentSkip to conversation

NFS synchronization problems on Beowulf cluster

DISCUSSION Mario Storti 2 participants 7 messages Jul 17, 2006 – Jul 20, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

We have still problems with NFS in our Beowulf cluster. The original post is here

http://thread.gmane.org/gmane.linux.nfs/9564/focus=9566

The news are that we reproduced the problem on Scientific Linux 4.2 and also without VNFS. In addition we have taken a tethereal capture that may help in understanding the problem.

In brief we have a Beowulf class cluster built on Scientific Linux (Beryllium) 4.2, (kernel 2.6.9-22.0.1). The cluster is disk-less (nodes don't have hard disks) based on the Warewulf package. NFS traffic is reduced by using VNFS filesystems at the nodes. However we reproduced the problem in a configuration with disks at the nodes.

The problem is that for some files in user accounts if we make some modifications to the file in the server, this changes are not seen in the compute nodes. The NFS server is NFS3 and with the standard configuration (8 instances of the server and default parameters). The cluster has 20 nodes at this time, but we have made experiments with a `cloned' cluster and even with only two nodes the problem persist.

The experiment is as follows: We change a text file in a user account with Emacs and checking whether the change is seen in the compute nodes. Sometimes the change is immediately seen in the compute nodes, but many times some nodes don't see the change.

This happens even when the cluster is idle (no computing or network load in the slave nodes, neither on the server), so it's not a problem of performance. It is plainly working bad.

rpcinfo -p' and nfsstat' on the server and on the client can be found in the riginal post.

We tried several combinations for the export and mount options (sync/async...) with no success.

We made en experiment and you can find the tethereal log here

http://venus.ceride.gov.ar/~mstorti/tempo/tethereal-raw10.log.gz

The process is as follows: we start with a file `foo5' and we verify that two compute nodes (node1 and node11) see the same file. After that we modify the file in the server and look what the compute nodes see. I verify that node1 sees (correctly) the change, while `node11' doesn't see the change.

These are the operations

* Modify file foo5 in server, adding a line with 'jaja27'
* grep jaja in node1 -> reports `jaja27' (OK)
* grep jaja in node11 -> reports `jaja27' (OK)
* Modify file foo5 in server, replace 'jaja27' by 'jaja28'
* grep jaja in node1 -> reports `jaja28' (OK)
* grep jaja in node11 -> reports `jaja27' (NOT OK)

These are the shell commands:
================================================================
[[turn on tethereal]]
[[Emacs writes #jaja27 to foo5 on server]]
[mstorti@aquiles ~]$ rsh node1 grep jaja foo5

# jaja27
[mstorti@aquiles ~]$ rsh node11 grep jaja foo5
# jaja27
                            [[Emacs changes 27 to 28 in foo5 on server]]
[mstorti@aquiles ~]$ rsh node1 grep jaja foo5
# jaja28
[mstorti@aquiles ~]$ rsh node11 grep jaja foo5
# jaja27
[mstorti@aquiles ~]$
                            [[turn off tethereal]]
================================================================

I examined the frames in the tethereal log and I found the following.

first rsh from node1: frame 183: GETATTR reply mtime:
21:36:21.975422000

first rsh from node11: frame 378: GETATTR reply mtime:
21:36:21.975422000

second rsh from node1: frame 564: GETATTR reply mtime:
21:37:37.515422000

                       frame 566: READ

second rsh from node11: frame 713: LOOKUP reply mtime:
21:37:37.515422000

Note: the inode for the file /u/mstorti/foo is 11590453.

So, during the second rsh from both nodes the clients receives an mtime for the file that is posterior to the one during the first read. The client at node1 correctly requires the new file contents with a READ, while the client from node11 doesn't. I assume that the client uses the contents in the cache (which is wrong).

Sorry for the way long post. Any help is appreciated,

TIA, Mario

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#1 lore raw
Quoted 24 lines · Mario Storti

Hi all,

We have still problems with NFS in our Beowulf cluster. The original post is here

http://thread.gmane.org/gmane.linux.nfs/9564/focus=9566

The news are that we reproduced the problem on Scientific Linux 4.2 and also without VNFS. In addition we have taken a tethereal capture that may help in understanding the problem.

In brief we have a Beowulf class cluster built on Scientific Linux (Beryllium) 4.2, (kernel 2.6.9-22.0.1). The cluster is disk-less (nodes don't have hard disks) based on the Warewulf package. NFS traffic is reduced by using VNFS filesystems at the nodes. However we reproduced the problem in a configuration with disks at the nodes.

The problem is that for some files in user accounts if we make some modifications to the file in the server, this changes are not seen in the compute nodes. The NFS server is NFS3 and with the standard configuration (8 instances of the server and default parameters). The cluster has 20 nodes at this time, but we have made experiments with a `cloned' cluster and even with only two nodes the problem persist.

Quoted 5 lines

The experiment is as follows: We change a text file in a user account with Emacs and checking whether the change is seen in the compute nodes. Sometimes the change is immediately seen in the compute nodes, but many times some nodes don't see the change.

This is a FAQ that has been well-publicised and discussed to death on this list:
http://nfs.sourceforge.net/#faq_a8

Trond

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw
Quoted 35 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On Mon, 17 Jul 2006 14:49:18 -0400,
Trond Myklebust <[email protected]> said:

Quoted 24 lines · Mario Storti

Hi all,

We have still problems with NFS in our Beowulf cluster. The original post is here

http://thread.gmane.org/gmane.linux.nfs/9564/focus=9566

The news are that we reproduced the problem on Scientific Linux 4.2 and also without VNFS. In addition we have taken a tethereal capture that may help in understanding the problem.

In brief we have a Beowulf class cluster built on Scientific Linux (Beryllium) 4.2, (kernel 2.6.9-22.0.1). The cluster is disk-less (nodes don't have hard disks) based on the Warewulf package. NFS traffic is reduced by using VNFS filesystems at the nodes. However we reproduced the problem in a configuration with disks at the nodes.

The problem is that for some files in user accounts if we make some modifications to the file in the server, this changes are not seen in the compute nodes. The NFS server is NFS3 and with the standard configuration (8 instances of the server and default parameters). The cluster has 20 nodes at this time, but we have made experiments with a `cloned' cluster and even with only two nodes the problem persist.

Quoted 5 lines

The experiment is as follows: We change a text file in a user account with Emacs and checking whether the change is seen in the compute nodes. Sometimes the change is immediately seen in the compute nodes, but many times some nodes don't see the change.

Quoted 3 lines

This is a FAQ that has been well-publicised and discussed to death on this list:
http://nfs.sourceforge.net/#faq_a8

I tried to mount with nocto' and noac' options without success. Also tried acregmax=0,acdirmax=0. We are trying to determine if there is some mistake in our configuration or if this is a normal' behavior. Your post seems to suggest that this is a normal' behavior. However, previously we had a cluster with RedHat 7.1 (kernel 2.2.16-22, NFS version 2) and we didn't had problems like this. We tried to use NFS version 2 on the new cluster and the problem persists. So I think that this is problem with the configuration and we are still trying to solve the problem tuning the mount/server options.

Thanks for your help,

Mario

-------------------------
Mario Alberto Storti
e-mail: mstorti at intec dot unl dot edu dot ar
http://www.cimec.org.ar/mstorti
-------------------------

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw
Quoted 4 lines · Mario Storti
Quoted 3 lines

This is a FAQ that has been well-publicised and discussed to death on this list:
http://nfs.sourceforge.net/#faq_a8

Quoted 9 lines

I tried to mount with nocto' and noac' options without success. Also tried acregmax=0,acdirmax=0. We are trying to determine if there is some mistake in our configuration or if this is a normal' behavior. Your post seems to suggest that this is a normal' behavior. However, previously we had a cluster with RedHat 7.1 (kernel 2.2.16-22, NFS version 2) and we didn't had problems like this. We tried to use NFS version 2 on the new cluster and the problem persists. So I think that this is problem with the configuration and we are still trying to solve the problem tuning the mount/server options.

Please reread the article. This is _normal_ behaviour. NFS has never been guaranteed to be cache coherent in this way.

NFS checks its data cache on calls to open()/opendir(), and on taking a lock. That is all.

Cheers,
Trond

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#4 lore raw
Quoted 19 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On Tue, 18 Jul 2006 09:48:51 -0400,
Trond Myklebust <[email protected]> said:

Quoted 4 lines · Mario Storti
Quoted 3 lines

This is a FAQ that has been well-publicised and discussed to death on this list:
http://nfs.sourceforge.net/#faq_a8

Quoted 9 lines

I tried to mount with nocto' and noac' options without success. Also tried acregmax=0,acdirmax=0. We are trying to determine if there is some mistake in our configuration or if this is a normal' behavior. Your post seems to suggest that this is a normal' behavior. However, previously we had a cluster with RedHat 7.1 (kernel 2.2.16-22, NFS version 2) and we didn't had problems like this. We tried to use NFS version 2 on the new cluster and the problem persists. So I think that this is problem with the configuration and we are still trying to solve the problem tuning the mount/server options.

Quoted 5 lines

Please reread the article. This is _normal_ behaviour. NFS has never been guaranteed to be cache coherent in this way.

NFS checks its data cache on calls to open()/opendir(), and on taking a lock. That is all.

I tried to do a ls -l' of the directory and the contents of the files are still out of sync. (I assume that ls -l' does an `opendir()' call.)

Perhaps these questions are too basic for this list. In that case, please redirect me to the appropriated list or newsgroup.

Thanks again, Mario

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#5 lore raw
Quoted 5 lines · Mario Storti

I tried to do a ls -l' of the directory and the contents of the files are still out of sync. (I assume that ls -l' does an `opendir()' call.)

Perhaps these questions are too basic for this list. In that case, please redirect me to the appropriated list or newsgroup.

'ls' does an opendir(), and should normally check that the cache is up to date. If that is not working, then something is indeed broken.

Which mount options are you seeing this with? What kind of server are you using (if it is a Linux server, please could you specify the kernel version and the filesystem).

Also please could you tell us if you can reproduce the problem on a more recent kernel version (2.6.9 is several years old).

Cheers,
Trond

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#6 lore raw
Quoted 14 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On Thu, 20 Jul 2006 07:04:24 -0400,
Trond Myklebust <[email protected]> said:

Quoted 5 lines · Mario Storti

I tried to do a ls -l' of the directory and the contents of the files are still out of sync. (I assume that ls -l' does an `opendir()' call.)

Perhaps these questions are too basic for this list. In that case, please redirect me to the appropriated list or newsgroup.

'ls' does an opendir(), and should normally check that the cache is up to date. If that is not working, then something is indeed broken.

Quoted 6 lines

Which mount options are you seeing this with? What kind of server are you using (if it is a Linux server, please could you specify the kernel version and the filesystem).

Also please could you tell us if you can reproduce the problem on a more recent kernel version (2.6.9 is several years old).

Initially we built a Warewulf cluster built on Linux Fedora Core 3 (kernel 2.6.15). The cluster is disk-less (nodes don't have hard disks). On the server the filesystem was XFS with Raid. On Warewulf systems NFS traffic is reduced by using VNFS filesystems (RAM disks containing /bin, /lib, etc...) at the nodes.

As we detected the problem, we first suspected from XFS and Raid, so we built a second system with FC3 but plain ext3 instead of XFS and no RAID. The problem persisted.

Then we switched to Scientific Linux 4.2 (It is equivalent to Red Hat Enterprise Linux (RHEL) 4.2). Error persisted.

Also we suspected of the Warewulf VNFS system (the RAM disks) and plainly installed one server and two slave nodes but with disks at the nodes, and the problem persisted.

Also tried NFS version 2, and version 3 with TCP only, and UDP only.

At this moment we are not using any mount options but we tried before to use acregmax=0,acdirmax=0,noac,nocto' at the nodes with no effect. Also tried async/sync' at the server side (we don't think this would work, though).

The cluster is for intensive numerical computations but all the problems persist even when there is no computing load, neither network traffic. Also we tried with only two slave nodes.

There are no errors on the network, as reported by ifconfig'. The network is Gigabit Ethernet with 3Com switches and cards. The driver is sk98lin' version 8-31.

Thanks for your time,

Mario

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs