NFS synchronization problems on Beowulf cluster
Subsystems (2)
Hi all,
We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel 2.6.15). The cluster is disk-less (nodes don't have hard disks) based on the Warewulf package. NFS traffic is reduced by using VNFS filesystems at the nodes.
We found something strange related with NFS. For some files in user accounts if we make some modifications to the file in the server, this changes are not seen in the compute nodes. The NFS server is NFS3 and with the standard configuration (8 instances of the server and default parameters). The cluster has 20 nodes at this time, but we have made experiments with a `cloned' cluster and even with only two nodes the problem persist.
The experiment is as follows: We change a text file in a user account with Emacs and checking whether the change is seen in the compute nodes. Sometimes the change is immediately seen in the compute nodes, but many times some nodes don't see the change.
This happens even when the cluster is idle (no computing or network load in the slave nodes, neither on the server), so it's not a problem of performance. It is plainly working bad.
Below are the rpcinfo -p' and nfsstat' on the server and on the client. The version of nfs-utils are 1.0.6.
We tried several combinations for the export and mount options (sync/async...) with no success.
Any help is appreciated,
TIA, Mario
http://www.cimec.org.ar/mstorti
======================= SERVER ================================= [root@aquiles ~]# rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 947 status
100024 1 tcp 950 status
100011 1 udp 777 rquotad
100011 2 udp 777 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 udp 32775 nlockmgr
100021 3 udp 32775 nlockmgr
100021 4 udp 32775 nlockmgr
100021 1 tcp 32769 nlockmgr
100021 3 tcp 32769 nlockmgr
100021 4 tcp 32769 nlockmgr
100005 1 udp 795 mountd
100005 1 tcp 798 mountd
100005 3 udp 795 mountd
100005 3 tcp 798 mountd
[root@aquiles ~]# nfsstat
Server rpc stats:
calls badcalls badauth badclnt xdrcall
3734126 0 0 0 0 Server nfs v3:
null getattr setattr lookup access readlink
240 0% 1375308 36% 86317 2% 135591 3% 605405 16% 1719 0%
read write create mkdir symlink mknod
72626 1% 1352149 36% 9697 0% 0 0% 0 0% 160 0%
remove rmdir rename link readdir readdirplus
4909 0% 0 0% 64 0% 30 0% 18 0% 60812 1%
fsstat fsinfo pathconf commit
0 0% 122 0% 0 0% 28958 0%
======================= COMPUTE NODE =============================
[root@node1 ~]# rpcinfo -p
programa vers proto puerto
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 892 status
100024 1 tcp 895 status
100021 1 udp 32768 nlockmgr
100021 3 udp 32768 nlockmgr
100021 4 udp 32768 nlockmgr
100021 1 tcp 53111 nlockmgr
100021 3 tcp 53111 nlockmgr
100021 4 tcp 53111 nlockmgr
[root@node1 ~]# nfsstat
Server rpc stats:
calls badcalls badauth badclnt xdrcall
0 0 0 0 0
Client rpc stats:
calls retrans authrefrsh
2950634 0 0 Client nfs v3:
null getattr setattr lookup access readlink
0 0% 988154 33% 86986 2% 44612 1% 353215 11% 102 0%
read write create mkdir symlink mknod
6821 0% 1367868 46% 9227 0% 0 0% 0 0% 8 0%
remove rmdir rename link readdir readdirplus
4577 0% 0 0% 7 0% 18 0% 18 0% 59879 2%
fsstat fsinfo pathconf commit
0 0% 2 0% 0 0% 29138 0%
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 5 lines · Mario Storti <mstorti <at> intec.unl.edu.ar> writes:
We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel 2.6.15). The cluster is disk-less (nodes don't have hard disks) based on the Warewulf package. NFS traffic is reduced by using VNFS filesystems at the nodes.
Quoted 7 lines
We found something strange related with NFS. For some files in user accounts if we make some modifications to the file in the server, this changes are not seen in the compute nodes. The NFS server is NFS3 and with the standard configuration (8 instances of the server and default parameters). The cluster has 20 nodes at this time, but we have made experiments with a `cloned' cluster and even with only two nodes the problem persist.
I see now that perhaps my reference to VNFS is somewhat obscure. VNFS is basically a RAM disk at the nodes (in our case of size 50MB approx.) with typically `light' directories /etc, /bin, /lib...
http://www.warewulf-cluster.org/cgi-bin/trac.cgi/wiki/VNFS
The `heavy' directories like /usr and /home are exported via standard NFS, so that the VNFS issue is not directly related with the problem.
On the other hand we tried to switch to NFS version 2 and the problem persists.
Thanks, Mario
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 23 lines · Mario Storti
Quoted 5 lines · Mario Storti <mstorti <at> intec.unl.edu.ar> writes:
We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel 2.6.15). The cluster is disk-less (nodes don't have hard disks) based on the Warewulf package. NFS traffic is reduced by using VNFS filesystems at the nodes.
Quoted 9 lines
We found something strange related with NFS. For some files in user accounts if we make some modifications to the file in the server, this changes are not seen in the compute nodes. The NFS server is NFS3 and with the standard configuration (8 instances of the server and default parameters). The cluster has 20 nodes at this time, but we have made experiments with a `cloned' cluster and even with only two nodes the problem persist.
I see now that perhaps my reference to VNFS is somewhat obscure. VNFS is basically a RAM disk at the nodes (in our case of size 50MB approx.) with typically `light' directories /etc, /bin, /lib...
Quoted 7 lines
http://www.warewulf-cluster.org/cgi-bin/trac.cgi/wiki/VNFS
The `heavy' directories like /usr and /home are exported via standard NFS, so that the VNFS issue is not directly related with the problem.
On the other hand we tried to switch to NFS version 2 and the problem persists.
What is the local file system type being exported on the NFS servers?
Thanx...
ps
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line · Peter Staubach <staubach <at> redhat.com> writes:
What is the local file system type being exported on the NFS servers?
It is software RAID mounted on top of XFS. At first time we suspected of this combination, and the first thing we do is to clone the system on a plain ext3 filesystem without RAID. The cloned cluster with two nodes exhibited exactly the same behaviour, so we don't think now that the problem is in the combination of NFS with RAID+XFS. It ssems to be something in the NFS setup/configuration itself.
Thanks, Mario
-------------------------
Mario Alberto Storti [cel. +54-342-156144983]
CIMEC (INTEC/CONICET-UNL), Guemes 3450 - 3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1015), Tel/Fax: +54-342-4511169
e-mail: mstorti at intec dot unl dot edu dot ar
http://www.cimec.org.ar/mstorti
-------------------------
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Mario Storti
Quoted 3 lines · Peter Staubach <staubach <at> redhat.com> writes:
What is the local file system type being exported on the NFS servers?
Quoted 7 lines
It is software RAID mounted on top of XFS. At first time we suspected of this combination, and the first thing we do is to clone the system on a plain ext3 filesystem without RAID. The cloned cluster with two nodes exhibited exactly the same behaviour, so we don't think now that the problem is in the combination of NFS with RAID+XFS. It ssems to be something in the NFS setup/configuration itself.
I see.
I see from the nfsstat output that this job mix is _very_ heavily write intensive. I might suspect that this may be contributing to the issue being seen. The consistency for files is very relaxed when those files are being written to.
Are any special mount parameters being used?
Thanx...
ps
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · Peter Staubach <staubach <at> redhat.com> writes:
I see from the `nfsstat' output that this job mix is _very_ heavily write intensive. I might suspect that this may be contributing to the issue being seen. The consistency for files is very relaxed when those files are being written to.
The nfsstat' results that I sent were cumulative. I learn now that in theory we can zero them with -z', but it seems not to work. Perhaps there is some intense traffic when the slaves boot but not at the time of making the `experiment'. At that time the NFS traffic es _extremely_ low. We change only a small text file (for reference it's my ~/.bashrc) and check if the changes are reflected in the nodes. There are no applications running at the nodes.
Quoted 1 line
Are any special mount parameters being used?
No, we tried vers=2 for forcing version 2, also tried noac'. Also tried async/sync' at the server side. None had effect on the problem. `noac' of course reflected in a lack of efficiency, but the problem persists.
Thanks, Mario
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · Mario Storti
Quoted 6 lines · Peter Staubach <staubach <at> redhat.com> writes:
I see from the `nfsstat' output that this job mix is _very_ heavily write intensive. I might suspect that this may be contributing to the issue being seen. The consistency for files is very relaxed when those files are being written to.
Quoted 10 lines
The
nfsstat' results that I sent were cumulative. I learn now that in theory we can zero them with-z', but it seems not to work. Perhaps there is some intense traffic when the slaves boot but not at the time of making the `experiment'. At that time the NFS traffic es _extremely_ low. We change only a small text file (for reference it's my ~/.bashrc) and check if the changes are reflected in the nodes. There are no applications running at the nodes.
I wouldn't have thought that there would be intense write activity when the systems are booting. I could see lots of read activity, but the stats didn't show that much compared with the number of writes.
Quoted 9 lines
Quoted 3 lines
Are any special mount parameters being used?
No, we tried vers=2 for forcing version 2, also tried
noac'. Also triedasync/sync' at the server side. None had effect on the problem. `noac' of course reflected in a lack of efficiency, but the problem persists.
I wouldn't think that sync/async would have any effect on this sort of consistency. Those options more control when the data/metadata is actually flushed to stable storage, whether before the server responds to the specific request or sometime after.
Can you get a raw tethereal capture file which shows the issue? Perhaps a capture which includes reading the original contents of the file and then reading it again after the file is modified on the server or by another client? Perhaps from that we can get some clues as to where the problem might be.
Thanx...
ps
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 5 lines · Peter Staubach <staubach <at> redhat.com> writes:
Can you get a raw tethereal capture file which shows the issue? Perhaps a capture which includes reading the original contents of the file and then reading it again after the file is modified on the server or by another client? Perhaps from that we can get some clues as to where the problem might be.
OK. Good idea. I added a line with #jaja12' to my file ~/.bashrc' and verified that all nodes see the change. Then I modify the line with Emacs in the server to #jaja13' and svae, then to #jaja14' and save, and to #jaja15'. Some nodes acknowledge the change but others not. If I do a parallel grep' I obtain the following
================================================================
[mstorti@aquiles ~]$ pdsh -a 'grep jaja ~/.bashrc'
node2: # jaja12
node1: # jaja15
node9: # jaja15
node10: # jaja14
node11: # jaja12
node13: # jaja15
node20: # jaja15
node3: # jaja15
node17: # jaja12
node4: # jaja15
node7: # jaja12
node8: # jaja15
node18: # jaja15
node6: # jaja12
node14: # jaja14
node12: # jaja15
node16: # jaja14
node5: # jaja15
node19: # jaja14
node15: # jaja15
[mstorti@aquiles ~]$
================================================================
So I choose for instance `node2' which has still the "# jaja12" string. Then
* I open a shell in node2,
* Launch a `tethereal' filtering the
communication between the server (192.168.0.201) and node2
(192.168.0.2).* I do a grep in node2
[mstorti@node2 ~]$ grep jaja .bashrc
# jaja12
[mstorti@node2 ~]$ * and I close the `tethereal'.
What I obtain is the following:
================================================================
[root@aquiles ~]# tethereal -i eth0 host aquiles and host node2 Capturing on eth0
0.000000 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
2.490445 192.168.0.201 -> 192.168.0.2 Rlogin Data: g
2.490548 192.168.0.2 -> 192.168.0.201 Rlogin Data: g
2.490567 192.168.0.201 -> 192.168.0.2 Rlogin Data: rep jaja .bashrc\n 2.490794 192.168.0.2 -> 192.168.0.201 Rlogin Data: rep jaja .bashrc\r\n 2.491546 192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0x76120002 2.491597 192.168.0.201 -> 192.168.0.2 NFS V3 GETATTR Reply (Call In 6) 2.491669 192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=108 Ack=116
Win=32645 Len=0 TSV=612612 TSER=27245753
2.491676 192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0xf8ea8c0a 2.491701 192.168.0.201 -> 192.168.0.2 NFS V3 GETATTR Reply (Call In 9) 2.529519 192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=216 Ack=232
Win=32674 Len=0 TSV=612616 TSER=27245753
2.536043 192.168.0.201 -> 192.168.0.2 TCP 1023 > login [ACK] Seq=18 Ack=19
Win=1460 Len=0 TSV=27245757 TSER=612612
2.536144 192.168.0.2 -> 192.168.0.201 Rlogin Data: # jaja12\r\n[mstorti@node2
~]$
2.539333 192.168.0.201 -> 192.168.0.2 TCP 1023 > login [ACK] Seq=18 Ack=48
Win=1460 Len=0 TSV=27245757 TSER=612616
7.549413 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549424 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549428 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549431 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549435 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549438 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
20 packets captured
[root@aquiles ~]#
================================================================
Do you see something in there? I am studying it, but I didn't know enough of NFS for understaing it yet.
Thank for your help, Mario
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 96 lines · Mario Storti
Quoted 7 lines · Peter Staubach <staubach <at> redhat.com> writes:
Can you get a raw tethereal capture file which shows the issue? Perhaps a capture which includes reading the original contents of the file and then reading it again after the file is modified on the server or by another client? Perhaps from that we can get some clues as to where the problem might be.
OK. Good idea. I added a line with
#jaja12' to my file~/.bashrc' and verified that all nodes see the change. Then I modify the line with Emacs in the server to#jaja13' and svae, then to#jaja14' and save, and to#jaja15'. Some nodes acknowledge the change but others not. If I do aparallel grep' I obtain the following================================================================
[mstorti@aquiles ~]$ pdsh -a 'grep jaja ~/.bashrc'
node2: # jaja12
node1: # jaja15
node9: # jaja15
node10: # jaja14
node11: # jaja12
node13: # jaja15
node20: # jaja15
node3: # jaja15
node17: # jaja12
node4: # jaja15
node7: # jaja12
node8: # jaja15
node18: # jaja15
node6: # jaja12
node14: # jaja14
node12: # jaja15
node16: # jaja14
node5: # jaja15
node19: # jaja14
node15: # jaja15
[mstorti@aquiles ~]$ ================================================================So I choose for instance `node2' which has still the "# jaja12" string. Then
* I open a shell in node2, * Launch a `tethereal' filtering the
communication between the server (192.168.0.201) and node2 (192.168.0.2).* I do a grep in node2
[mstorti@node2 ~]$ grep jaja .bashrc # jaja12 [mstorti@node2 ~]$* and I close the `tethereal'.
What I obtain is the following:
================================================================
[root@aquiles ~]# tethereal -i eth0 host aquiles and host node2 Capturing on eth0
0.000000 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
2.490445 192.168.0.201 -> 192.168.0.2 Rlogin Data: g
2.490548 192.168.0.2 -> 192.168.0.201 Rlogin Data: g
2.490567 192.168.0.201 -> 192.168.0.2 Rlogin Data: rep jaja .bashrc\n 2.490794 192.168.0.2 -> 192.168.0.201 Rlogin Data: rep jaja .bashrc\r\n 2.491546 192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0x76120002 2.491597 192.168.0.201 -> 192.168.0.2 NFS V3 GETATTR Reply (Call In 6) 2.491669 192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=108 Ack=116
Win=32645 Len=0 TSV=612612 TSER=27245753
2.491676 192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0xf8ea8c0a 2.491701 192.168.0.201 -> 192.168.0.2 NFS V3 GETATTR Reply (Call In 9) 2.529519 192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=216 Ack=232
Win=32674 Len=0 TSV=612616 TSER=27245753
2.536043 192.168.0.201 -> 192.168.0.2 TCP 1023 > login [ACK] Seq=18 Ack=19
Win=1460 Len=0 TSV=27245757 TSER=612612
2.536144 192.168.0.2 -> 192.168.0.201 Rlogin Data: # jaja12\r\n[mstorti@node2
~]$
2.539333 192.168.0.201 -> 192.168.0.2 TCP 1023 > login [ACK] Seq=18 Ack=48
Win=1460 Len=0 TSV=27245757 TSER=612616
7.549413 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549424 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549428 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549431 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549435 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549438 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
20 packets captured
[root@aquiles ~]# ================================================================
Quoted 3 lines
Do you see something in there? I am studying it, but I didn't know enough of NFS for understaing it yet.
Actually, what I asked about was a "raw capture file". I think that the values returned in the GETATTR calls may be interesting. To see them, we need more then just the summary lines.
Can you do this again, but have tethereal capture into a file (via the -w option) and then attach that file to a note to me or this list, please?
Thanx...
ps
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs