Skip to contentSkip to conversation

NFS synchronization problems on Beowulf cluster

DISCUSSION Mario Storti 2 participants 9 messages Jun 1, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel 2.6.15). The cluster is disk-less (nodes don't have hard disks) based on the Warewulf package. NFS traffic is reduced by using VNFS filesystems at the nodes.

We found something strange related with NFS. For some files in user accounts if we make some modifications to the file in the server, this changes are not seen in the compute nodes. The NFS server is NFS3 and with the standard configuration (8 instances of the server and default parameters). The cluster has 20 nodes at this time, but we have made experiments with a `cloned' cluster and even with only two nodes the problem persist.

The experiment is as follows: We change a text file in a user account with Emacs and checking whether the change is seen in the compute nodes. Sometimes the change is immediately seen in the compute nodes, but many times some nodes don't see the change.

This happens even when the cluster is idle (no computing or network load in the slave nodes, neither on the server), so it's not a problem of performance. It is plainly working bad.

Below are the rpcinfo -p' and nfsstat' on the server and on the client. The version of nfs-utils are 1.0.6.

We tried several combinations for the export and mount options (sync/async...) with no success.

Any help is appreciated,

TIA, Mario

http://www.cimec.org.ar/mstorti

======================= SERVER ================================= [root@aquiles ~]# rpcinfo -p
program vers proto port

    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    947  status
    100024    1   tcp    950  status
    100011    1   udp    777  rquotad
    100011    2   udp    777  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   udp  32775  nlockmgr
    100021    3   udp  32775  nlockmgr
    100021    4   udp  32775  nlockmgr
    100021    1   tcp  32769  nlockmgr
    100021    3   tcp  32769  nlockmgr
    100021    4   tcp  32769  nlockmgr
    100005    1   udp    795  mountd
    100005    1   tcp    798  mountd
    100005    3   udp    795  mountd
    100005    3   tcp    798  mountd
[root@aquiles ~]# nfsstat
Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
3734126    0          0          0          0       

Server nfs v3:
null getattr setattr lookup access readlink
240 0% 1375308 36% 86317 2% 135591 3% 605405 16% 1719 0%
read write create mkdir symlink mknod
72626 1% 1352149 36% 9697 0% 0 0% 0 0% 160 0%
remove rmdir rename link readdir readdirplus
4909 0% 0 0% 64 0% 30 0% 18 0% 60812 1%
fsstat fsinfo pathconf commit
0 0% 122 0% 0 0% 28958 0%

======================= COMPUTE NODE =============================

[root@node1 ~]# rpcinfo -p

   programa vers proto   puerto
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    892  status
    100024    1   tcp    895  status
    100021    1   udp  32768  nlockmgr
    100021    3   udp  32768  nlockmgr
    100021    4   udp  32768  nlockmgr
    100021    1   tcp  53111  nlockmgr
    100021    3   tcp  53111  nlockmgr
    100021    4   tcp  53111  nlockmgr
[root@node1 ~]# nfsstat
Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
0          0          0          0          0       
Client rpc stats:
calls      retrans    authrefrsh
2950634    0          0       

Client nfs v3:
null getattr setattr lookup access readlink
0 0% 988154 33% 86986 2% 44612 1% 353215 11% 102 0%
read write create mkdir symlink mknod
6821 0% 1367868 46% 9227 0% 0 0% 0 0% 8 0%
remove rmdir rename link readdir readdirplus
4577 0% 0 0% 7 0% 18 0% 18 0% 59879 2%
fsstat fsinfo pathconf commit
0 0% 2 0% 0 0% 29138 0%

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#1 lore raw
Quoted 5 lines · Mario Storti <mstorti <at> intec.unl.edu.ar> writes:

We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel 2.6.15). The cluster is disk-less (nodes don't have hard disks) based on the Warewulf package. NFS traffic is reduced by using VNFS filesystems at the nodes.

Quoted 7 lines

We found something strange related with NFS. For some files in user accounts if we make some modifications to the file in the server, this changes are not seen in the compute nodes. The NFS server is NFS3 and with the standard configuration (8 instances of the server and default parameters). The cluster has 20 nodes at this time, but we have made experiments with a `cloned' cluster and even with only two nodes the problem persist.

I see now that perhaps my reference to VNFS is somewhat obscure. VNFS is basically a RAM disk at the nodes (in our case of size 50MB approx.) with typically `light' directories /etc, /bin, /lib...

http://www.warewulf-cluster.org/cgi-bin/trac.cgi/wiki/VNFS

The `heavy' directories like /usr and /home are exported via standard NFS, so that the VNFS issue is not directly related with the problem.

On the other hand we tried to switch to NFS version 2 and the problem persists.

Thanks, Mario

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw
Quoted 23 lines · Mario Storti
Quoted 5 lines · Mario Storti <mstorti <at> intec.unl.edu.ar> writes:

We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel 2.6.15). The cluster is disk-less (nodes don't have hard disks) based on the Warewulf package. NFS traffic is reduced by using VNFS filesystems at the nodes.

Quoted 9 lines

We found something strange related with NFS. For some files in user accounts if we make some modifications to the file in the server, this changes are not seen in the compute nodes. The NFS server is NFS3 and with the standard configuration (8 instances of the server and default parameters). The cluster has 20 nodes at this time, but we have made experiments with a `cloned' cluster and even with only two nodes the problem persist.

I see now that perhaps my reference to VNFS is somewhat obscure. VNFS is basically a RAM disk at the nodes (in our case of size 50MB approx.) with typically `light' directories /etc, /bin, /lib...

Quoted 7 lines

http://www.warewulf-cluster.org/cgi-bin/trac.cgi/wiki/VNFS

The `heavy' directories like /usr and /home are exported via standard NFS, so that the VNFS issue is not directly related with the problem.

On the other hand we tried to switch to NFS version 2 and the problem persists.

What is the local file system type being exported on the NFS servers?

Thanx...

ps

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw
Quoted 1 line · Peter Staubach <staubach <at> redhat.com> writes:

What is the local file system type being exported on the NFS servers?

It is software RAID mounted on top of XFS. At first time we suspected of this combination, and the first thing we do is to clone the system on a plain ext3 filesystem without RAID. The cloned cluster with two nodes exhibited exactly the same behaviour, so we don't think now that the problem is in the combination of NFS with RAID+XFS. It ssems to be something in the NFS setup/configuration itself.

Thanks, Mario

-------------------------
Mario Alberto Storti [cel. +54-342-156144983]
CIMEC (INTEC/CONICET-UNL), Guemes 3450 - 3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1015), Tel/Fax: +54-342-4511169
e-mail: mstorti at intec dot unl dot edu dot ar
http://www.cimec.org.ar/mstorti
-------------------------

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#4 lore raw
Quoted 8 lines · Mario Storti
Quoted 3 lines · Peter Staubach <staubach <at> redhat.com> writes:

What is the local file system type being exported on the NFS servers?

Quoted 7 lines

It is software RAID mounted on top of XFS. At first time we suspected of this combination, and the first thing we do is to clone the system on a plain ext3 filesystem without RAID. The cloned cluster with two nodes exhibited exactly the same behaviour, so we don't think now that the problem is in the combination of NFS with RAID+XFS. It ssems to be something in the NFS setup/configuration itself.

I see.

I see from the nfsstat output that this job mix is _very_ heavily write intensive. I might suspect that this may be contributing to the issue being seen. The consistency for files is very relaxed when those files are being written to.

Are any special mount parameters being used?

Thanx...

ps

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#5 lore raw
Quoted 4 lines · Peter Staubach <staubach <at> redhat.com> writes:

I see from the `nfsstat' output that this job mix is _very_ heavily write intensive. I might suspect that this may be contributing to the issue being seen. The consistency for files is very relaxed when those files are being written to.

The nfsstat' results that I sent were cumulative. I learn now that in theory we can zero them with -z', but it seems not to work. Perhaps there is some intense traffic when the slaves boot but not at the time of making the `experiment'. At that time the NFS traffic es _extremely_ low. We change only a small text file (for reference it's my ~/.bashrc) and check if the changes are reflected in the nodes. There are no applications running at the nodes.

Quoted 1 line

Are any special mount parameters being used?

No, we tried vers=2 for forcing version 2, also tried noac'. Also tried async/sync' at the server side. None had effect on the problem. `noac' of course reflected in a lack of efficiency, but the problem persists.

Thanks, Mario

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#6 lore raw
Quoted 11 lines · Mario Storti
Quoted 6 lines · Peter Staubach <staubach <at> redhat.com> writes:

I see from the `nfsstat' output that this job mix is _very_ heavily write intensive. I might suspect that this may be contributing to the issue being seen. The consistency for files is very relaxed when those files are being written to.

Quoted 10 lines

The nfsstat' results that I sent were cumulative. I learn now that in theory we can zero them with -z', but it seems not to work. Perhaps there is some intense traffic when the slaves boot but not at the time of making the `experiment'. At that time the NFS traffic es _extremely_ low. We change only a small text file (for reference it's my ~/.bashrc) and check if the changes are reflected in the nodes. There are no applications running at the nodes.

I wouldn't have thought that there would be intense write activity when the systems are booting. I could see lots of read activity, but the stats didn't show that much compared with the number of writes.

Quoted 9 lines
Quoted 3 lines

Are any special mount parameters being used?

No, we tried vers=2 for forcing version 2, also tried noac'. Also tried async/sync' at the server side. None had effect on the problem. `noac' of course reflected in a lack of efficiency, but the problem persists.

I wouldn't think that sync/async would have any effect on this sort of consistency. Those options more control when the data/metadata is actually flushed to stable storage, whether before the server responds to the specific request or sometime after.

Can you get a raw tethereal capture file which shows the issue? Perhaps a capture which includes reading the original contents of the file and then reading it again after the file is modified on the server or by another client? Perhaps from that we can get some clues as to where the problem might be.

Thanx...

ps

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#7 lore raw
Quoted 5 lines · Peter Staubach <staubach <at> redhat.com> writes:

Can you get a raw tethereal capture file which shows the issue? Perhaps a capture which includes reading the original contents of the file and then reading it again after the file is modified on the server or by another client? Perhaps from that we can get some clues as to where the problem might be.

OK. Good idea. I added a line with #jaja12' to my file ~/.bashrc' and verified that all nodes see the change. Then I modify the line with Emacs in the server to #jaja13' and svae, then to #jaja14' and save, and to #jaja15'. Some nodes acknowledge the change but others not. If I do a parallel grep' I obtain the following

================================================================
[mstorti@aquiles ~]$ pdsh -a 'grep jaja ~/.bashrc'
node2: # jaja12
node1: # jaja15
node9: # jaja15
node10: # jaja14
node11: # jaja12
node13: # jaja15
node20: # jaja15
node3: # jaja15
node17: # jaja12
node4: # jaja15
node7: # jaja12
node8: # jaja15
node18: # jaja15
node6: # jaja12
node14: # jaja14
node12: # jaja15
node16: # jaja14
node5: # jaja15
node19: # jaja14
node15: # jaja15
[mstorti@aquiles ~]$
================================================================

So I choose for instance `node2' which has still the "# jaja12" string. Then

* I open a shell in node2,
* Launch a `tethereal' filtering the

        communication between the server (192.168.0.201) and node2
        (192.168.0.2).

* I do a grep in node2

    [mstorti@node2 ~]$ grep jaja .bashrc
    # jaja12
    [mstorti@node2 ~]$ 

* and I close the `tethereal'.

What I obtain is the following:

================================================================
[root@aquiles ~]# tethereal -i eth0 host aquiles and host node2 Capturing on eth0
0.000000 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
2.490445 192.168.0.201 -> 192.168.0.2 Rlogin Data: g
2.490548 192.168.0.2 -> 192.168.0.201 Rlogin Data: g
2.490567 192.168.0.201 -> 192.168.0.2 Rlogin Data: rep jaja .bashrc\n 2.490794 192.168.0.2 -> 192.168.0.201 Rlogin Data: rep jaja .bashrc\r\n 2.491546 192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0x76120002 2.491597 192.168.0.201 -> 192.168.0.2 NFS V3 GETATTR Reply (Call In 6) 2.491669 192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=108 Ack=116
Win=32645 Len=0 TSV=612612 TSER=27245753
2.491676 192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0xf8ea8c0a 2.491701 192.168.0.201 -> 192.168.0.2 NFS V3 GETATTR Reply (Call In 9) 2.529519 192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=216 Ack=232
Win=32674 Len=0 TSV=612616 TSER=27245753
2.536043 192.168.0.201 -> 192.168.0.2 TCP 1023 > login [ACK] Seq=18 Ack=19
Win=1460 Len=0 TSV=27245757 TSER=612612
2.536144 192.168.0.2 -> 192.168.0.201 Rlogin Data: # jaja12\r\n[mstorti@node2
~]$
2.539333 192.168.0.201 -> 192.168.0.2 TCP 1023 > login [ACK] Seq=18 Ack=48
Win=1460 Len=0 TSV=27245757 TSER=612616
7.549413 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549424 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549428 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549431 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549435 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549438 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
20 packets captured
[root@aquiles ~]#
================================================================

Do you see something in there? I am studying it, but I didn't know enough of NFS for understaing it yet.

Thank for your help, Mario

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#8 lore raw
Quoted 96 lines · Mario Storti
Quoted 7 lines · Peter Staubach <staubach <at> redhat.com> writes:

Can you get a raw tethereal capture file which shows the issue? Perhaps a capture which includes reading the original contents of the file and then reading it again after the file is modified on the server or by another client? Perhaps from that we can get some clues as to where the problem might be.

OK. Good idea. I added a line with #jaja12' to my file ~/.bashrc' and verified that all nodes see the change. Then I modify the line with Emacs in the server to #jaja13' and svae, then to #jaja14' and save, and to #jaja15'. Some nodes acknowledge the change but others not. If I do a parallel grep' I obtain the following

================================================================
[mstorti@aquiles ~]$ pdsh -a 'grep jaja ~/.bashrc'
node2: # jaja12
node1: # jaja15
node9: # jaja15
node10: # jaja14
node11: # jaja12
node13: # jaja15
node20: # jaja15
node3: # jaja15
node17: # jaja12
node4: # jaja15
node7: # jaja12
node8: # jaja15
node18: # jaja15
node6: # jaja12
node14: # jaja14
node12: # jaja15
node16: # jaja14
node5: # jaja15
node19: # jaja14
node15: # jaja15
[mstorti@aquiles ~]$ ================================================================

So I choose for instance `node2' which has still the "# jaja12" string. Then

* I open a shell in node2, * Launch a `tethereal' filtering the

       communication between the server (192.168.0.201) and node2
       (192.168.0.2).

* I do a grep in node2

   [mstorti@node2 ~]$ grep jaja .bashrc
   # jaja12
   [mstorti@node2 ~]$ 

* and I close the `tethereal'.

What I obtain is the following:

================================================================
[root@aquiles ~]# tethereal -i eth0 host aquiles and host node2 Capturing on eth0
0.000000 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
2.490445 192.168.0.201 -> 192.168.0.2 Rlogin Data: g
2.490548 192.168.0.2 -> 192.168.0.201 Rlogin Data: g
2.490567 192.168.0.201 -> 192.168.0.2 Rlogin Data: rep jaja .bashrc\n 2.490794 192.168.0.2 -> 192.168.0.201 Rlogin Data: rep jaja .bashrc\r\n 2.491546 192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0x76120002 2.491597 192.168.0.201 -> 192.168.0.2 NFS V3 GETATTR Reply (Call In 6) 2.491669 192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=108 Ack=116
Win=32645 Len=0 TSV=612612 TSER=27245753
2.491676 192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0xf8ea8c0a 2.491701 192.168.0.201 -> 192.168.0.2 NFS V3 GETATTR Reply (Call In 9) 2.529519 192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=216 Ack=232
Win=32674 Len=0 TSV=612616 TSER=27245753
2.536043 192.168.0.201 -> 192.168.0.2 TCP 1023 > login [ACK] Seq=18 Ack=19
Win=1460 Len=0 TSV=27245757 TSER=612612
2.536144 192.168.0.2 -> 192.168.0.201 Rlogin Data: # jaja12\r\n[mstorti@node2
~]$
2.539333 192.168.0.201 -> 192.168.0.2 TCP 1023 > login [ACK] Seq=18 Ack=48
Win=1460 Len=0 TSV=27245757 TSER=612616
7.549413 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549424 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549428 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549431 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549435 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
7.549438 192.168.0.2 -> 192.168.0.201 UDP Source port: 32769 Destination
port: 8649
20 packets captured
[root@aquiles ~]# ================================================================

Quoted 3 lines

Do you see something in there? I am studying it, but I didn't know enough of NFS for understaing it yet.

Actually, what I asked about was a "raw capture file". I think that the values returned in the GETATTR calls may be interesting. To see them, we need more then just the summary lines.

Can you do this again, but have tethereal capture into a file (via the -w option) and then attach that file to a note to me or this list, please?

Thanx...

ps

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs