Skip to contentSkip to conversation

NFS mounts being mounted read-only depending upon mount order???

DISCUSSION Michael Walker 2 participants 6 messages Nov 17, 2006 – Nov 20, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

All,

I'm having problems with the latest version of FC5 which contains kernel 2.6.18 - the problem is that NFS mounts of read-write file-systems are being mounted read-only, sometimes.

Whether a mount is mounted read-only or read-write seems to very depending upon what other directories were mounted from the same server and with what options.

Take the following file-systems exported from my
Linux Server server:

    root@polyslo:/proc/fs/nfs# cat /etc/exports
    /disk1/dir_ro   *(ro,no_root_squash,sync,wdelay,no_subtree_check)
    /disk1/dir_rw   *(rw,no_root_squash,sync,wdelay,no_subtree_check)
    root@polyslo:/proc/fs/nfs# showmount -e 10.20.20.48
    Export list for 10.20.20.48:
    /disk1/dir_rw *
    /disk1/dir_ro *
    root@polyslo:/proc/fs/nfs#

If I now do the following mounts (in this order):

# Mount initial directory read-only - this is intentional mount -t nfs -o ro 10.20.20.48:/disk1/dir_ro /tmp/dir_ro

# Attempt to mount second directory read-write mount -t nfs -o rw 10.20.20.48:/disk1/dir_rw /tmp/dir_rw

# Now I attempt to touch a file in the rw directory:
[root@vt-1 tmp]# touch /tmp/dir_rw/hello
touch: cannot touch `/tmp/dir_rw/hello': Read-only file system

# And if I take a look in the /proc/mounts directory - I see # that the directory was truly mounted read-only

[root@vt-1 tmp]# grep 10.20.20.48 /proc/mounts
10.20.20.48:/disk1/dir_ro /tmp/dir_ro nfs ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0 10.20.20.48:/disk1/dir_rw /tmp/dir_rw nfs ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0 [root@vt-1 tmp]#

The above seems wrong to me - can someone clarify what's going on. I know that this is new behavior that was not true in the past. My RHEL_4U4 server running 2.6.9 kernel does not have this problem.

Notice that if I change the order of the mounts I get different behavior:

# Initially mount the read-write directory:
mount -t nfs -o rw 10.20.20.48:/disk1/dir_rw /tmp/dir_rw

# Then mount the read-only directory:
mount -t nfs -o ro 10.20.20.48:/disk1/dir_ro /tmp/dir_ro

# Now the directories are mounted as I wish, I can write to the # read-write mount:
[root@vt-1 tmp]# touch /tmp/dir_rw/hello
[root@vt-1 tmp]#

# And the read-only directory is read-only:
[root@vt-1 tmp]# touch /tmp/dir_ro/hello
touch: cannot touch `/tmp/dir_ro/hello': Read-only file system [root@vt-1 tmp]#

# And the /proc/mounts file reflects the states I expect
[root@vt-1 tmp]# grep 10.20.20.48 /proc/mounts
10.20.20.48:/disk1/dir_rw /tmp/dir_rw nfs rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0 10.20.20.48:/disk1/dir_ro /tmp/dir_ro nfs rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0 [root@vt-1 tmp]#

Note that previously the order did not matter and the permissions always reflected the permissions of the individual mount requests.

If someone could provide insights on what's going on above that would be great. This is a artificial test case - but it's actually hitting code we have in production and will be a significant issue for us.

Thanx much!

_Mike_

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#1 lore raw

All,

Any insights on this?

I did find the following post from Trond which appears

http://lkml.org/lkml/2006/10/18/264

Which seems to imply that this is a 'feature' and not a 'bug'? Is that true - or have I missed something?

Thanx,
_Mike_

Quoted 83 lines · Michael Walker

All,

I'm having problems with the latest version of FC5 which contains kernel 2.6.18 - the problem is that NFS mounts of read-write file-systems are being mounted read-only, sometimes.

Whether a mount is mounted read-only or read-write seems to very depending upon what other directories were mounted from the same server and with what options.

Take the following file-systems exported from my
Linux Server server:

   root@polyslo:/proc/fs/nfs# cat /etc/exports
   /disk1/dir_ro   *(ro,no_root_squash,sync,wdelay,no_subtree_check)
   /disk1/dir_rw   *(rw,no_root_squash,sync,wdelay,no_subtree_check)
   root@polyslo:/proc/fs/nfs# showmount -e 10.20.20.48
   Export list for 10.20.20.48:
   /disk1/dir_rw *
   /disk1/dir_ro *
   root@polyslo:/proc/fs/nfs#

If I now do the following mounts (in this order):

# Mount initial directory read-only - this is intentional mount -t nfs -o ro 10.20.20.48:/disk1/dir_ro /tmp/dir_ro

# Attempt to mount second directory read-write mount -t nfs -o rw 10.20.20.48:/disk1/dir_rw /tmp/dir_rw

# Now I attempt to touch a file in the rw directory:
[root@vt-1 tmp]# touch /tmp/dir_rw/hello
touch: cannot touch `/tmp/dir_rw/hello': Read-only file system

# And if I take a look in the /proc/mounts directory - I see

   # that the directory was truly mounted read-only
   [root@vt-1 tmp]# grep 10.20.20.48 /proc/mounts
   10.20.20.48:/disk1/dir_ro /tmp/dir_ro nfs 

ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
10.20.20.48:/disk1/dir_rw /tmp/dir_rw nfs
ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
[root@vt-1 tmp]#

The above seems wrong to me - can someone clarify what's going on. I know that this is new behavior that was not true in the past. My RHEL_4U4 server running 2.6.9 kernel does not have this problem.

Notice that if I change the order of the mounts I get different behavior:

# Initially mount the read-write directory:
mount -t nfs -o rw 10.20.20.48:/disk1/dir_rw /tmp/dir_rw

# Then mount the read-only directory:
mount -t nfs -o ro 10.20.20.48:/disk1/dir_ro /tmp/dir_ro

# Now the directories are mounted as I wish, I can write to the

   # read-write mount:
   [root@vt-1 tmp]# touch /tmp/dir_rw/hello
   [root@vt-1 tmp]#

# And the read-only directory is read-only:
[root@vt-1 tmp]# touch /tmp/dir_ro/hello
touch: cannot touch `/tmp/dir_ro/hello': Read-only file system [root@vt-1 tmp]#

   # And the /proc/mounts file reflects the states I expect
   [root@vt-1 tmp]# grep 10.20.20.48 /proc/mounts
   10.20.20.48:/disk1/dir_rw /tmp/dir_rw nfs 

rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
10.20.20.48:/disk1/dir_ro /tmp/dir_ro nfs
rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
[root@vt-1 tmp]#

Note that previously the order did not matter and the permissions always reflected the permissions of the individual mount requests.

Quoted 8 lines

If someone could provide insights on what's going on above that would be great. This is a artificial test case - but it's actually hitting code we have in production and will be a significant issue for us.

Thanx much!

_Mike_

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw
Quoted 11 lines · Michael Walker

All,

Any insights on this?

I did find the following post from Trond which appears

   http://lkml.org/lkml/2006/10/18/264

Which seems to imply that this is a 'feature' and not a 'bug'? Is that true - or have I missed something?

That is quite correct (thanks for taking the time to search the list archives). For various reasons, we're trying to tighten the NFS cache consistency. That means that practices that were previously allowed (such as mounting different directories from the same filesystem using different mount options) have been restricted.
The ability to have the same filesystem mounted both read-only and read-write will hopefully soon be allowed again, but it needs to be fixed at the VFS level. The patches to do so have circulated for quite some time now, but have not been merged due to disagreements over how the user interface for 'mount --bind' should be in this case.

Cheers
Trond

Quoted 98 lines

Thanx,
_Mike_

Quoted 83 lines · Michael Walker

All,

I'm having problems with the latest version of FC5 which contains kernel 2.6.18 - the problem is that NFS mounts of read-write file-systems are being mounted read-only, sometimes.

Whether a mount is mounted read-only or read-write seems to very depending upon what other directories were mounted from the same server and with what options.

Take the following file-systems exported from my
Linux Server server:

   root@polyslo:/proc/fs/nfs# cat /etc/exports
   /disk1/dir_ro   *(ro,no_root_squash,sync,wdelay,no_subtree_check)
   /disk1/dir_rw   *(rw,no_root_squash,sync,wdelay,no_subtree_check)
   root@polyslo:/proc/fs/nfs# showmount -e 10.20.20.48
   Export list for 10.20.20.48:
   /disk1/dir_rw *
   /disk1/dir_ro *
   root@polyslo:/proc/fs/nfs#

If I now do the following mounts (in this order):

# Mount initial directory read-only - this is intentional mount -t nfs -o ro 10.20.20.48:/disk1/dir_ro /tmp/dir_ro

# Attempt to mount second directory read-write mount -t nfs -o rw 10.20.20.48:/disk1/dir_rw /tmp/dir_rw

# Now I attempt to touch a file in the rw directory:
[root@vt-1 tmp]# touch /tmp/dir_rw/hello
touch: cannot touch `/tmp/dir_rw/hello': Read-only file system

# And if I take a look in the /proc/mounts directory - I see

   # that the directory was truly mounted read-only
   [root@vt-1 tmp]# grep 10.20.20.48 /proc/mounts
   10.20.20.48:/disk1/dir_ro /tmp/dir_ro nfs 

ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
10.20.20.48:/disk1/dir_rw /tmp/dir_rw nfs
ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
[root@vt-1 tmp]#

The above seems wrong to me - can someone clarify what's going on. I know that this is new behavior that was not true in the past. My RHEL_4U4 server running 2.6.9 kernel does not have this problem.

Notice that if I change the order of the mounts I get different behavior:

# Initially mount the read-write directory:
mount -t nfs -o rw 10.20.20.48:/disk1/dir_rw /tmp/dir_rw

# Then mount the read-only directory:
mount -t nfs -o ro 10.20.20.48:/disk1/dir_ro /tmp/dir_ro

# Now the directories are mounted as I wish, I can write to the

   # read-write mount:
   [root@vt-1 tmp]# touch /tmp/dir_rw/hello
   [root@vt-1 tmp]#

# And the read-only directory is read-only:
[root@vt-1 tmp]# touch /tmp/dir_ro/hello
touch: cannot touch `/tmp/dir_ro/hello': Read-only file system [root@vt-1 tmp]#

   # And the /proc/mounts file reflects the states I expect
   [root@vt-1 tmp]# grep 10.20.20.48 /proc/mounts
   10.20.20.48:/disk1/dir_rw /tmp/dir_rw nfs 

rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
10.20.20.48:/disk1/dir_ro /tmp/dir_ro nfs
rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
[root@vt-1 tmp]#

Note that previously the order did not matter and the permissions always reflected the permissions of the individual mount requests.

Quoted 8 lines

If someone could provide insights on what's going on above that would be great. This is a artificial test case - but it's actually hitting code we have in production and will be a significant issue for us.

Thanx much!

_Mike_

Quoted 8 lines

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw

feature - Wow - I'm surprised part of this could be integrated without having the full solution put together.

At the moment we have a diskless NFS system setup
for booting our systems. We have the root as read-only and other mount points mounted read-write (/usr, /var, ...)

With this feature integrated we have to make sure our read-only and our read-write directories are in different physical partitions on the server? Wow - in my mind this is a drastic change to integrate without having the full solution present? Previously the only thing we had to worry about on the server was how the NFS exports were setup.

Can you point me to where the required 'mount --bind' fixes are being discussed? I haven't been able to find the thread in the 'lkml' (not that it isn't there somewhere).

Without the matching 'mount --bind' fixes this is a very serious regression for us?

Thanx for info.

_Mike_

Quoted 23 lines · Trond Myklebust
Quoted 10 lines · Michael Walker

All,

Any insights on this?

I did find the following post from Trond which appears

   http://lkml.org/lkml/2006/10/18/264

Which seems to imply that this is a 'feature' and not a 'bug'? Is that true - or have I missed something?

That is quite correct (thanks for taking the time to search the list archives). For various reasons, we're trying to tighten the NFS cache consistency. That means that practices that were previously allowed (such as mounting different directories from the same filesystem using different mount options) have been restricted.
The ability to have the same filesystem mounted both read-only and read-write will hopefully soon be allowed again, but it needs to be fixed at the VFS level. The patches to do so have circulated for quite some time now, but have not been merged due to disagreements over how the user interface for 'mount --bind' should be in this case.

Quoted 107 lines

Cheers
Trond

Quoted 104 lines

Thanx,
_Mike_

Quoted 82 lines · Michael Walker

All,

I'm having problems with the latest version of FC5 which contains kernel 2.6.18 - the problem is that NFS mounts of read-write file-systems are being mounted read-only, sometimes.

Whether a mount is mounted read-only or read-write seems to very depending upon what other directories were mounted from the same server and with what options.

Take the following file-systems exported from my
Linux Server server:

   root@polyslo:/proc/fs/nfs# cat /etc/exports
   /disk1/dir_ro   *(ro,no_root_squash,sync,wdelay,no_subtree_check)
   /disk1/dir_rw   *(rw,no_root_squash,sync,wdelay,no_subtree_check)
   root@polyslo:/proc/fs/nfs# showmount -e 10.20.20.48
   Export list for 10.20.20.48:
   /disk1/dir_rw *
   /disk1/dir_ro *
   root@polyslo:/proc/fs/nfs#

If I now do the following mounts (in this order):

# Mount initial directory read-only - this is intentional mount -t nfs -o ro 10.20.20.48:/disk1/dir_ro /tmp/dir_ro

# Attempt to mount second directory read-write mount -t nfs -o rw 10.20.20.48:/disk1/dir_rw /tmp/dir_rw

# Now I attempt to touch a file in the rw directory:
[root@vt-1 tmp]# touch /tmp/dir_rw/hello
touch: cannot touch `/tmp/dir_rw/hello': Read-only file system

# And if I take a look in the /proc/mounts directory - I see

   # that the directory was truly mounted read-only
   [root@vt-1 tmp]# grep 10.20.20.48 /proc/mounts
   10.20.20.48:/disk1/dir_ro /tmp/dir_ro nfs 

ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
10.20.20.48:/disk1/dir_rw /tmp/dir_rw nfs
ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
[root@vt-1 tmp]#

The above seems wrong to me - can someone clarify what's going on. I know that this is new behavior that was not true in the past. My RHEL_4U4 server running 2.6.9 kernel does not have this problem.

Notice that if I change the order of the mounts I get different behavior:

# Initially mount the read-write directory:
mount -t nfs -o rw 10.20.20.48:/disk1/dir_rw /tmp/dir_rw

# Then mount the read-only directory:
mount -t nfs -o ro 10.20.20.48:/disk1/dir_ro /tmp/dir_ro

# Now the directories are mounted as I wish, I can write to the

   # read-write mount:
   [root@vt-1 tmp]# touch /tmp/dir_rw/hello
   [root@vt-1 tmp]#

# And the read-only directory is read-only:
[root@vt-1 tmp]# touch /tmp/dir_ro/hello
touch: cannot touch `/tmp/dir_ro/hello': Read-only file system [root@vt-1 tmp]#

   # And the /proc/mounts file reflects the states I expect
   [root@vt-1 tmp]# grep 10.20.20.48 /proc/mounts
   10.20.20.48:/disk1/dir_rw /tmp/dir_rw nfs 

rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
10.20.20.48:/disk1/dir_ro /tmp/dir_ro nfs
rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.20.20.48 0 0
[root@vt-1 tmp]#

Note that previously the order did not matter and the permissions always reflected the permissions of the individual mount requests.

Quoted 8 lines

If someone could provide insights on what's going on above that would be great. This is a artificial test case - but it's actually hitting code we have in production and will be a significant issue for us.

Thanx much!

_Mike_

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#4 lore raw
Quoted 9 lines · Michael Walker

feature - Wow - I'm surprised part of this could be integrated without having the full solution put together.

At the moment we have a diskless NFS system setup
for booting our systems. We have the root as read-only and other mount points mounted read-write (/usr, /var, ...)

Quoted 6 lines

With this feature integrated we have to make sure our read-only and our read-write directories are in different physical partitions on the server? Wow - in my mind this is a drastic change to integrate without having the full solution present? Previously the only thing we had to worry about on the server was how the NFS exports were setup.

There _are_ workarounds available. You can use the 'fsid' export option to tell knfsd to export the directories as if they were different filesystems.

For instance if I have

        /export/home	 141.211.133.0/24(rw,sync,no_subtree_check,fsid=0)
        /export/home/trondmy 141.211.133.0/24(rw,sync,no_subtree_check,fsid=3)

in /etc/exports on the server, then I can do

mount -t nfs -oro server:/export/home /mnt/foo mount -t nfs -orw server:/export/home/trondmy /mnt/bar

with no trouble. From 'cat /proc/mounts'

server:/export/home /mnt/foo nfs rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=server 0 0 server:/export/home/trondmy /mnt/bar nfs ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=server 0 0

as expected.

Quoted 6 lines

Can you point me to where the required 'mount --bind' fixes are being discussed? I haven't been able to find the thread in the 'lkml' (not that it isn't there somewhere).

Without the matching 'mount --bind' fixes this is a very serious regression for us?

I believe the most recent patchsets can be found from

http://marc.theaimsgroup.com/?l=linux-kernel&m=115447687730218&w=2

Cheers,
Trond

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#5 lore raw
Quoted 16 lines · Trond Myklebust

On Mon, 2006-11-20 at 13:22 -0800, Michael Walker wrote:

There _are_ workarounds available. You can use the 'fsid' export option to tell knfsd to export the directories as if they were different filesystems.

For instance if I have

        /export/home	 141.211.133.0/24(rw,sync,no_subtree_check,fsid=0)
        /export/home/trondmy 141.211.133.0/24(rw,sync,no_subtree_check,fsid=3)

in /etc/exports on the server, then I can do

mount -t nfs -oro server:/export/home /mnt/foo mount -t nfs -orw server:/export/home/trondmy /mnt/bar

Quoted 6 lines

with no trouble. From 'cat /proc/mounts'

server:/export/home /mnt/foo nfs rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=server 0 0 server:/export/home/trondmy /mnt/bar nfs ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=server 0 0

as expected.

Oh - promising.

So - would I need to make the fsid unique for each mount - or could I just divide them into two pools. 'fsid=1' for all 'read-only' exported filesystems and 'fsid=2' for all read-write exported filesystems?

We could possibly make something like this work.

I could possibly generate unique fsid's for each export - but if the above simple logic would be sufficient - that would be easiest.

Quoted 5 lines
Quoted 5 lines

Can you point me to where the required 'mount --bind' fixes are being discussed? I haven't been able to find the thread in the 'lkml' (not that it isn't there somewhere).

Quoted 7 lines
Quoted 2 lines

Without the matching 'mount --bind' fixes this is a very serious regression for us?

I believe the most recent patchsets can be found from

http://marc.theaimsgroup.com/?l=linux-kernel&m=115447687730218&w=2

Thanx for the pointer - I'll track this thread down....

_Mike_

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs