Performance: NFS v3 mounting with 'noac' option

Hi All,

Trying to understand what performance impact could result if we mount the file system with 'noac' option.

We have multiple file systems begin mounted from the same NFS server. One of our user's application requires to mount the file system with 'noac' option.

We have user directories under /scratch file system, Now we would like to mount this file system with 'noac' option so that we can make one of our user happy. But this would afftect every user. Isn't it true?

So Now if we just decide to create a new FS /newScratchUserA and mount it with 'noac' option; WILL THIS AFFECT the other files systems mounted from the same server, if so how bad could it be.

Are there any other alternatives to reduce the impact of using this option on other mounted File systems.

Quoted 1 line

We have multiple file systems begin mounted from

the

Quoted 2 lines

same NFS server. One of our user's application
requires to mount the file system with 'noac'

option.

Quoted 2 lines

What about this application requires that the "noac" option be used?

One of the impacts of using the "noac" option is much more network
traffic and it will be composed of small packets. If the server is
busy handling all of the additional GETATTR requests, then it won't
be handling other requests. The client will also be a bit busier
with generating the additional requests, instead of just doing what
might be considered more useful work.

So, there should be no direct impact to other
mounted file systems.
However, there may be indirect impacts.

Quoted 7 lines · --- Peter Staubach

Quoted 2 lines

We have multiple file systems begin mounted from

the

Quoted 3 lines

same NFS server. One of our user's application
requires to mount the file system with 'noac'

Quoted 7 lines

option.

Quoted 2 lines

What about this application requires that the "noac" option be used?

application is romio with mpich2 ...

Quoted 12 lines

One of the impacts of using the "noac" option is much more network
traffic and it will be composed of small packets. If the server is
busy handling all of the additional GETATTR requests, then it won't
be handling other requests. The client will also be a bit busier
with generating the additional requests, instead of just doing what
might be considered more useful work.

Quoted 4 lines

So, there should be no direct impact to other
mounted file systems.
However, there may be indirect impacts.

indirect impacts: can you please elaborate on this.

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

What about this application requires that the

"noac"

Quoted 3 lines

Quoted 2 lines

option be used?

application is romio with mpich2 ...

Sorry, I'm not familiar with those. What about either one of those
require "noac" semantics?

Quoted 9 lines

Quoted 9 lines

One of the impacts of using the "noac" option is
much more network
traffic and it will be composed of small packets.

If the server is
busy handling all of the additional GETATTR
requests, then it won't
be handling other requests. The client will also

be

Quoted 2 lines

Quoted 2 lines

a bit busier
with generating the additional requests, instead

of

Quoted 9 lines

Quoted 7 lines

just doing what
might be considered more useful work.

So, there should be no direct impact to other
mounted file systems.
However, there may be indirect impacts.

indirect impacts: can you please elaborate on

this.

I thought that I did. :-)

If the system is busy handling the "noac" aspects of one file system,
then it isn't doing something else on one of the other file systems.
System resources are limited and using up more of them for one thing
leaves less for others.

ps

Quoted 6 lines · --- Peter Staubach

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

What about this application requires that the

"noac"

Quoted 9 lines

Quoted 5 lines

Quoted 3 lines

option be used?

application is romio with mpich2 ...

Sorry, I'm not familiar with those. What about
either one of those
require "noac" semantics?

"To use ROMIO on NFS, file locking with fcntl must work correctly on the NFS installation. On some installations, fcntl locks don't work. To get them to work, you need to use Version 3 of NFS, ensure that the lockd daemon is running on all the machines, and have the system administrator mount the NFS file system with the "noac" option (no attribute caching). Turning off attribute caching may reduce performance, but it is necessary for correct behavior."

Quoted 9 lines · AK

"To use ROMIO on NFS, file locking with fcntl must work correctly on the NFS installation. On some installations, fcntl locks don't work. To get them to work, you need to use Version 3 of NFS, ensure that the lockd daemon is running on all the machines, and have the system administrator mount the NFS file system with the "noac" option (no attribute caching). Turning off attribute caching may reduce performance, but it is necessary for correct behavior."

I think that I would assume that fcntl style locking works on all current implementations, that this is not NFS version dependent, and also that "noac" is not required in combination with the locking in order to get correct behavior.

"The following are some instructions we received from Ian Wells of HP for setting the noac option on NFS. We have not tried them ourselves. We are including them here because you may find them useful. Note that some of the steps may be specific to HP systems, and ..."

Quoted 2 lines · Peter Staubach

Quoted 1 line · AK

"To use ROMIO on NFS, file locking with fcntl

must

Quoted 2 lines

Quoted 2 lines

work correctly on the NFS installation. On some
installations, fcntl locks don't work. To get

them to

Quoted 1 line

Quoted 1 line

work, you need to use Version 3 of NFS, ensure

that

Quoted 1 line

Quoted 1 line

the lockd daemon is running on all the machines,

and

Quoted 2 lines

Quoted 2 lines

have the system administrator mount the NFS file
system with the "noac" option (no attribute

caching).

Quoted 1 line

Quoted 1 line

Turning off attribute caching may reduce

performance,

Quoted 3 lines

Quoted 1 line

but it is necessary for correct behavior."

I think that I would assume that fcntl style

locking works on all current

Quoted 1 line

implementations, that this is not NFS version

dependent, and also that

Quoted 1 line

"noac" is not required in combination with the

locking in order to get

Quoted 1 line

correct behavior.

Ouch - I agree with Peter, this advice is way, way out of date.
<http://www-unix.mcs.anl.gov/romio/>
<http://www-unix.mcs.anl.gov/romio/nfs-instr.txt>

Quoted 1 line

"The following are some instructions we received

from Ian Wells of HP

Quoted 1 line

for setting the noac option on NFS. We have not

tried them

Quoted 1 line

ourselves. We are including them here because you

may find

Quoted 1 line

them useful. Note that some of the steps may be

specific to HP

Quoted 1 line

systems, and ..."

Leave noac alone, the Linux client will do the right thing with
its cache when locking is available.

Tom.

Hmmm..
Tom & Peter, thank you for feedbacks ...

I agree, but when I read through the web and other doucmentation ... I found that if NFS's local caching is enable, doing a parallel IO on a file is a problem some, because there is no global synchronization.

Please correct me if I am wrong. I believe GFS works that way and we don't need to turn off caching there.

With the current setup when the user runs the MPI I/O program, he runs into following errors.

File locking failed in ADIOI_Set_lock. If the file system is NFS, you need to use NFS version 3, ensure that the lockd daemon is running on all the machines, and mount the directory with the 'noac' option (no attribute caching).

Is there an alternative to this ?? One more thing since we have nfs on our rocks cluster, the nfsd, rpc.statd etc are not started on the compute nodes ...will starting this help the file locking ??

~thank you!

Quoted 38 lines · Thomas

Quoted 2 lines · Peter Staubach

Quoted 1 line · AK

"To use ROMIO on NFS, file locking with fcntl

must

Quoted 2 lines

Quoted 2 lines

work correctly on the NFS installation. On some
installations, fcntl locks don't work. To get

them to

Quoted 1 line

Quoted 1 line

work, you need to use Version 3 of NFS, ensure

that

Quoted 1 line

Quoted 1 line

the lockd daemon is running on all the machines,

and

Quoted 2 lines

Quoted 2 lines

have the system administrator mount the NFS file
system with the "noac" option (no attribute

caching).

Quoted 1 line

Quoted 1 line

Turning off attribute caching may reduce

performance,

Quoted 3 lines

Quoted 1 line

but it is necessary for correct behavior."

I think that I would assume that fcntl style

locking works on all current

Quoted 1 line

implementations, that this is not NFS version

dependent, and also that

Quoted 1 line

"noac" is not required in combination with the

locking in order to get

Quoted 1 line

correct behavior.

Ouch - I agree with Peter, this advice is way, way out of date.
<http://www-unix.mcs.anl.gov/romio/>
<http://www-unix.mcs.anl.gov/romio/nfs-instr.txt>

Quoted 1 line

"The following are some instructions we received

from Ian Wells of HP

Quoted 1 line

for setting the noac option on NFS. We have not

tried them

Quoted 1 line

ourselves. We are including them here because you

may find

Quoted 1 line

them useful. Note that some of the steps may be

Quoted 10 lines

specific to HP

Quoted 1 line

systems, and ..."

Leave noac alone, the Linux client will do the right thing with
its cache when locking is available.

Tom.

______________________________________________________________________
______________
Now that's room service! Choose from over 150,000 hotels in 45,000 destinations on Yahoo! Travel to find your fit.
http://farechase.yahoo.com/promo-generic-14795097