troubling interaction with quotas and nfs

Hi all,
I'm running red hat 7.2 with the 2.4.9-31 kernel.
Its an ext3 file system, exported for nfsv3 udp and nfsv2 clients.

nothing fancy at all.

I have quotas set for my users. If a user hits their hard quota and the process continues to attempt to write to a file in their quota'd space
- the process load on the server shoots through the roof.

I can reliably recreate this by setting a hard quota, having the system hit the hard quota and continue to attempt to write to that file - the server becomes terribly unhappy.

I never saw this problem on older 2.2.X servers and I'm not sure why I'm seeing it now.

Any places I should be looking first?

thanks
-sv

GPG Public Key: http://www.phy.duke.edu/~skvidal/skvidal.gpg

tcpdump?
The server should return nfserr_dquot (69 == 0x45) and the client should return this to the application.
But it sounds like the client is repeating the request for some reason.

You could also
echo 32 > /proc/sys/sunrpc/nfsd_debug
and watch for
"nfsd: write complete err=%d"
messages in the kernel log. These will tell you whether nfsd thinks it got an error for the write.

NeilBrown

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

It appears to return -122

However, the program will exit correctly if it catches that problem. But the system does not respond well if there is a continued attempt to write a lot of data when the quota is filled.

That doesn't seem to make sense.

-sv

Can you elaborate on what the various values one could write to /proc/sys/sunrpc/nfsd_debug and what you would get with different values in terms of logging? This sounds like something that could be useful to many others and included in the HOWTO or FAQ as well. I've checked the kernel docs and they basically say to 'use the source,' but since we have some of the core NFS client/server developers here perhaps someone can offer a few hints for people debugging NFS configurations.

Also, is there any known or performance or stability hit to running with NFS debug on?

thanks,
andrew

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

"use the source" is really the right answer :-)
If you look in the source, it is fairly easy to see what to put in /proc/sys/sunrpc/nfsd_debug, and you can see what the results means.

If you don't look at the source, then the debugging output probably isn't very helpful.

However:
If you look in
include/linux/nfsd/debug.h
you will see:
/*
* knfsd debug flags
*/
#define NFSDDBG_SOCK 0x0001
#define NFSDDBG_FH 0x0002
#define NFSDDBG_EXPORT 0x0004
#define NFSDDBG_SVC 0x0008
#define NFSDDBG_PROC 0x0010
#define NFSDDBG_FILEOP 0x0020
#define NFSDDBG_AUTH 0x0040
#define NFSDDBG_REPCACHE 0x0080
#define NFSDDBG_XDR 0x0100
#define NFSDDBG_LOCKD 0x0200

So if you
echo 1 > /proc/sys/sunrpc/nfsd_debug
it will turn on "SOCK" debugging and turn the others off. If you
echo 1023 > /proc/sys/sunrpc/nfsd_debug
it will turn on all debugging.

Each file in fs/nfsd/*.c has something like
#define NFSDDBG_FACILITY NFSDDBG_FILEOP

so all dprintk calls in that file will use that facility and will only be printed if the right number has been written to
..../nfsd_debug

It will generate lots and lots (and lots) of messages which will definately hurt performance, just scrolling the console if nothing else.

I only turn it on for a few seconds at a time, or when I am generating a very small amount of carefully controlled traffic.

NeilBrown

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs