Skip to contentSkip to conversation

nfsd4_stateowners eating memory like candy... sometimes....

DISCUSSION Todd Freeman 2 participants 3 messages Feb 8, 2012 – Feb 9, 2012

Subsystems (2)
NFS client NFS server
lore
Share & tools
lore

Todd Freeman

author
#0 lore raw

Good day all!

I have a nfs server handling the load for a shared file system for 5 web servers... some relevant info:
libnfsidmap2
0.23-2
nfs-common
1:1.2.2-1ubuntu1.1
nfs-kernel-server
1:1.2.2-1ubuntu1.1

Linux webnfs 2.6.35-31-server #63-Ubuntu SMP Mon Nov 28 21:03:37 UTC
2011 x86_64 GNU/Linux

On this server everything runs great for a couple weeks to a month and then we start getting sluggish performance... and within a couple days it seizes up (at least all nfs services stop... console is still accessible)

In trying to debug this we have been taking a snap shot every 5 minutes of the slabinfo... we got a totally clean capture this time and I see nfsd4_stateowners running away with memory. When we start the server and for the first several days the most memory it uses is 200MB or so... over time though there come points were it suddenly starts munching more... sometimes slowly... other times instantly. It finally kills the machine when it reaches the 1.7-1.8 GB level (just under the memory size of the machine). oom-killer is killing everything left and right at the end and we end up with a machine that is comatose NFS wise till we do a full reboot.

You can see a graph of this usage pattern at: http://imgur.com/ecLPh

I see mentions of a problem along this line back in the 2.6.16-18 types days... but supposedly it was fixed.

Does anyone have any ideas?

Signature · 5 lines
Todd Freeman  Ext 6103                   .^.    Don't fear the penguins!
Programming Department                   /V\
Andrews University                      // \\    http://www.linux.org/
http://www.andrews.edu/~freeman/       /(   )\   http://www.debian.org/
                                        ^^ ^^
#1 lore raw
Quoted 30 lines · Todd Freeman

Good day all!

I have a nfs server handling the load for a shared file system for 5 web servers... some relevant info:
libnfsidmap2
0.23-2
nfs-common
1:1.2.2-1ubuntu1.1
nfs-kernel-server
1:1.2.2-1ubuntu1.1

Linux webnfs 2.6.35-31-server #63-Ubuntu SMP Mon Nov 28 21:03:37
UTC 2011 x86_64 GNU/Linux

On this server everything runs great for a couple weeks to a month and then we start getting sluggish performance... and within a couple days it seizes up (at least all nfs services stop... console is still accessible)

In trying to debug this we have been taking a snap shot every 5
minutes of the slabinfo... we got a totally clean capture this
time and I see nfsd4_stateowners running away with memory. When we start the server and for the first several days the most memory it uses is 200MB or so... over time though there come points were it suddenly starts munching more... sometimes slowly... other times instantly. It finally kills the machine when it reaches the 1.7-1.8 GB level (just under the memory size of the machine). oom-killer is killing everything left and right at the end and we end up with a machine that is comatose NFS wise till we do a full reboot.

Quoted 4 lines

You can see a graph of this usage pattern at: http://imgur.com/ecLPh

I see mentions of a problem along this line back in the 2.6.16-18 types days... but supposedly it was fixed.

There have been a number of stateowner leaks fixed since 2.6.35. I think all the ones I know of were fixes as of 3.1 or so.

--b.

author
#2 lore raw

Thanks! I'll try a kernel update and see what sticks :)

Quoted 43 lines · J. Bruce Fields
Quoted 30 lines · Todd Freeman

Good day all!

I have a nfs server handling the load for a shared file system for 5 web servers... some relevant info:
libnfsidmap2
0.23-2
nfs-common
1:1.2.2-1ubuntu1.1
nfs-kernel-server
1:1.2.2-1ubuntu1.1

   Linux webnfs 2.6.35-31-server #63-Ubuntu SMP Mon Nov 28 21:03:37

UTC 2011 x86_64 GNU/Linux

On this server everything runs great for a couple weeks to a month and then we start getting sluggish performance... and within a couple days it seizes up (at least all nfs services stop... console is still accessible)

In trying to debug this we have been taking a snap shot every 5
minutes of the slabinfo... we got a totally clean capture this
time and I see nfsd4_stateowners running away with memory. When we start the server and for the first several days the most memory it uses is 200MB or so... over time though there come points were it suddenly starts munching more... sometimes slowly... other times instantly. It finally kills the machine when it reaches the 1.7-1.8 GB level (just under the memory size of the machine). oom-killer is killing everything left and right at the end and we end up with a machine that is comatose NFS wise till we do a full reboot.

Quoted 4 lines

You can see a graph of this usage pattern at: http://imgur.com/ecLPh

I see mentions of a problem along this line back in the 2.6.16-18 types days... but supposedly it was fixed.

There have been a number of stateowner leaks fixed since 2.6.35. I think all the ones I know of were fixes as of 3.1 or so.

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Todd Freeman  Ext 6103                   .^.    Don't fear the penguins!
Programming Department                   /V\
Andrews University                      // \\    http://www.linux.org/
http://www.andrews.edu/~freeman/       /(   )\   http://www.debian.org/
                                        ^^ ^^