Kernel 2.6 vs. Kernel 2.4: higher NFS traffic
Subsystems (2)
Hello.
I have an environment where most of my hosts use kernel 2.4, and I'm testing a new host with kernel 2.6 on this environment. Both hosts run the same software, it was only ported to the more recent O.S. The software is POP3/IMAP, the mailboxes are in NFS, so that it reads a lot from the eth1 network interface. That interface is used only for NFS.
I noticed that the host that has kernel 2.6 (smtp-fe14) has a much higher network traffic than the other 2.4 hosts (smtp-fe01 up to smtp-fe13). You can see it on the MRTG attached. The gray line is the inbound traffic at smtp-fe14's network interface used for NFS only. It reaches around 80-90Mbps, while the other hosts use around 60Mbps.
I use only NFSv3, and all accesses are done on NFS with non-root user.
So I ask: Is this behaviour expected? Did someone else notice that on kernel 2.6? Can this be affecting my storage, for example, increasing the RPC's that are being done in the server? (I don't have how to check it, because this host is responsible for only a small part of RPC's on the server, and I can't really see such a difference on total of RPC's on the server). Can this be related to prefetch? How can I tune NFS prefetch in a kernel 2.6?
Thanks a lot,
Filipe
Hi,
Regarding my net traffic problem, I noticed with tcpdump that a lot of operations are RPC calls to the ACCESS function, then I read the FAQ and found out the following:
http://nfs.sourceforge.net/#faq_b11
"The Linux NFS client should cache the results of these ACCESS operations. In fact, in the new 2.6.x kernels, it does this and it extends ACCESS checking to all users to allow for generic uid/gid mapping on the server. This also enables proper support for Access Control Lists in the server's local file system. In pre-2.6 kernels, the stock NFS client does not cache the results of ACCESS operations."
Is there a way to disable this behaviour? To disable the ACCESS calls for non-root users on kernel 2.6? I run all my applications as non-root user, and one of the reasons for that is the lots of RPC calls to v3access that begun to appear when upgrading from NFSv2 to NFSv3, so that I actually see this behaviour on 2.6 a little as a negative factor for upgrading from 2.4 to 2.6...
Thanks,
Filipe
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
ty den 26.07.2005 Klokka 20:18 (-0300) skreiv Filipe Brandenburger:
Quoted 6 lines
Is there a way to disable this behaviour? To disable the ACCESS calls for non-root users on kernel 2.6? I run all my applications as non-root user, and one of the reasons for that is the lots of RPC calls to v3access that begun to appear when upgrading from NFSv2 to NFSv3, so that I actually see this behaviour on 2.6 a little as a negative factor for upgrading from 2.4 to 2.6...
There were a couple of kernels in the mid-2.6 series where the caching didn't work too well. More recent kernels should be fine, though.
There are no plans to allow users to disable ACCESS checking, however there are places where the access calls may be optimised away (I think I've done it for most of them now).
The main optimisation that remains is in the open() codepath, where we currently do a GETATTR, then ACCESS. In the case where the server supports post-op attributes on ACCESS, then that case can be reduced to a single call to ACCESS (i.e. we get rid of the GETATTR call).
Cheers,
Trond
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Hi.
Thanks for the answer.
* Tue, 26 Jul 2005 20:46:03 -0400, Trond Myklebust <[email protected]>:
Quoted 2 lines
There were a couple of kernels in the mid-2.6 series where the caching didn't work too well. More recent kernels should be fine, though.
My kernel is 2.6.9, is it recent enough for the right behaviour of caching code?
As to the lots of "access" calls that I found, can it be that they're causing the increase of about 20% to the eth1 network inbound traffic? As I see, "access" responses are or should be small. I don't think it could be the reason why I'm having such an increase in NFS network traffic... Any other thoughts about that?
I also gave a look at nfsstat, and found out something interesting. More specifically, nfsstat -rc.
On kernel 2.4, I have this:
[root@congo root]# nfsstat -rc
Client rpc stats:
calls retrans authrefrsh
844988800 8307 0
The average growth of "calls" is about 240k/min, and "retrans" grows about 3 or 4 per minute.
On kernel 2.6:
[root@talara ~]# nfsstat -rc
Client rpc stats:
calls retrans authrefrsh 8933863 9543 0
The average growth of "calls" is about 410k/min. Ok, I can see that as the "access" calls that are being done only on kernel 2.6, this could justify the big growth on RPC calls done on these clients. But on this host, "retrans" grows at a rate of 400 to 500 per minute, and that seems to be a lot to me.
I think it has something to do with the timeo= mount parameter, so I tried to set timeo=30 (3 seconds) on all my NFS mounts (I used the default of 7 tenths of seconds before). But even after the change, the "retrans" rate seems to keep high.
I don't seem to be with problems in reassembling fragments, as seen below.
On kernel 2.4:
[root@congo root]# netstat -s | grep -i -e frag -e reass
137 fragments dropped after timeout
1016022512 reassemblies required
213674528 packets reassembled ok
137 packet reassembles failed
2234663 fragments createdOn kernel 2.6:
[root@talara ~]# netstat -s | grep -i -e frag -e reass
5 fragments dropped after timeout 10776696 reassemblies required
2124805 packets reassembled ok
5 packet reassembles failed
8825 fragments received ok
Both seem very low, and neither of them is growing.
Anyone has any ideas on this?
Thanks a lot,
Filipe.
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
on den 27.07.2005 Klokka 11:41 (-0300) skreiv Filipe Brandenburger:
Quoted 10 lines
Hi.
Thanks for the answer.
* Tue, 26 Jul 2005 20:46:03 -0400, Trond Myklebust <[email protected]>:
Quoted 2 lines
There were a couple of kernels in the mid-2.6 series where the caching didn't work too well. More recent kernels should be fine, though.
My kernel is 2.6.9, is it recent enough for the right behaviour of caching code?
I'd move to 2.6.12. There have been several patches to the access code since 2.6.9.
Quoted 5 lines
As to the lots of "access" calls that I found, can it be that they're causing the increase of about 20% to the eth1 network inbound traffic? As I see, "access" responses are or should be small. I don't think it could be the reason why I'm having such an increase in NFS network traffic... Any other thoughts about that?
GETATTR?
Quoted 17 lines
I also gave a look at nfsstat, and found out something interesting. More specifically, nfsstat -rc.
On kernel 2.4, I have this:
[root@congo root]# nfsstat -rc
Client rpc stats:
calls retrans authrefrsh
844988800 8307 0The average growth of "calls" is about 240k/min, and "retrans" grows about 3 or 4 per minute.
On kernel 2.6:
[root@talara ~]# nfsstat -rc
Client rpc stats:
Quoted 8 lines
calls retrans authrefrsh 8933863 9543 0The average growth of "calls" is about 410k/min. Ok, I can see that as the "access" calls that are being done only on kernel 2.6, this could justify the big growth on RPC calls done on these clients. But on this host, "retrans" grows at a rate of 400 to 500 per minute, and that seems to be a lot to me.
Sounds like a UDP networking regression. Could be related to anything from scheduling changes to NIC driver changes.
Quoted 4 lines
I think it has something to do with the timeo= mount parameter, so I tried to set timeo=30 (3 seconds) on all my NFS mounts (I used the default of 7 tenths of seconds before). But even after the change, the "retrans" rate seems to keep high.
Recent kernels (both 2.4 and 2.6) use a self-tuning algorithm for UDP.
Cheers,
Trond
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
* Wed, 27 Jul 2005 13:17:15 -0400, Trond Myklebust <[email protected]>:
Quoted 2 lines
I'd move to 2.6.12. There have been several patches to the access code since 2.6.9.
Ok, I'll consider that. Thanks for the tip.
Quoted 7 lines
Quoted 5 lines
As to the lots of "access" calls that I found, can it be that they're causing the increase of about 20% to the eth1 network inbound traffic? As I see, "access" responses are or should be small. I don't think it could be the reason why I'm having such an increase in NFS network traffic... Any other thoughts about that?
GETATTR?
Yes, actually the number of GETATTR calls raised a lot from 2.4 to 2.6. Why exactly is that? 2.4 didn't call GETATTR as much as 2.6 does.
I found out with tcpdump that these are the average number of operation calls in 10.000 packets on kernel 2.6:
12 link
13 rename
14 commit
14 write
32 readdir
45 remove
200 setattr
666 readdirplus 1902 lookup
1929 read
2085 getattr
3088 access
And the same on 10.000 packets on kernel 2.4:
2 create
52 rename
60 commit
62 write
79 getattr
93 link
115 remove
238 setattr
667 readdir 3243 read
5389 lookup
Why is there so much difference? Why does 2.6 make 21% getattr while 2.4 does less then 1%?
Are there any mount options that I can use to do some tuning on this?
Thanks,
Filipe
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
on den 27.07.2005 Klokka 14:25 (-0300) skreiv Filipe Brandenburger:
Quoted 2 lines
Why is there so much difference? Why does 2.6 make 21% getattr while 2.4 does less then 1%?
We no longer allow post-op attributes to update the attribute cache. It leads to some nasty races.
Quoted 1 line
Are there any mount options that I can use to do some tuning on this?
No.
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs