Where in the server code is fsinfo rtpref calculated?
Subsystems (2)
Hi. I'm struggling to locate the region in the NFSv3 server-side code where it figures out the block sizes for the FSINFO reply. We have servers that do not specify r/wsizes in their exports and so I need to find where this negotiated value between server->client actually comes from. How does the server reach the preferred block size for a given export?
Cheers,
Jim
Signature · 3 lines
Jim Vanns Senior Software Developer Framestore
Quoted 3 lines · James Vanns
Hi. I'm struggling to locate the region in the NFSv3 server-side code where it figures out the block sizes for the FSINFO reply. We have servers that do not specify r/wsizes in their exports
There's no way to specify that as an export option. You can configure it server-wide using /proc/fs/nfsd/max_block_size.
Quoted 4 lines
and so I need to
find where this negotiated value betweenserver->clientactually comes from. How does the server reach the preferred block size for a given export?
fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller, nfsd_create_serv(), calls svc_create_pooled() with the result that's calculated.
For fsinfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses svc_max_payload().
--b.
Quoted 3 lines
There's no way to specify that as an export option. You can configure
it server-wide using/proc/fs/nfsd/max_block_size.
Ah ha! Bingo. There it is. I can see it on our SL6 (2.6.32) servers but not on older RHEL5 (2.6.18) servers so I guess at some point this was hardcoded to 32k?
Quoted 10 lines
Quoted 6 lines
and so I need to
find where this negotiated value betweenserver->clientactually comes
from. How does the server reach the preferred block size for a given
export?fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller,
nfsd_create_serv(), callssvc_create_pooled()with the result that's calculated.
Thanks. I shall look there.
Jim
Quoted 10 lines
For fsinfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses
svc_max_payload().--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
Jim Vanns Senior Software Developer Framestore
Quoted 3 lines
fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller,
nfsd_create_serv(), callssvc_create_pooled()with the result that's calculated.
Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default and preferred blocksize negotiated with clients will almost always be 1MB - the maximum RPC payload. The nfsd_get_default_maxblksize() function seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll always prefer this upper bound instead of any value calculated according to available RAM.
For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.
Anyway, this file/function appears to set the default *max* blocksize. I haven't read all the related code yet, but does the preferred block size derive from this maximum too?
Thanks,
Jim
Quoted 10 lines
For fsinfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses
svc_max_payload().--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
Jim Vanns Senior Software Developer Framestore
Quoted 11 lines · James Vanns
Quoted 3 lines
fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller,
nfsd_create_serv(), callssvc_create_pooled()with the result that's calculated.Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default and preferred blocksize negotiated with clients will almost always be 1MB - the maximum RPC payload. The
nfsd_get_default_maxblksize()function seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll always prefer this upper bound instead of any value calculated according to available RAM.
Well, "obsolete" is an odd way to put it--the code is still expected to work on smaller machines.
Arguments welcome about the defaults, though I wonder whether it would be better to be doing this sort of calculation in user space.
Quoted 5 lines
For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.
Anyway, this file/function appears to set the default *max* blocksize. I haven't read all the related code yet, but does the preferred block size derive from this maximum too?
See:
Quoted 2 lines
Quoted 2 lines
For fsinfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses
svc_max_payload().
I'm not sure what the history is behind that logic, though.
--b.
Quoted 19 lines · James Vanns
Quoted 4 lines
Quoted 3 lines
fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller,
nfsd_create_serv(), callssvc_create_pooled()with the result that's calculated.Quoted 12 lines
Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default
and preferred blocksize negotiated with clients will almost always be
1MB - the maximum RPC payload. Thenfsd_get_default_maxblksize()function
seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll
always prefer this upper bound instead of any value calculated according to
available RAM.
Quoted 2 lines
Well, "obsolete" is an odd way to put it--the code is still expected to work on smaller machines.
Poor choice of words perhaps. I guess I'm just used to NFS servers being pretty hefty pieces of kit and 'small' workstations having a couple of GB of RAM too.
Quoted 2 lines
Arguments welcome about the defaults, thoodd ugh I wonder whether it would be better to be doing this sort of calculation in user space.
See below.
Quoted 12 lines
Quoted 8 lines
For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.
Anyway, this file/function appears to set the default *max* blocksize. I haven't
read all the related code yet, but does the preferred block size derive
from this maximum too?See
Quoted 2 lines
Quoted 2 lines
For finfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses
svc_max_payload().
I've just returned from nfsd3_proc_fsinfo() and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply stuffs the max_block_size value in both the maximum *and* preferred block sizes for both read and write. A 1MB block size for a preferred default is a little high! If a disk is reading at 33MB/s and we have just a single server running 64 knfsd and each READ call is requesting 1MB of data then all of a sudden we have an aggregate read speed of ~512k/s and that is without network latencies. And of course we will probably have 100s of requests queued behind each knfsd waiting for these 512k reads to finish. All of a sudden our user experience is rather poor :(
Perhaps a better suggestion would be to at least expose the maximum and preferred block sizes (for both read and write) via a sysctl key so an administrator can set it to the underlying block sizes of the file system or physical device?
Perhaps the defaults should at least be a smaller multiple of the page size or somewhere between that and the PDU of the network layer the service is bound too.
Just my tuppence - and my maths might be flawed ;)
Jim
Quoted 4 lines
I'm not sure what the history is behind that logic, though.
--b.
Signature · 3 lines
Jim Vanns Senior Software Developer Framestore
Quoted 44 lines · James Vanns
Quoted 19 lines · James Vanns
Quoted 4 lines
Quoted 3 lines
fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller,
nfsd_create_serv(), callssvc_create_pooled()with the result that's calculated.Quoted 12 lines
Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default
and preferred blocksize negotiated with clients will almost always be
1MB - the maximum RPC payload. Thenfsd_get_default_maxblksize()function
seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll
always prefer this upper bound instead of any value calculated according to
available RAM.Quoted 2 lines
Well, "obsolete" is an odd way to put it--the code is still expected to work on smaller machines.
Poor choice of words perhaps. I guess I'm just used to NFS servers being pretty hefty pieces of kit and 'small' workstations having a couple of GB of RAM too.
Quoted 2 lines
Arguments welcome about the defaults, thoodd ugh I wonder whether it would be better to be doing this sort of calculation in user space.
See below.
Quoted 12 lines
Quoted 8 lines
For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.
Anyway, this file/function appears to set the default *max* blocksize. I haven't
read all the related code yet, but does the preferred block size derive
from this maximum too?See
Quoted 2 lines
Quoted 2 lines
For finfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses
svc_max_payload().
Quoted 8 lines
I've just returned from
nfsd3_proc_fsinfo()and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply stuffs themax_block_sizevalue in both the maximum *and* preferred block sizes for both read and write. A 1MB block size for a preferred default is a little high! If a disk is reading at 33MB/s and we have just a single server running 64 knfsd and each READ call is requesting 1MB of data then all of a sudden we have an aggregate read speed of ~512k/s
I lost you here.
Quoted 4 lines
and
that is without network latencies. And of course we will probably have 100s of requests queued behind each knfsd waiting for these 512k reads to finish. All of a sudden our user experience is rather poor :(
Note the preferred size is not a minimum--the client isn't forced to do 1MB reads if it really only wants 1 page, for example, if that's what you mean.
(I haven't actually looked at how typical clients used rt/wtpref.)
--b.
Quoted 20 lines
Perhaps a better suggestion would be to at least expose the maximum and preferred block sizes (for both read and write) via a sysctl key so an administrator can set it to the underlying block sizes of the file system or physical device?
Perhaps the defaults should at least be a smaller multiple of the page size or somewhere between that and the PDU of the network layer the service is bound too.
Just my tuppence - and my maths might be flawed ;)
Jim
Quoted 4 lines
I'm not sure what the history is behind that logic, though.
--b.
Signature · 3 lines
Jim Vanns Senior Software Developer Framestore
Quoted 61 lines · J. Bruce Fields
Quoted 44 lines · James Vanns
Quoted 19 lines · James Vanns
Quoted 4 lines
Quoted 3 lines
fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller,
nfsd_create_serv(), callssvc_create_pooled()with the result that's calculated.Quoted 12 lines
Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default
and preferred blocksize negotiated with clients will almost always be
1MB - the maximum RPC payload. Thenfsd_get_default_maxblksize()function
seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll
always prefer this upper bound instead of any value calculated according to
available RAM.Quoted 2 lines
Well, "obsolete" is an odd way to put it--the code is still expected to work on smaller machines.
Poor choice of words perhaps. I guess I'm just used to NFS servers being pretty hefty pieces of kit and 'small' workstations having a couple of GB of RAM too.
Quoted 2 lines
Arguments welcome about the defaults, thoodd ugh I wonder whether it would be better to be doing this sort of calculation in user space.
See below.
Quoted 12 lines
Quoted 8 lines
For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.
Anyway, this file/function appears to set the default *max* blocksize. I haven't
read all the related code yet, but does the preferred block size derive
from this maximum too?See
Quoted 2 lines
Quoted 2 lines
For finfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses
svc_max_payload().Quoted 8 lines
I've just returned from
nfsd3_proc_fsinfo()and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply stuffs themax_block_sizevalue in both the maximum *and* preferred block sizes for both read and write. A 1MB block size for a preferred default is a little high! If a disk is reading at 33MB/s and we have just a single server running 64 knfsd and each READ call is requesting 1MB of data then all of a sudden we have an aggregate read speed of ~512k/sI lost you here.
Quoted 4 lines
and
that is without network latencies. And of course we will probably have 100s of requests queued behind each knfsd waiting for these 512k reads to finish. All of a sudden our user experience is rather poor :(
Quoted 5 lines
Note the preferred size is not a minimum--the client isn't forced to do 1MB reads if it really only wants 1 page, for example, if that's what you mean.
(I haven't actually looked at how typical clients used rt/wtpref.)
For our client, the answer is:
rtpref == default rsize
wtpref == default wsize and default f_bsize
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
<snip>
Quoted 15 lines
Quoted 6 lines
I've just returned from
nfsd3_proc_fsinfo()and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply
stuffs themax_block_sizevalue in both the maximum *and* preferredQuoted 8 lines
block sizes for both read and write. A 1MB block size for a preferred
default is a little high! If a disk is reading at 33MB/s and we have just
a single server running 64 knfsd and each READ call is requesting 1MB of
data then all of a sudden we have an aggregate read speed of ~512k/s
Quoted 1 line
I lost you here.
OK, so what we're seeing is the large majority of our nr. ~700 clients (all Linux 2.6.32 based NFS clients) issuing READ requests of 1MB in size.
After the initial MOUNT request has been granted an FSINFO call is made. The contents of the REPLY from the server (another Linux 2.6.32 server) include rtmax, rtpref, wtmax and wtpref all of which are set to 1MB. This 1MB appears to come from that code/explanation I described earlier - all values are basically getting set to whatever comes out of nfsd_get_default_max_blksize().
Quoted 10 lines
Quoted 5 lines
that is without network latencies. And of course we will probably have 100s of
requests queued behind each knfsd waiting for these 512k reads to finish. All of a
sudden our user experience is rather poor :(Note the preferred size is not a minimum--the client isn't forced to do
1MB reads if it really only wants 1 page, for example, if that's what you mean.
If no r/wsize has been specified on the client mount then the negotiated values above will be used by the client for any read() by an application exceeding that maximum. That maximum (the default 1MB) is still quite large I reckon.
I'm not sure at which point the preferred or optimal block size will be used by the client - because they're set as the same on the server side, I can't tell which is being used ;)
Quoted 30 lines
(I haven't actually looked at how typical clients used rt/wtpref.)
--b.
Quoted 25 lines
Perhaps a better suggestion would be to at least expose the maximum and preferred
block sizes (for both read and write) via a sysctl key so an administrator can set
it to the underlying block sizes of the file system or physical device?Perhaps the defaults should at least be a smaller multiple of the page size or somewhere
between that and the PDU of the network layer the service is bound too.Just my tuppence - and my maths might be flawed ;)
Jim
Quoted 4 lines
I'm not sure what the history is behind that logic, though.
--b.
--
Jim Vanns
Senior Software Developer
Framestore
Signature · 3 lines
Jim Vanns Senior Software Developer Framestore
Quoted 12 lines · James Vanns
<snip>
Quoted 10 lines
Quoted 9 lines
I've just returned from
nfsd3_proc_fsinfo()and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply stuffs themax_block_sizevalue in both the maximum *and* preferred block sizes for both read and write. A 1MB block size for a preferred default is a little high! If a disk is reading at 33MB/s and we have just a single server running 64 knfsd and each READ call is requesting 1MB of data then all of a sudden we have an aggregate read speed of ~512k/s
Quoted 5 lines
Quoted 1 line
I lost you here.
OK, so what we're seeing is the large majority of our nr. ~700 clients (all Linux 2.6.32 based NFS clients) issuing READ requests of 1MB in size.
Knowing nothing about your situation, I'd assume the clients are doing that because they actually want that 1MB of data.
Would you prefer they each send 1024 1k READs? I don't understand why it's the read size you're focused on here.
--b.
Quoted 7 lines
After the initial MOUNT request has been granted an FSINFO call is made. The contents of the REPLY from the server (another Linux 2.6.32 server) include rtmax, rtpref, wtmax and wtpref all of which are set to 1MB. This 1MB appears to come from that code/explanation I described earlier - all values are basically getting set to whatever comes out of
nfsd_get_default_max_blksize().
Quoted 2 lines
Knowing nothing about your situation, I'd assume the clients are doing that because they actually want that 1MB of data.
Possibly. But we have no control over that (the application read size, I mean).
Quoted 2 lines
Would you prefer they each send 1024 1k READs? I don't understand why it's the read size you're focused on here.
No. But 32x 32k reads is reasonable (because it gives other RPCs a look-in). I'm focused on reads because it makes up the majority of our NFS traffic. I'm concerned because as it stands (out of the box) if the majority of our n knfsd threads are waiting for a 1MB read to return then no other RPC request will be serviced and will just contribute to the backlog. This backlog itself will probably also contain a hefty no. of 1MB read requests too. In short, a lot of other RPC calls that are not reads will just be blocking and this will appear to an end user as poor performance.
We deal with a great number of fairly large files - 10s of GBs in size. We just don't want others to suffer because of large request sizes coming in (writes end up being of the same size too but there are less of them). Our use cases are varied but they all have to share the same resource (the array of NFS servers).
We've only really seen this since our upgrade to SL6/kernel 2.6.32. I guess previously that 32k was some sort of default or limit?
Related to this was my query on when/how the (Linux) client may honour the preferred or optimal block size given in the FSINFO reply. Any ideas? Is it if a read of less than that preferred block size is requested then the preferred is used anyway because it comes at the same cost?
Thanks,
Jim
Quoted 20 lines
--b.
Quoted 10 lines
After the initial MOUNT request has been granted an FSINFO call is made. The contents of the REPLY from the server (another Linux 2.6.32
server) include rtmax, rtpref, wtmax and wtpref all of which are set
to 1MB. This 1MB appears to come from that code/explanation I described earlier - all values are basically getting set to whatever
comes out ofnfsd_get_default_max_blksize().--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
Jim Vanns Senior Software Developer Framestore
Quoted 11 lines · James Vanns
Quoted 2 lines
Knowing nothing about your situation, I'd assume the clients are doing that because they actually want that 1MB of data.
Possibly. But we have no control over that (the application read size, I mean).
Quoted 2 lines
Would you prefer they each send 1024 1k READs? I don't understand why it's the read size you're focused on here.
No. But 32x 32k reads is reasonable (because it gives other RPCs a look-in).
Maybe. In any case I'd want to see data before changing our defaults.
Quoted 8 lines
I'm focused on reads because it makes up the majority of
our NFS traffic. I'm concerned because as it stands (out of the box) if the majority of our n knfsd threads are waiting for a 1MB read to return then no other RPC request will be serviced and will just contribute to the backlog. This backlog itself will probably also contain a hefty no. of 1MB read requests too. In short, a lot of other RPC calls that are not reads will just be blocking and this will appear to an end user as poor performance.
Do you have a performance problem that you've actually measured, and if so could you share the details?
Quoted 6 lines
We deal with a great number of fairly large files - 10s of GBs in size. We just don't want others to suffer because of large request sizes coming in (writes end up being of the same size too but there are less of them). Our use cases are varied but they all have to share the same resource (the array of NFS servers).
Quoted 8 lines
We've only really seen this since our upgrade to SL6/kernel 2.6.32. I guess previously that 32k was some sort of default or limit?
Related to this was my query on when/how the (Linux) client may honour the preferred or optimal block size given in the FSINFO reply. Any ideas? Is it if a read of less than that preferred block size is requested then the preferred is used anyway because it comes at the same cost?
I'm not terribly familiar with the client logic, but would expect this to vary depending on kernel version, read-ahead policy, application behavior and a number of other factors, so I'd recommend testing with your workload and finding out.
--b.