Skip to contentSkip to conversation

Where in the server code is fsinfo rtpref calculated?

DISCUSSION James Vanns 3 participants 12 messages May 14, 2013 – May 17, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi. I'm struggling to locate the region in the NFSv3 server-side code where it figures out the block sizes for the FSINFO reply. We have servers that do not specify r/wsizes in their exports and so I need to find where this negotiated value between server->client actually comes from. How does the server reach the preferred block size for a given export?

Cheers,

Jim

Signature · 3 lines
Jim Vanns
Senior Software Developer
Framestore
#1 lore raw
Quoted 3 lines · James Vanns

Hi. I'm struggling to locate the region in the NFSv3 server-side code where it figures out the block sizes for the FSINFO reply. We have servers that do not specify r/wsizes in their exports

There's no way to specify that as an export option. You can configure it server-wide using /proc/fs/nfsd/max_block_size.

Quoted 4 lines

and so I need to
find where this negotiated value between server->client actually comes from. How does the server reach the preferred block size for a given export?

fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller, nfsd_create_serv(), calls svc_create_pooled() with the result that's calculated.

For fsinfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses svc_max_payload().

--b.

#2 lore raw
Quoted 3 lines

There's no way to specify that as an export option. You can configure
it server-wide using /proc/fs/nfsd/max_block_size.

Ah ha! Bingo. There it is. I can see it on our SL6 (2.6.32) servers but not on older RHEL5 (2.6.18) servers so I guess at some point this was hardcoded to 32k?

Quoted 10 lines
Quoted 6 lines

and so I need to
find where this negotiated value between server->client actually comes
from. How does the server reach the preferred block size for a given
export?

fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller, nfsd_create_serv(), calls svc_create_pooled() with the result that's calculated.

Thanks. I shall look there.

Jim

Quoted 10 lines

For fsinfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses svc_max_payload().

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Jim Vanns
Senior Software Developer
Framestore
#3 lore raw
In reply to J. Bruce Fields
Quoted 3 lines

fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller, nfsd_create_serv(), calls svc_create_pooled() with the result that's calculated.

Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default and preferred blocksize negotiated with clients will almost always be 1MB - the maximum RPC payload. The nfsd_get_default_maxblksize() function seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll always prefer this upper bound instead of any value calculated according to available RAM.

For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.

Anyway, this file/function appears to set the default *max* blocksize. I haven't read all the related code yet, but does the preferred block size derive from this maximum too?

Thanks,

Jim

Quoted 10 lines

For fsinfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses svc_max_payload().

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Jim Vanns
Senior Software Developer
Framestore
#4 lore raw
Quoted 11 lines · James Vanns
Quoted 3 lines

fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller, nfsd_create_serv(), calls svc_create_pooled() with the result that's calculated.

Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default and preferred blocksize negotiated with clients will almost always be 1MB - the maximum RPC payload. The nfsd_get_default_maxblksize() function seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll always prefer this upper bound instead of any value calculated according to available RAM.

Well, "obsolete" is an odd way to put it--the code is still expected to work on smaller machines.

Arguments welcome about the defaults, though I wonder whether it would be better to be doing this sort of calculation in user space.

Quoted 5 lines

For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.

Anyway, this file/function appears to set the default *max* blocksize. I haven't read all the related code yet, but does the preferred block size derive from this maximum too?

See:

Quoted 2 lines
Quoted 2 lines

For fsinfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses svc_max_payload().

I'm not sure what the history is behind that logic, though.

--b.

#5 lore raw
Quoted 19 lines · James Vanns
Quoted 4 lines
Quoted 3 lines

fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller, nfsd_create_serv(), calls svc_create_pooled() with the result that's calculated.

Quoted 12 lines

Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default
and preferred blocksize negotiated with clients will almost always be
1MB - the maximum RPC payload. The nfsd_get_default_maxblksize() function
seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll
always prefer this upper bound instead of any value calculated according to
available RAM.

Quoted 2 lines

Well, "obsolete" is an odd way to put it--the code is still expected to work on smaller machines.

Poor choice of words perhaps. I guess I'm just used to NFS servers being pretty hefty pieces of kit and 'small' workstations having a couple of GB of RAM too.

Quoted 2 lines

Arguments welcome about the defaults, thoodd ugh I wonder whether it would be better to be doing this sort of calculation in user space.

See below.

Quoted 12 lines
Quoted 8 lines

For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.

Anyway, this file/function appears to set the default *max* blocksize. I haven't
read all the related code yet, but does the preferred block size derive
from this maximum too?

See

Quoted 2 lines
Quoted 2 lines

For finfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses svc_max_payload().

I've just returned from nfsd3_proc_fsinfo() and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply stuffs the max_block_size value in both the maximum *and* preferred block sizes for both read and write. A 1MB block size for a preferred default is a little high! If a disk is reading at 33MB/s and we have just a single server running 64 knfsd and each READ call is requesting 1MB of data then all of a sudden we have an aggregate read speed of ~512k/s and that is without network latencies. And of course we will probably have 100s of requests queued behind each knfsd waiting for these 512k reads to finish. All of a sudden our user experience is rather poor :(

Perhaps a better suggestion would be to at least expose the maximum and preferred block sizes (for both read and write) via a sysctl key so an administrator can set it to the underlying block sizes of the file system or physical device?

Perhaps the defaults should at least be a smaller multiple of the page size or somewhere between that and the PDU of the network layer the service is bound too.

Just my tuppence - and my maths might be flawed ;)

Jim

Quoted 4 lines

I'm not sure what the history is behind that logic, though.

--b.

Signature · 3 lines
Jim Vanns
Senior Software Developer
Framestore
#6 lore raw
Quoted 44 lines · James Vanns
Quoted 19 lines · James Vanns
Quoted 4 lines
Quoted 3 lines

fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller, nfsd_create_serv(), calls svc_create_pooled() with the result that's calculated.

Quoted 12 lines

Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default
and preferred blocksize negotiated with clients will almost always be
1MB - the maximum RPC payload. The nfsd_get_default_maxblksize() function
seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll
always prefer this upper bound instead of any value calculated according to
available RAM.

Quoted 2 lines

Well, "obsolete" is an odd way to put it--the code is still expected to work on smaller machines.

Poor choice of words perhaps. I guess I'm just used to NFS servers being pretty hefty pieces of kit and 'small' workstations having a couple of GB of RAM too.

Quoted 2 lines

Arguments welcome about the defaults, thoodd ugh I wonder whether it would be better to be doing this sort of calculation in user space.

See below.

Quoted 12 lines
Quoted 8 lines

For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.

Anyway, this file/function appears to set the default *max* blocksize. I haven't
read all the related code yet, but does the preferred block size derive
from this maximum too?

See

Quoted 2 lines
Quoted 2 lines

For finfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses svc_max_payload().

Quoted 8 lines

I've just returned from nfsd3_proc_fsinfo() and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply stuffs the max_block_size value in both the maximum *and* preferred block sizes for both read and write. A 1MB block size for a preferred default is a little high! If a disk is reading at 33MB/s and we have just a single server running 64 knfsd and each READ call is requesting 1MB of data then all of a sudden we have an aggregate read speed of ~512k/s

I lost you here.

Quoted 4 lines

and
that is without network latencies. And of course we will probably have 100s of requests queued behind each knfsd waiting for these 512k reads to finish. All of a sudden our user experience is rather poor :(

Note the preferred size is not a minimum--the client isn't forced to do 1MB reads if it really only wants 1 page, for example, if that's what you mean.

(I haven't actually looked at how typical clients used rt/wtpref.)

--b.

Quoted 20 lines

Perhaps a better suggestion would be to at least expose the maximum and preferred block sizes (for both read and write) via a sysctl key so an administrator can set it to the underlying block sizes of the file system or physical device?

Perhaps the defaults should at least be a smaller multiple of the page size or somewhere between that and the PDU of the network layer the service is bound too.

Just my tuppence - and my maths might be flawed ;)

Jim

Quoted 4 lines

I'm not sure what the history is behind that logic, though.

--b.

Signature · 3 lines
Jim Vanns
Senior Software Developer
Framestore
#7 lore raw
Quoted 61 lines · J. Bruce Fields
Quoted 44 lines · James Vanns
Quoted 19 lines · James Vanns
Quoted 4 lines
Quoted 3 lines

fs/nfsd/nfssvc.c:nfsd_get_default_maxblksize() is probably a good starting point. Its caller, nfsd_create_serv(), calls svc_create_pooled() with the result that's calculated.

Quoted 12 lines

Hmm. If I've read this section of code correctly, it seems to me that on most modern NFS servers (using TCP as the transport) the default
and preferred blocksize negotiated with clients will almost always be
1MB - the maximum RPC payload. The nfsd_get_default_maxblksize() function
seems obsolete for modern 64-bit servers with at least 4G of RAM as it'll
always prefer this upper bound instead of any value calculated according to
available RAM.

Quoted 2 lines

Well, "obsolete" is an odd way to put it--the code is still expected to work on smaller machines.

Poor choice of words perhaps. I guess I'm just used to NFS servers being pretty hefty pieces of kit and 'small' workstations having a couple of GB of RAM too.

Quoted 2 lines

Arguments welcome about the defaults, thoodd ugh I wonder whether it would be better to be doing this sort of calculation in user space.

See below.

Quoted 12 lines
Quoted 8 lines

For what it's worth (not sure if I specified this) I'm running kernel 2.6.32.

Anyway, this file/function appears to set the default *max* blocksize. I haven't
read all the related code yet, but does the preferred block size derive
from this maximum too?

See

Quoted 2 lines
Quoted 2 lines

For finfo see fs/nfsd/nfs3proc.c:nfsd3_proc_fsinfo, which uses svc_max_payload().

Quoted 8 lines

I've just returned from nfsd3_proc_fsinfo() and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply stuffs the max_block_size value in both the maximum *and* preferred block sizes for both read and write. A 1MB block size for a preferred default is a little high! If a disk is reading at 33MB/s and we have just a single server running 64 knfsd and each READ call is requesting 1MB of data then all of a sudden we have an aggregate read speed of ~512k/s

I lost you here.

Quoted 4 lines

and
that is without network latencies. And of course we will probably have 100s of requests queued behind each knfsd waiting for these 512k reads to finish. All of a sudden our user experience is rather poor :(

Quoted 5 lines

Note the preferred size is not a minimum--the client isn't forced to do 1MB reads if it really only wants 1 page, for example, if that's what you mean.

(I haven't actually looked at how typical clients used rt/wtpref.)

For our client, the answer is:

rtpref == default rsize
wtpref == default wsize and default f_bsize

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
#8 lore raw
In reply to J. Bruce Fields

<snip>

Quoted 15 lines
Quoted 6 lines

I've just returned from nfsd3_proc_fsinfo() and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply
stuffs the max_block_size value in both the maximum *and* preferred

Quoted 8 lines

block sizes for both read and write. A 1MB block size for a preferred
default is a little high! If a disk is reading at 33MB/s and we have just
a single server running 64 knfsd and each READ call is requesting 1MB of
data then all of a sudden we have an aggregate read speed of ~512k/s

Quoted 1 line

I lost you here.

OK, so what we're seeing is the large majority of our nr. ~700 clients (all Linux 2.6.32 based NFS clients) issuing READ requests of 1MB in size.

After the initial MOUNT request has been granted an FSINFO call is made. The contents of the REPLY from the server (another Linux 2.6.32 server) include rtmax, rtpref, wtmax and wtpref all of which are set to 1MB. This 1MB appears to come from that code/explanation I described earlier - all values are basically getting set to whatever comes out of nfsd_get_default_max_blksize().

Quoted 10 lines
Quoted 5 lines

that is without network latencies. And of course we will probably have 100s of
requests queued behind each knfsd waiting for these 512k reads to finish. All of a
sudden our user experience is rather poor :(

Note the preferred size is not a minimum--the client isn't forced to do
1MB reads if it really only wants 1 page, for example, if that's what you mean.

If no r/wsize has been specified on the client mount then the negotiated values above will be used by the client for any read() by an application exceeding that maximum. That maximum (the default 1MB) is still quite large I reckon.

I'm not sure at which point the preferred or optimal block size will be used by the client - because they're set as the same on the server side, I can't tell which is being used ;)

Quoted 30 lines

(I haven't actually looked at how typical clients used rt/wtpref.)

--b.

Quoted 25 lines

Perhaps a better suggestion would be to at least expose the maximum and preferred
block sizes (for both read and write) via a sysctl key so an administrator can set
it to the underlying block sizes of the file system or physical device?

Perhaps the defaults should at least be a smaller multiple of the page size or somewhere
between that and the PDU of the network layer the service is bound too.

Just my tuppence - and my maths might be flawed ;)

Jim

Quoted 4 lines

I'm not sure what the history is behind that logic, though.

--b.

--
Jim Vanns
Senior Software Developer
Framestore

Signature · 3 lines
Jim Vanns
Senior Software Developer
Framestore
#9 lore raw
Quoted 12 lines · James Vanns

<snip>

Quoted 10 lines
Quoted 9 lines

I've just returned from nfsd3_proc_fsinfo() and found what I would consider an odd decision - perhaps nothing better was suggested at the time. It seems to me that in response to an FSINFO call the reply stuffs the max_block_size value in both the maximum *and* preferred block sizes for both read and write. A 1MB block size for a preferred default is a little high! If a disk is reading at 33MB/s and we have just a single server running 64 knfsd and each READ call is requesting 1MB of data then all of a sudden we have an aggregate read speed of ~512k/s

Quoted 5 lines
Quoted 1 line

I lost you here.

OK, so what we're seeing is the large majority of our nr. ~700 clients (all Linux 2.6.32 based NFS clients) issuing READ requests of 1MB in size.

Knowing nothing about your situation, I'd assume the clients are doing that because they actually want that 1MB of data.

Would you prefer they each send 1024 1k READs? I don't understand why it's the read size you're focused on here.

--b.

Quoted 7 lines

After the initial MOUNT request has been granted an FSINFO call is made. The contents of the REPLY from the server (another Linux 2.6.32 server) include rtmax, rtpref, wtmax and wtpref all of which are set to 1MB. This 1MB appears to come from that code/explanation I described earlier - all values are basically getting set to whatever comes out of nfsd_get_default_max_blksize().

#10 lore raw
Quoted 2 lines

Knowing nothing about your situation, I'd assume the clients are doing that because they actually want that 1MB of data.

Possibly. But we have no control over that (the application read size, I mean).

Quoted 2 lines

Would you prefer they each send 1024 1k READs?  I don't understand why it's the read size you're focused on here.

No. But 32x 32k reads is reasonable (because it gives other RPCs a look-in). I'm focused on reads because it makes up the majority of our NFS traffic. I'm concerned because as it stands (out of the box) if the majority of our n knfsd threads are waiting for a 1MB read to return then no other RPC request will be serviced and will just contribute to the backlog. This backlog itself will probably also contain a hefty no. of 1MB read requests too. In short, a lot of other RPC calls that are not reads will just be blocking and this will appear to an end user as poor performance.

We deal with a great number of fairly large files - 10s of GBs in size. We just don't want others to suffer because of large request sizes coming in (writes end up being of the same size too but there are less of them). Our use cases are varied but they all have to share the same resource (the array of NFS servers).

We've only really seen this since our upgrade to SL6/kernel 2.6.32. I guess previously that 32k was some sort of default or limit?

Related to this was my query on when/how the (Linux) client may honour the preferred or optimal block size given in the FSINFO reply. Any ideas? Is it if a read of less than that preferred block size is requested then the preferred is used anyway because it comes at the same cost?

Thanks,

Jim

Quoted 20 lines

--b.

Quoted 10 lines

After the initial MOUNT request has been granted an FSINFO call is made. The contents of the REPLY from the server (another Linux 2.6.32
server) include rtmax, rtpref, wtmax and wtpref all of which are set
to 1MB. This 1MB appears to come from that code/explanation I described earlier  - all values are basically getting set to whatever
comes out of nfsd_get_default_max_blksize().

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Jim Vanns
Senior Software Developer
Framestore
#11 lore raw
Quoted 11 lines · James Vanns
Quoted 2 lines

Knowing nothing about your situation, I'd assume the clients are doing that because they actually want that 1MB of data.

Possibly. But we have no control over that (the application read size, I mean).

Quoted 2 lines

Would you prefer they each send 1024 1k READs?  I don't understand why it's the read size you're focused on here.

No. But 32x 32k reads is reasonable (because it gives other RPCs a look-in).

Maybe. In any case I'd want to see data before changing our defaults.

Quoted 8 lines

I'm focused on reads because it makes up the majority of
our NFS traffic. I'm concerned because as it stands (out of the box) if the majority of our n knfsd threads are waiting for a 1MB read to return then no other RPC request will be serviced and will just contribute to the backlog. This backlog itself will probably also contain a hefty no. of 1MB read requests too. In short, a lot of other RPC calls that are not reads will just be blocking and this will appear to an end user as poor performance.

Do you have a performance problem that you've actually measured, and if so could you share the details?

Quoted 6 lines

We deal with a great number of fairly large files - 10s of GBs in size. We just don't want others to suffer because of large request sizes coming in (writes end up being of the same size too but there are less of them). Our use cases are varied but they all have to share the same resource (the array of NFS servers).

Quoted 8 lines

We've only really seen this since our upgrade to SL6/kernel 2.6.32. I guess previously that 32k was some sort of default or limit?

Related to this was my query on when/how the (Linux) client may honour the preferred or optimal block size given in the FSINFO reply. Any ideas? Is it if a read of less than that preferred block size is requested then the preferred is used anyway because it comes at the same cost?

I'm not terribly familiar with the client logic, but would expect this to vary depending on kernel version, read-ahead policy, application behavior and a number of other factors, so I'd recommend testing with your workload and finding out.

--b.