Different NFSSVC_MAXBLKSIZE for udp and tcp clients?
Subsystems (2)
Is it possible to force a different maximum [rw]size on the server depending on whether the client uses udp or tcp?
Currently, when a client NFS mounts a file system without specifying a [rw]size, it defaults to whatever NFSSVC_MAXBLKSIZE was defined as (at compile time) on the server. However, this is the same for udp and tcp clients.
What I would like is to have a smaller default [rw]size for udp clients, but allow tcp clients to use 32k.
Is this possible?
Thanks
James Pearson
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 3 lines · James Pearson
Is it possible to force a different maximum [rw]size on the server depending on whether the client uses udp or tcp?
If someone writes the code, e.g. some new export options?
Quoted 4 lines
Currently, when a client NFS mounts a file system without specifying a [rw]size, it defaults to whatever
NFSSVC_MAXBLKSIZEwas defined as (at compile time) on the server. However, this is the same for udp and tcp clients.
Also NFSSVC_MAXBLKSIZE acts as a limit even when [wr]size is specifically set in mount options.
Quoted 2 lines
What I would like is to have a smaller default [rw]size for udp clients, but allow tcp clients to use 32k.
What I would like is to allow UDP clients to use up to 48K and TCP clients to use up to 4M. And on 2.4 kernels, ensure that MAXBLKSIZE is at least PAGE_CACHE_SIZE.
Greg.
Signature · 12 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Greg Banks <[email protected]> writes:
> What I would like is to allow UDP clients to use up to 48K and > TCP clients to use up to 4M.
Those are already on the roadmap, but haven't yet made it to the top of the priority list...
They will never make it into 2.4.x though. The main reason is the xdr_kmap() stuff, which doesn't really scale too well.
Cheers,
Trond
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Trond Myklebust
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
" " == Greg Banks <[email protected]> writes:
> What I would like is to allow UDP clients to use up to 48K and > TCP clients to use up to 4M.
Those are already on the roadmap, but haven't yet made it to the top of the priority list...
Are there preliminary patches or analysis/random thoughts somewhere online?
Quoted 2 lines
They will never make it into 2.4.x though. The main reason is the
xdr_kmap()stuff, which doesn't really scale too well.
Suits me.
Greg.
Signature · 12 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Greg Banks <[email protected]> writes:
Those are already on the roadmap, but haven't yet made it to the top of the priority list...
> Are there preliminary patches or analysis/random thoughts > somewhere online?
Most of the work should already have been done now that we've eliminated MAX_IOVEC from the RPC code (those tests in nfs_xdr.h are bogus and should be removed).
The remaining job is to make the 'pagevec' fields in nfs_read_data and nfs_write_data support arbitrary size arrays.
Cheers,
Trond
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Greg Banks
Quoted 3 lines · James Pearson
Is it possible to force a different maximum [rw]size on the server depending on whether the client uses udp or tcp?
If someone writes the code, e.g. some new export options?
Do you know if this would be possible with 2.4 kernels?
If it is possible, and someone could give me some pointers, I won't mind giving it a a go ...
James Pearson
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · On Wednesday November 26, [email protected] wrote:
Quoted 6 lines · Greg Banks
Quoted 3 lines · James Pearson
Is it possible to force a different maximum [rw]size on the server depending on whether the client uses udp or tcp?
Quoted 7 lines
Quoted 1 line
If someone writes the code, e.g. some new export options?
Do you know if this would be possible with 2.4 kernels?
If it is possible, and someone could give me some pointers, I won't mind giving it a a go ...
I have yet to see the point, so presumably I am missing something....
request size is completely under the control of the client via rsize and wsize (except that you cannot exceed the server's maximum). So why do you want to add an option to the server? Why not just use the already-existing-and-working option on the client?
NeilBrown
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 3 lines · Neil Brown
request size is completely under the control of the client via rsize and wsize (except that you cannot exceed the server's maximum).
The point is that the server's maximum is too low, at least when the client and server are powerful machines with a clean network between them (i.e. a significant part of SGI's customers).
When I say too low, I have two separate issues:
1. wsize<16K means an Altix client does synchronous writes, which results in a catastrophic performance hit (writes proceed at no more than 0.7% of line rate on gige). This is a hot issue right now.
2. Experience with IRIX shows that (up to a point) better performance can be achieved for some loads with larger transfer sizes. On IRIX that point is beyond Linux' current hardcoded maximums. This is more of a long-term issue.
Greg.
Signature · 12 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · On Thursday November 27, [email protected] wrote:
Quoted 3 lines · Neil Brown
request size is completely under the control of the client via rsize and wsize (except that you cannot exceed the server's maximum).
The point is that the server's maximum is too low, at least when the client and server are powerful machines with a clean network between them (i.e. a significant part of SGI's customers).
Why didn't you say so then (or did you and I missed it?).
The code in 2.4 is not ready to cope well with larger NFS packets. It requires any nfs message to be linear in memory, so that means that a large (multi-page) kmalloc has to succeed, and there is no gurantee that it will.
For udp, any incoming packet that is not already 'linear' is 'linearize'd (in svc_udp_recvfrom). For a 32k write, that means a kmalloc(32768) has to succeed, and very often free memory is too fragmented for it to succeed.
For tcp, the request will be copied into a pre-allocated buffer. The (large) buffer is allocated when the nfsd thread is started, and I have several times found that I couldn't start as many nfsd threads as I wanted because the system could not do the order-3 allocations that were needed. (this was when I was playing with large block sizes).
2.6 copes with page-lists and so can handle requests and replies that are not cntiguous in memory, so this is not an issue and MAXBLKSIZE can safely be increased.
I suggest that you simply recompile the kernel with a larger NFSSVC_MAXBLKSIZE and see how well it works.
I can see no reasonable for having different values for udp and tcp though. Just make it uniformly 32768 and see how it goes.
NeilBrown
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Neil Brown
Quoted 3 lines · On Thursday November 27, [email protected] wrote:
Neil Brown wrote:
The point is that the server's maximum is too low, [...]
Why didn't you say so then (or did you and I missed it?).
Sorry, just being obscure ;-)
Quoted 7 lines
The code in 2.4 is not ready to cope well with larger NFS packets. [...] For a 32k write, that means a
kmalloc(32768) has to succeed, and very often free memory is too fragmented for it to succeed.For tcp, the request will be copied into a pre-allocated buffer. The (large) buffer is allocated when the nfsd thread is started,[...]
Ok, so if physical memory fragmentation is the only issue I'll just try it. With 16K pages the allocation orders will be more reasonable.
Quoted 3 lines
2.6 copes with page-lists and so can handle requests and replies that are not cntiguous in memory, so this is not an issue and MAXBLKSIZE can safely be increased.
Excellent, this sounds like it deals with my long-term issue.
Quoted 4 lines
I suggest that you simply recompile the kernel with a larger
NFSSVC_MAXBLKSIZEand see how well it works.
I can see no reasonable for having different values for udp and tcp though. Just make it uniformly 32768 and see how it goes.
How about I conditionally define NFSSVC_MAXBLKSIZE to 32k if the page size is sufficently large? Would such a patch make it into 2.4?
Greg.
Signature · 12 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs