Regarding client fairness
Subsystems (2)
I came across a rather annoying issue where a single NFS client caused resource starvation for NFS server. The server has several storage pools which are used, in this particular case a single client did fairly large read requests and effectively ate all nfsd threads on the server and during that other clients were getting hardly any I/O through to the other storage pool which was completely idle.
I then proceeded to make a simple testcase and noticed that reading a file with large blocksize causes NFS server to read using multiple threads, effectively consuming all nfsd threads on the server and causing starvation to other clients regardless of the share/backing disk they were accessing.
In my testcase a simple (ridiculous) dd was able to effectively reserve the entire NFS server for itself:
# dd if=fgsfds bs=1000M count=10000 iflag=direct
Also several similar dd runs with blocksize of 100M caused the same effect. During those dd-runs the server was responding at a very slow rate to any other requests by other clients (or to other NFS shares on different disks on the server).
My question here is that are there any methods to ensure client fairness with Linux NFS and/or are there some best common practices to ensure something like that. I think it would be pretty awesome if clients had some kind of limit/fairness that would be scoped like {client, share-on-server} so client which accesses a single share on a server (with large read IO requests) would not effectively cause denial of service for the entire NFS server but rather only to the share it is accessing and at same time other clients accessing different/same share would get fair amount of access to the data.
Quoted 7 lines · On Wed, Mar 28, 2018 at 02:04:57PM +0300, [email protected] wrote:
I came across a rather annoying issue where a single NFS client caused resource starvation for NFS server. The server has several storage pools which are used, in this particular case a single client did fairly large read requests and effectively ate all nfsd threads on the server and during that other clients were getting hardly any I/O through to the other storage pool which was completely idle.
What version of the kernel are you running on your server?
--b.
Quoted 30 lines
I then proceeded to make a simple testcase and noticed that reading a file with large blocksize causes NFS server to read using multiple threads, effectively consuming all nfsd threads on the server and causing starvation to other clients regardless of the share/backing disk they were accessing.
In my testcase a simple (ridiculous) dd was able to effectively reserve the entire NFS server for itself:
# dd if=fgsfds bs=1000M count=10000 iflag=directAlso several similar dd runs with blocksize of 100M caused the same effect. During those dd-runs the server was responding at a very slow rate to any other requests by other clients (or to other NFS shares on different disks on the server).
My question here is that are there any methods to ensure client fairness with Linux NFS and/or are there some best common practices to ensure something like that. I think it would be pretty awesome if clients had some kind of limit/fairness that would be scoped like {client, share-on-server} so client which accesses a single share on a server (with large read IO requests) would not effectively cause denial of service for the entire NFS server but rather only to the share it is accessing and at same time other clients accessing different/same share would get fair amount of access to the data. --
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · bfields
Quoted 7 lines · On Wed, Mar 28, 2018 at 02:04:57PM +0300, [email protected] wrote:
I came across a rather annoying issue where a single NFS client caused resource starvation for NFS server. The server has several storage pools which are used, in this particular case a single client did fairly large read requests and effectively ate all nfsd threads on the server and during that other clients were getting hardly any I/O through to the other storage pool which was completely idle.
What version of the kernel are you running on your server?
I'm thinking that if it includes upstream 637600f3ffbf "SUNRPC: Change TCP socket space reservation" (in upstream 4.8), then you may want to experiment setting the sunrpc.svc_rpc_per_connection_limit module parameter added in ff3ac5c3dc23 "SUNRPC: Add a server side per-connection limit".
You probably want to experiment with values greater than 0 (the default, no limit) and the number of server threads.
--b.
Quoted 33 lines
--b.
Quoted 30 lines
I then proceeded to make a simple testcase and noticed that reading a file with large blocksize causes NFS server to read using multiple threads, effectively consuming all nfsd threads on the server and causing starvation to other clients regardless of the share/backing disk they were accessing.
In my testcase a simple (ridiculous) dd was able to effectively reserve the entire NFS server for itself:
# dd if=fgsfds bs=1000M count=10000 iflag=directAlso several similar dd runs with blocksize of 100M caused the same effect. During those dd-runs the server was responding at a very slow rate to any other requests by other clients (or to other NFS shares on different disks on the server).
My question here is that are there any methods to ensure client fairness with Linux NFS and/or are there some best common practices to ensure something like that. I think it would be pretty awesome if clients had some kind of limit/fairness that would be scoped like {client, share-on-server} so client which accesses a single share on a server (with large read IO requests) would not effectively cause denial of service for the entire NFS server but rather only to the share it is accessing and at same time other clients accessing different/same share would get fair amount of access to the data. --
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · J. Bruce Fields
Quoted 9 lines · bfields
Quoted 7 lines · On Wed, Mar 28, 2018 at 02:04:57PM +0300, [email protected] wrote:
I came across a rather annoying issue where a single NFS client caused resource starvation for NFS server. The server has several storage pools which are used, in this particular case a single client did fairly large read requests and effectively ate all nfsd threads on the server and during that other clients were getting hardly any I/O through to the other storage pool which was completely idle.
What version of the kernel are you running on your server?
4.15.10 on the system I am testing on.
Quoted 8 lines
I'm thinking that if it includes upstream
637600f3ffbf"SUNRPC: Change TCP socket space reservation" (in upstream 4.8), then you may want to experiment setting the sunrpc.svc_rpc_per_connection_limit module parameter added inff3ac5c3dc23"SUNRPC: Add a server side per-connection limit".You probably want to experiment with values greater than 0 (the default, no limit) and the number of server threads.
That helps for the client slowing down the whole server, thanks for the tip! Of course this doesn't help with the case of client accessing 2 different shares on the same server but that is something I can work around.
Quoted 34 lines
--b.
Quoted 31 lines
--b.
Quoted 29 lines
I then proceeded to make a simple testcase and noticed that reading a file with large blocksize causes NFS server to read using multiple threads, effectively consuming all nfsd threads on the server and causing starvation to other clients regardless of the share/backing disk they were accessing.
In my testcase a simple (ridiculous) dd was able to effectively reserve the entire NFS server for itself:
# dd if=fgsfds bs=1000M count=10000 iflag=directAlso several similar dd runs with blocksize of 100M caused the same effect. During those dd-runs the server was responding at a very slow rate to any other requests by other clients (or to other NFS shares on different disks on the server).
My question here is that are there any methods to ensure client fairness with Linux NFS and/or are there some best common practices to ensure something like that. I think it would be pretty awesome if clients had some kind of limit/fairness that would be scoped like {client, share-on-server} so client which accesses a single share on a server (with large read IO requests) would not effectively cause denial of service for the entire NFS server but rather only to the share it is accessing and at same time other clients accessing different/same share would get fair amount of access to the data. --
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 18 lines · Antti Tönkyrä
Quoted 10 lines · J. Bruce Fields
Quoted 9 lines · bfields
Quoted 7 lines · On Wed, Mar 28, 2018 at 02:04:57PM +0300, [email protected] wrote:
I came across a rather annoying issue where a single NFS client caused resource starvation for NFS server. The server has several storage pools which are used, in this particular case a single client did fairly large read requests and effectively ate all nfsd threads on the server and during that other clients were getting hardly any I/O through to the other storage pool which was completely idle.
What version of the kernel are you running on your server?
4.15.10 on the system I am testing on.
Quoted 6 lines
I'm thinking that if it includes upstream
637600f3ffbf"SUNRPC: Change TCP socket space reservation" (in upstream 4.8), then you may want to experiment setting the sunrpc.svc_rpc_per_connection_limit module parameter added inff3ac5c3dc23"SUNRPC: Add a server side per-connection limit".
Quoted 6 lines
Quoted 2 lines
You probably want to experiment with values greater than 0 (the default, no limit) and the number of server threads.
That helps for the client slowing down the whole server, thanks for the tip! Of course this doesn't help with the case of client accessing 2 different shares on the same server but that is something I can work around.
I thought the Linux client shared a single connection in that case, but I could be wrong.
--b.
Quoted 18 lines · Antti Tönkyrä
Quoted 10 lines · J. Bruce Fields
Quoted 9 lines · bfields
Quoted 7 lines · On Wed, Mar 28, 2018 at 02:04:57PM +0300, [email protected] wrote:
I came across a rather annoying issue where a single NFS client caused resource starvation for NFS server. The server has several storage pools which are used, in this particular case a single client did fairly large read requests and effectively ate all nfsd threads on the server and during that other clients were getting hardly any I/O through to the other storage pool which was completely idle.
What version of the kernel are you running on your server?
4.15.10 on the system I am testing on.
Quoted 6 lines
I'm thinking that if it includes upstream
637600f3ffbf"SUNRPC: Change TCP socket space reservation" (in upstream 4.8), then you may want to experiment setting the sunrpc.svc_rpc_per_connection_limit module parameter added inff3ac5c3dc23"SUNRPC: Add a server side per-connection limit".
Quoted 4 lines
Quoted 2 lines
You probably want to experiment with values greater than 0 (the default, no limit) and the number of server threads.
That helps for the client slowing down the whole server, thanks for the tip!
We should probably revisit 637600f3ffbf "SUNRPC: Change TCP socket space reservation". There's got to be some way to keep high bandwidth pipes filled with read data without introducing this problem where a single client can tie up every server thread.
Just out of curiosity, do you know (approximately) the network and disk bandwidth in this case?
--b.
Quoted 28 lines · J. Bruce Fields
Quoted 18 lines · Antti Tönkyrä
Quoted 10 lines · J. Bruce Fields
Quoted 9 lines · bfields
Quoted 7 lines · On Wed, Mar 28, 2018 at 02:04:57PM +0300, [email protected] wrote:
I came across a rather annoying issue where a single NFS client caused resource starvation for NFS server. The server has several storage pools which are used, in this particular case a single client did fairly large read requests and effectively ate all nfsd threads on the server and during that other clients were getting hardly any I/O through to the other storage pool which was completely idle.
What version of the kernel are you running on your server?
4.15.10 on the system I am testing on.
Quoted 6 lines
I'm thinking that if it includes upstream
637600f3ffbf"SUNRPC: Change TCP socket space reservation" (in upstream 4.8), then you may want to experiment setting the sunrpc.svc_rpc_per_connection_limit module parameter added inff3ac5c3dc23"SUNRPC: Add a server side per-connection limit".Quoted 4 lines
Quoted 2 lines
You probably want to experiment with values greater than 0 (the default, no limit) and the number of server threads.
That helps for the client slowing down the whole server, thanks for the tip!
We should probably revisit
637600f3ffbf"SUNRPC: Change TCP socket space reservation". There's got to be some way to keep high bandwidth pipes filled with read data without introducing this problem where a single client can tie up every server thread.
Quoted 4 lines
Just out of curiosity, do you know (approximately) the network and disk bandwidth in this case?
--b.
Locally I can read at about 150-200MB/s (two spindles in mirror). Additionally I have another mount which is a NVMe drive which performs >500MB/s which I used to verify that I was not bottlenecked when running the dd read test which tied up the server threads. Network bandwidth is 10Gbps.