Add write throttling to NFS client
Subsystems (2)
PATCH 1/1 Add write throttling to NFS client
Hi,
I posted an earlier version of the attached patch to the kernel mailing list a few days back but did not receive any feedback. Hopefully, I'll have better luck here. ;-)
The stock NFS client does not regulate the # async write requests causing other accesses to block in the presence of streaming writes. This patch adds such support. For instance, a single dd running in the background writing to a file in my home directory causes my X session to hang until dd exits. With this patch the session does not experience such hangs. Please test it out and let me know if you see anything problems. I'd like to see this integrated soon.
Thanks,
Shantanu
__________________________________
Do you Yahoo!?
Yahoo! Mail SpamGuard - Read only the mail you want.
http://antispam.yahoo.com/tools
include/linux/nfs_page.h +2 -1
@@ -53,7 +53,8 @@extern void nfs_list_add_request (struct nfs_page *,struct list_head *);extern int nfs_scan_list (struct list_head *,struct list_head *, -struct file *,unsigned long ,unsigned int ); +struct file *,unsigned long ,unsigned int , +unsigned int ,unsigned int );extern int nfs_coalesce_requests (struct list_head *,struct list_head *,unsigned int );extern int nfs_wait_on_request (struct nfs_page *);
include/linux/nfs_fs_sb.h +1 -0
@@ -28,6 +28,7 @@char * hostname;/* remote hostname */ struct nfs_fh fh;struct sockaddr_in addr; +atomic_t wactive;/* # active write requests */ #ifdef CONFIG_NFS_V4 /* Our own IP address, as a null-terminated string. * This is used to generate the clientid, and the callback address.
fs/nfs/inode.c +2 -0
@@ -364,6 +364,8 @@if (sb->s_maxbytes >MAX_LFS_FILESIZE ) sb->s_maxbytes =MAX_LFS_FILESIZE ; +atomic_set (&server->wactive,0 ); +/* We're airborne Set socket buffersize */ rpc_setbufsize (server->client, server->wsize +100 , server->rsize +100 );return 0 ;
fs/nfs/pagelist.c +28 -3
@@ -249,6 +249,8 @@* @file: if set, ensure we match requests from this file * @idx_start: lower bound of page->index to scan * @npages: idx_start + npages sets the upper bound to scan. +* @nreq: if set, stop after this many coalesced requests. +* @rpages: if nreq is set, # pages per request. * * Moves elements from one of the inode request lists. * If the number of requests is set to 0, the entire address_space
@@ -259,18 +261,22 @@int nfs_scan_list (struct list_head *head,struct list_head *dst,struct file *file, -unsigned long idx_start,unsigned int npages) +unsigned long idx_start,unsigned int npages, +unsigned int nreq,unsigned int rpages) {struct list_head *pos, *tmp; -struct nfs_page *req; +struct nfs_page *req, *prev;unsigned long idx_end; -int res; +int res, is_contig; +unsigned int pages, nr; res =0 ;if (npages ==0 ) idx_end = ~0 ;else idx_end = idx_start + npages -1 ; + nr = pages =0 ; + prev =NULL ;list_for_each_safe (pos, tmp, head) {
@@ -284,11 +290,30 @@if (req->wb_index > idx_end)break ; + is_contig = (nreq && + prev && + pages < rpages && + req->wb_pgbase ==0 && + prev->wb_pgbase + prev->wb_bytes ==PAGE_CACHE_SIZE && + req->wb_index == prev->wb_index +1 && + req->wb_cred == prev->wb_cred); + +if (nreq && !is_contig && nr == nreq) +break ; +if (!nfs_lock_request (req))continue ;nfs_list_remove_request (req);nfs_list_add_request (req, dst); res++; + +if (is_contig) + pages++; +else { + pages =1 ; + nr++; + } + prev = req; }return res; }
fs/nfs/write.c +108 -16
229 lines not on this page · raw message
Quoted 4 lines · Shantanu Goel
The stock NFS client does not regulate the # async write requests causing other accesses to block in the presence of streaming writes.
This is a real problem; the same problem in IRIX was fixed at 6.5.17 (about 2 years ago).
Quoted 7 lines
This patch adds such
support. For instance, a single dd running in the background writing to a file in my home directory causes my X session to hang until dd exits. With this patch the session does not experience such hangs. Please test it out and let me know if you see anything problems. I'd like to see this integrated soon.
I discussed this with Dave Chinner (who implemented IRIX' write throttling), and our conclusion was that this is implementing throttling at the wrong level. A better approach would be to change the allocation algorithm of rpc_rqsts to divide the RPC_MAXREQS space into three fixed spaces by class: sync calls, async (writes & commits), and all other async calls. Attempts to allocate a request slot can then be classified into one of these classes in the struct rpc_procinfo, and when a class' space is full the allocation fails; the rpc_task already knows how to sleep and retry request slot allocation.
Alternately, the allocation could treat the classes as priority classes with sync calls given the highest priority; allocations would succeed only if there were slots available at the same or less important priority. For example, there were patches to do similar things to the block device layer.
The exact numbers chosen for the class sizes will be a subtle and important tuning decision. It will probably pay to bump up RPC_MAXREQS to compensate for the subdivision; you'll want the class for async writes to still be 16 slots in size.
Greg.
Signature · 12 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------- SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now! http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
While the solution you propose certainly sounds more elegant, in my experience, asynchronous writes seem to eat up most of the RPC slots in normal use and this patch only modifies the write path without affecting the core RPC scheduler. Surely, a solution that fixes the common case should be acceptable until the more elegant one is available? The NFS client in Linux has suffered from this issue for quite a while and I don't recall seeing a patch that addressed this issue. If there is something available to test, I'd certainly like to look at it. As such, the current NFS client is basically useless for concurrent activity in the presence of heavy writers.
Shantanu
Quoted 59 lines · --- Greg Banks
Quoted 3 lines · Shantanu Goel
The stock NFS client does not regulate the # async write requests causing other accesses to block in
the
Quoted 1 line
presence of streaming writes.
This is a real problem; the same problem in IRIX was fixed at
6.5.17 (about 2 years ago).Quoted 4 lines
This patch adds such
support. For instance, a single dd running in the background writing to a file in my home directory causes my X session to hang until dd exits. Withthis
Quoted 2 lines
patch the session does not experience such hangs. Please test it out and let me know if you see
anything
Quoted 1 line
problems. I'd like to see this integrated soon.
I discussed this with Dave Chinner (who implemented IRIX' write
throttling), and our conclusion was that this is implementing
throttling at the wrong level. A better approach would be to
change the allocation algorithm ofrpc_rqststo divide theRPC_MAXREQSspace into three fixed spaces by class:
sync calls,
async (writes & commits), and all other async calls.
Attempts to
allocate a request slot can then be classified into one of
these classes in the structrpc_procinfo, and when a class'
space is full the allocation fails; therpc_taskalready knows
how to sleep and retry request slot allocation.Alternately, the allocation could treat the classes as priority
classes with sync calls given the highest priority; allocations
would succeed only if there were slots available at the same or
less important priority. For example, there were patches to do
similar things to the block device layer.The exact numbers chosen for the class sizes will be a subtle
and important tuning decision. It will probably pay to bump upRPC_MAXREQSto compensate for the subdivision; you'll want the
class for async writes to still be 16 slots in size.
Quoted 5 lines
Greg.
Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI.
__________________________________
Do you Yahoo!?
Get better spam protection with Yahoo! Mail.
http://antispam.yahoo.com/tools
-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 3 lines · Greg Banks
change the allocation algorithm of
rpc_rqststo divide theRPC_MAXREQSspace into three fixed spaces by class: sync calls, async (writes & commits), and all other async calls. Attempts to
I don't think this is a good solution for UDP. If you bump the number of slots at the same time, you may still end up with the same symptoms, as your async write requests may eat your congestion window, leaving just the crumbs for sync requests. If you don't bump the number of slots, write throughput will suffer.
What about modifying __rpc_sleep_on to put sync tasks always at the head of the queue? This way sync tasks get priority when it comes to call reservation.
(for fairness reasons, you may want to walk the queue and insert your sync task before the first async task at least)
Olaf
Signature · 13 lines
Olaf Kirch | Stop wasting entropy - start using predictable [email protected] | tempfile names today! ---------------+ ------------------------------------------------------- SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now! http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
På to , 26/02/2004 klokka 00:41, skreiv Olaf Kirch:
Quoted 3 lines
What about modifying
__rpc_sleep_onto put sync tasks always at the head of the queue? This way sync tasks get priority when it comes to call reservation.
How do you avoid the inverse problem that async tasks end up never making any progress? I'm not a huge fan of playing with rpc task
priorities: just witness the type of tuning problems Ingo & co. had (and still have IMO) when they played around with the task scheduler algorithm for 2.6.x.
I think I prefer an approach like that taken by Shantanu, however there are 2 problems with his code as it stands today:
- It only addresses the problem of writes (yes, we sometimes do read a
bit too).
- It is a fairly heavy layering violation. It would be nice to move
some (all?) of that code into the RPC layer if possible.
Cheers,
Trond
-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 6 lines · On Thu, 26 Feb 2004 [email protected] wrote:
P=E5 to , 26/02/2004 klokka 00:41, skreiv Olaf Kirch:
Quoted 3 lines
What about modifying
__rpc_sleep_onto put sync tasks always at the head of the queue? This way sync tasks get priority when it comes to call reservation.=20
How do you avoid the inverse problem that async tasks end up never maki=
ng
Quoted 1 line
any progress?
How about having a ratio (that can be changed from userland) between sync and async operations ? Let's say that you move 3 sync ops at the head of the queue then let 1 async one go through. However, for this to be done efficiently it's probably better to not have only one queue but more and then take one or more ops from the head of each queue corresponding to the ratio.
--=20
Bogdan Costescu
IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]
-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
På to , 26/02/2004 klokka 05:20, skreiv Bogdan Costescu:
Quoted 4 lines
Quoted 2 lines
How do you avoid the inverse problem that async tasks end up never making any progress?
How about having a ratio (that can be changed from userland) between
sync and async
Quoted 1 line
operations ? Let's say that you move 3 sync ops at the head of the queue
then let 1
Quoted 1 line
async one go through. However, for this to be done efficiently it's
probably better to not
Quoted 1 line
have only one queue but more and then take one or more ops from the head
of each
Quoted 1 line
queue corresponding to the ratio.
I'm not convinced that is flexible enough. Imagine that the machine needs to flush out some writes in order to reclaim memory for use by other tasks. Is it then correct to be letting through a bunch of stat() requests in order to satisfy some "ls" command instead of the memory reclaim?
To simply assert that "async" is somehow equivalent to "not important" is just plain wrong.
Cheers,
Trond
-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 2 lines · Trond Myklebust
To simply assert that "async" is somehow equivalent to "not important" is just plain wrong.
Agreed, and I also agree that my suggestion of always preferring sync tasks over async tasks is simplistic. But I think the patch proposed by Shantanu adds more complexity to an already overly complex rpc implementation.
The general "unfairness" observed in writes is due to the fact that we allow a writing process to dirty a large number of pages without blocking for the actual IO. So if you write a large file, nfs_flushd can easily saturate the transport (which is a good thing for performance) but it's bad for sync tasks.
One problem is that currently, there seems to be no upper bound at all on the number of write requests we schedule. So one step in the right direction may be to check the number of backlogged tasks and refrain from scheduling more write requests if it exceeds a certain threshold.
Applications doing lots of stats etc are punished so severely because all these operations are synchronous. So if you have a maximum backlog of N RPC requests, your average delay per sync rpc operation is O(N).
So I think it could help if nfs_flushd and friends check the transport backlog before scheduling new writes. There should be tunables for the minimum number of pending writes we're always permitted to schedule, and a maximum backlog length.
To avoid any ugliness related to layering violations, you could use "color" to tag different RPC requests rather than referring to "writes".
Of course, that's just changing the queuing, not real scheduling. But that can be an advantage, too, because it keeps things relatively simple.
BTW I think reading is much less of a problem. The number of readaheads scheduled by generic_file_read is limited, so it's entirely different from the "lets blow the router's fuse" approach of the NFS write code.
Olaf
Signature · 13 lines
Olaf Kirch | Stop wasting entropy - start using predictable [email protected] | tempfile names today! ---------------+ ------------------------------------------------------- SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now! http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 3 lines · Olaf Kirch
The general "unfairness" observed in writes is due to the fact that we allow a writing process to dirty a large number of pages without blocking for the actual IO.
Yes, and I think that this is pretty similar to the situation encountered with block IO, where queued writes had to be serviced before later queued reads, giving an impression of unresponsivness. It's easier to accept a delayed write than a delayed read.
Quoted 2 lines
Applications doing lots of stats etc are punished so severely because all these operations are synchronous.
That's exactly the case for block IO reads.
Quoted 2 lines
BTW I think reading is much less of a problem. The number of readaheads scheduled by
generic_file_readis limited,
On the other hand, based on the block IO experience, reads should have higher priority than writes :-)
Signature · 17 lines
Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: [email protected] ------------------------------------------------------- SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now! http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 2 lines · On Thu, 26 Feb 2004 [email protected] wrote:
To simply assert that "async" is somehow equivalent to "not important" is just plain wrong.
That was the point of the userland modifiable ratio. Then userland can decide what is more important...
Signature · 17 lines
Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: [email protected] ------------------------------------------------------- SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now! http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs