slowness due to splitting into pages in nfs3svc_decode_writeargs()
Subsystems (2)
Hi,
I'm presently investigating why writing to a nfs exported lustre filesystem is rather slow. Reading from lustre over nfs about 200-300 MB/s, but writing to it over nfs is only 20-50MB/s (both with IPoIB). Writing directly to this lustre cluster is about 600-700 MB/s both reading and writing. Well, 200-300 MB/s over NFS per client would be acceptable.
After several dozens of printks, systemtaps, etc I think its not the fault of lustre, but a generic nfsd and/or vfs problem.
In nfs3svc_decode_writeargs() all the data received are splitted into PAGE_SIZE, except the very first page. This page only gets PAGE_SIZE - header_length. So far no problem, but now on writing the pages in generic_file_buffered_write(), this function tries to write PAGE_SIZE. So it takes the first nfs page, which is PAGE_SIZE - header_length.
To fill up to PAGE_SIZE it will take header_length from the second page. Of course, now there's also only PAGE_SIZE - header_length for the 2nd nfs page left.
It will continue this way until the last page is written. Don't know why this doesn't show a big effect on other file system. Well, maybe it does, but nobody did notice it before?
Well, I have no idea if generic_file_buffered_write() really has to do what it presently does. But lets first stay at nfs, is it really necessary to already split up the data into pages?
Using this patch I get write speed of about 200 MB/s, even with kernel debugging enabled and several left-over printks
-- nfs3xdr.c.bak 2007-07-09 01:32:17.000000000 +0200
+++ nfs3xdr.c 2007-08-31 19:29:31.000000000 +0200
@@ -405,16 +405,8 @@ nfs3svc_decode_writeargs(struct svc_rqst
len = args->len = max_blocksize;
}
rqstp->rq_vec[0].iov_base = (void*)p;
- rqstp->rq_vec[0].iov_len = rqstp->rq_arg.head[0].iov_len - hdr;
- v = 0;
- while (len > rqstp->rq_vec[v].iov_len) {
- len -= rqstp->rq_vec[v].iov_len;
- v++;
- rqstp->rq_vec[v].iov_base = page_address(rqstp->rq_pages[v]);
- rqstp->rq_vec[v].iov_len = PAGE_SIZE;
- }
- rqstp->rq_vec[v].iov_len = len;
- args->vlen = v + 1;
+ rqstp->rq_vec[0].iov_len = len;
+ args->vlen = 1;
return 1;
}
Cheers,
Bernd
Signature · 11 lines
Bernd Schubert Q-Leap Networks GmbH ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Bernd Schubert
I'm presently investigating why writing to a nfs exported lustre filesystem is rather slow. Reading from lustre over nfs about 200-300 MB/s, but writing to it over nfs is only 20-50MB/s (both with IPoIB). Writing directly to this lustre cluster is about 600-700 MB/s both reading and writing. Well, 200-300 MB/s over NFS per client would be acceptable.
After several dozens of printks, systemtaps, etc I think its not the fault of lustre, but a generic nfsd and/or vfs problem.
Thanks for looking into this!
Quoted 11 lines
In
nfs3svc_decode_writeargs()all the data received are splitted intoPAGE_SIZE, except the very first page. This page only getsPAGE_SIZE-header_length. So far no problem, but now on writing the pages ingeneric_file_buffered_write(), this function tries to writePAGE_SIZE. So it takes the first nfs page, which isPAGE_SIZE-header_length.
To fill up toPAGE_SIZEit will takeheader_lengthfrom the second page. Of course, now there's also onlyPAGE_SIZE-header_lengthfor the 2nd nfs page left.
It will continue this way until the last page is written. Don't know why this doesn't show a big effect on other file system. Well, maybe it does, but nobody did notice it before?
Hm. Any chance this is the same problem?:
http://marc.info/?l=linux-nfs&m=112289652218095&w=2
Quoted 2 lines
Using this patch I get write speed of about 200 MB/s, even with kernel debugging enabled and several left-over printks
At too high a cost, unfortunately:
Quoted 2 lines
-- nfs3xdr.c.bak 2007-07-09 01:32:17.000000000 +0200
rqstp->rq_vec[0].iov_base = (void*)p;
...
Review
Quoted 2 lines
(quoted diff) +2
-0+ rqstp->rq_vec[0].iov_len = len; + args->vlen = 1;
There's no guarantee the later pages in the rq_pages array are contiguous in memory after the first one, so the rest of that iovec probably has random data in it.
(You might want to add to your tests some checks that the right data still gets to the file afterwards.)
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · J. Bruce Fields
Hm. Any chance this is the same problem?:
http://marc.info/?l=linux-nfs&m=112289652218095&w=2
Did this ever land anywhere?
b.
Signature · 3 lines
A day in the yard with my son is just like a day at work. He goes hunting around for stuff and brings it back to me and says: "Hey Dad, look what I found. The money is for me and the screw is for you."
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Brian J. Murrell
Quoted 4 lines · J. Bruce Fields
Hm. Any chance this is the same problem?:
http://marc.info/?l=linux-nfs&m=112289652218095&w=2Did this ever land anywhere?
No--I think there were some discussion of problems in the followup posts.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Re: slowness due to splitting into pages in nfs3svc_decode_writeargs ()
Quoted 10 lines · J. Bruce Fields
Quoted 6 lines · Brian J. Murrell
Quoted 3 lines · J. Bruce Fields
Hm. Any chance this is the same problem?:
http://marc.info/?l=linux-nfs&m=112289652218095&w=2Did this ever land anywhere?
No--I think there were some discussion of problems in the followup posts.
To sum up this discussion:
There are two choices to move the data:
1.) To page 2 ... n - this will overwrite nfsv4 at the end of page n.
2.) To page 1 ... n - 1 - this will overwrite the header, thus, all pointers to that memory will point to wrong data now.
Seems both aproaches are troublesome and nobody bothered to implement it. Not that I much like the idea of data moving at all, but what about
3.) On allocating the pages, allocate one page more than required. After filling in page 1, skip page 2 and proceed with page 3.
Now we would have space to properly move the data later on, thus:
memcpy (page2, page1 + hdr, PAGE_SIZE - hdr_length) memcpy (page2 + PAGE_SIZE - hdr_length, page3, hdr_length)
memmove(page3, page3 + hdr_length, PAGE_SIZE - hdr_length) [...]
Can you point me to the function assigning the data-block from the network to the page-vector?
Thanks,
Bernd
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Re: slowness due to splitting into pages in nfs3svc_decode_writeargs ()
In reply to J. Bruce FieldsHello Bruce,
thanks for your help!
Quoted 8 lines · J. Bruce Fields
Quoted 7 lines · Bernd Schubert
I'm presently investigating why writing to a nfs exported lustre filesystem is rather slow. Reading from lustre over nfs about 200-300 MB/s, but writing to it over nfs is only 20-50MB/s (both with IPoIB). Writing directly to this lustre cluster is about 600-700 MB/s both reading and writing. Well, 200-300 MB/s over NFS per client would be acceptable.
Quoted 4 lines
Quoted 2 lines
After several dozens of printks, systemtaps, etc I think its not the fault of lustre, but a generic nfsd and/or vfs problem.
Thanks for looking into this!
I will give these thanks to my boss who is paying me for this work :)
Quoted 14 lines
Quoted 13 lines
In
nfs3svc_decode_writeargs()all the data received are splitted intoPAGE_SIZE, except the very first page. This page only getsPAGE_SIZE-header_length. So far no problem, but now on writing the pages ingeneric_file_buffered_write(), this function tries to writePAGE_SIZE. So it takes the first nfs page, which isPAGE_SIZE-header_length.
To fill up toPAGE_SIZEit will takeheader_lengthfrom the second page. Of course, now there's also onlyPAGE_SIZE-header_lengthfor the 2nd nfs page left.
It will continue this way until the last page is written. Don't know why this doesn't show a big effect on other file system. Well, maybe it does, but nobody did notice it before?
Quoted 3 lines
Hm. Any chance this is the same problem?:
http://marc.info/?l=linux-nfs&m=112289652218095&w=2
Looks similar.
+ if (vec[0].iov_len + vec[vlen-1].iov_len != PAGE_CACHE_SIZE)
+ return 0;
+ for (i = 1; i < vlen - 1; ++i) {
+ if (vec[i].iov_len != PAGE_CACHE_SIZE)
+ return 0;
+ }
I tried to say in my last mail:
vec[0].iov_len = PAGE_PAGE_SIZE - headerlength
vec[1 ... n - 1].iov_len = PAGE_PAGE_SIZE
vec[n].iov_len = headerlength
This looks like it needs quite some cpu cycles
+ memmove(this_page + chunk0, this_page, chunk1);
+ memcpy(this_page, prev_page + chunk1, chunk0);
I will test the patch tomorrow.
Quoted 8 lines
Quoted 3 lines
Using this patch I get write speed of about 200 MB/s, even with kernel debugging enabled and several left-over printks
At too high a cost, unfortunately:
Quoted 2 lines
-- nfs3xdr.c.bak 2007-07-09 01:32:17.000000000 +0200
rqstp->rq_vec[0].iov_base = (void*)p;
Quoted 8 lines
...
Review
Quoted 2 lines
(quoted diff) +2
-0+ rqstp->rq_vec[0].iov_len = len; + args->vlen = 1;There's no guarantee the later pages in the
rq_pagesarray are contiguous in memory after the first one, so the rest of that iovec probably has random data in it.
Hmm, its some time since I last read rfc1813, but I can't remember something like 'data are send in pages and pages may have random order'. So I guess some kind of multi-threading is filling in the data the client is sending? Given the performance impact this has, maybe single-threading per client request would be better?
Can you point me to the corresponding function?
Quoted 3 lines
(You might want to add to your tests some checks that the right data still gets to the file afterwards.)
Hmm, I need to put the data on a ram-disk. All raid-boxes sufficiently fast for this operation are in use for lustre storage.
Thanks again,
Bernd
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Re: slowness due to splitting into pages in?nfs3svc_decode_writeargs()
Quoted 8 lines · Bernd Schubert
Quoted 3 lines · J. Bruce Fields
There's no guarantee the later pages in the
rq_pagesarray are contiguous in memory after the first one, so the rest of that iovec probably has random data in it.Hmm, its some time since I last read rfc1813, but I can't remember something like 'data are send in pages and pages may have random order'. So I guess some kind of multi-threading is filling in the data the client is sending?
The data all arrives in one big chunk, in order. But then we have to put it some place. The kernel almost never tries to allocate more than one contiguous page of memory--memory fragmentation can make it difficult to do that reliably--so we just ask for a bunch of pages to put the data in, which may represent memory from all over the place, store those pages into an array, and receive the data into those pages in the order they're listed in the array.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Re: slowness due to splitting into pages in?nfs3svc_decode_writeargs()
Quoted 11 lines · J. Bruce Fields
Quoted 9 lines · Bernd Schubert
Quoted 3 lines · J. Bruce Fields
There's no guarantee the later pages in the
rq_pagesarray are contiguous in memory after the first one, so the rest of that iovec probably has random data in it.Hmm, its some time since I last read rfc1813, but I can't remember something like 'data are send in pages and pages may have random order'. So I guess some kind of multi-threading is filling in the data the client is sending?
Quoted 7 lines
The data all arrives in one big chunk, in order. But then we have to put it some place. The kernel almost never tries to allocate more than one contiguous page of memory--memory fragmentation can make it difficult to do that reliably--so we just ask for a bunch of pages to put the data in, which may represent memory from all over the place, store those pages into an array, and receive the data into those pages in the order they're listed in the array.
Ah, now I understand, thanks! I'm still used to userspace programming (*)
Thanks,
Bernd
PS: (*) Don't know if I ever will really like kernel programming - it remembers me to metal-organic chemistry, everything is a 100 times more difficult than usually, even simple weighing 100g of a substance might take a few hours.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Re: slowness due to splitting into pages in nfs3svc_decode_writeargs ()
In reply to J. Bruce FieldsQuoted 5 lines · J. Bruce Fields
Hm. Any chance this is the same problem?:
http://marc.info/?l=linux-nfs&m=112289652218095&w=2
I have slightly modified Olafs patch, now the last page is saved and restored. Well there's a problem with it, if there's more than one thread, files over nfs become corrupted, if there's only one thread everything is fine. So far I have no idea whats going on.
In principal we could do the same in do_readv_writev(), the iov vector there should hopefully only belong to one thread.
Anyway, using this patch, the nfs-write speed on lustre is about 200 MB/s.
Cheers,
Bernd
fs/nfsd/vfs.c +70 -0
@@ -900,6 +900,48 @@mutex_unlock (&dentry->d_inode->i_mutex); } +/* +* Helper function to page-align the write payload. +*/ +static int +nfsd_page_align_payload (struct kvec *vec,int vlen) +{ +unsigned char *this_page, *prev_page; +int i, chunk0, chunk1; + +/* The following checks are just paranoia */ +if (vlen <2 ) +return 0 ; + +if (vec[0 ].iov_len + vec[vlen-1 ].iov_len !=PAGE_CACHE_SIZE ) +return 0 ; +for (i =1 ; i < vlen -1 ; ++i) { +if (vec[i].iov_len !=PAGE_CACHE_SIZE ) +return 0 ; + } + + chunk0 = vec[0 ].iov_len; + chunk1 =PAGE_CACHE_SIZE - chunk0; + + this_page = (unsigned char *) vec[vlen-1 ].iov_base; +for (i = vlen-1 ; i; --i) { + prev_page = (unsigned char *) vec[i-1 ].iov_base; + +/* Push trailing partial page so it's +* aligned with the end of the page, then +* pull up the missing chunk from the previous +* page */ +memmove (this_page + chunk0, this_page, chunk1); +memcpy (this_page, prev_page + chunk1, chunk0); + vec[i].iov_len =PAGE_CACHE_SIZE ; + this_page = prev_page; + } + +return 1 ; +} + + + static __be32 nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, struct file *file, loff_t offset, struct kvec *vec, int vlen,
@@ -912,6 +954,8 @@__be32 err =0 ;int host_err;int stable = *stablep; +char *tmp_page;/* used for page alignement */ +int did_align =0 ;#ifdef MSNFS err = nfserr_perm;
@@ -944,6 +988,26 @@if (stable && !EX_WGATHER (exp)) file->f_flags |=O_SYNC ; +/* save the last vector element, besides file data it may have +* nfsv4 data, which will be overwritten by the alignement hack */ + tmp_page = (char *)__get_free_page (GFP_KERNEL ); +if (tmp_page + && (offset < inode->i_size) + && vec->iov_len !=PAGE_CACHE_SIZE ) { +memcpy (tmp_page, vec[vlen -1 ].iov_base,PAGE_SIZE ); + +/* Hack: if we're rewriting the file, make sure +* we align the iovec properly to avoid costly +* read-modify-write operations on the block devices. +* This hack can go away once we have generic_file_writev. +*/ + +if (nfsd_page_align_payload (vec, vlen)) { + did_align =1 ; + vec++, vlen--; + } + } +/* Write the data. */ oldfs =get_fs ();set_fs (KERNEL_DS ); host_err =vfs_writev (file, (struct iovec __user *)vec, vlen, &offset);
@@ -953,6 +1017,12 @@fsnotify_modify (file->f_path.dentry); } +/* restore the last vector element */ +if (did_align) { +memcpy (vec[vlen -1 ].iov_base, tmp_page,PAGE_SIZE ); +free_page ((unsigned long ) tmp_page); + } +/* clear setuid/setgid flag after write */ if (host_err >=0 && (inode->i_mode & (S_ISUID |S_ISGID )))kill_suid (dentry);
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs