The return of the hanging "ls"...
Subsystems (2)
Hi Trond,
I just noticed commit acdc53b2146c7ee67feb1f02f7bc3020126514b8 from 2010
reverts the effect commit 28c494c5c8d425e15b7b82571e4df6d6bc34594d from Chunk in 2007.
Specifically it removes mutex lock/unlock in nfs_getattr. Chuck added them:
- /* Flush out writes to the server in order to update c/mtime */
- if (S_ISREG(inode->i_mode))
+ /*
+ * Flush out writes to the server in order to update c/mtime.
+ *
+ * Hold the i_mutex to suspend application writes temporarily;
+ * this prevents long-running writing applications from blocking
+ * nfs_wb_nocommit.
+ */
+ if (S_ISREG(inode->i_mode)) {
+ mutex_lock(&inode->i_mutex);
nfs_wb_nocommit(inode);+ mutex_unlock(&inode->i_mutex);
+ }
You removed them.
- /*
- * Flush out writes to the server in order to update c/mtime.
- *
- * Hold the i_mutex to suspend application writes temporarily;
- * this prevents long-running writing applications from blocking
- * nfs_wb_nocommit.
- */
+ /* Flush out writes to the server in order to update c/mtime. */
if (S_ISREG(inode->i_mode)) {
- mutex_lock(&inode->i_mutex);
- nfs_wb_nocommit(inode);
- mutex_unlock(&inode->i_mutex);
+ err = filemap_write_and_wait(inode->i_mapping);
+ if (err)
+ goto out;
}
/*Do you recall why?
I noticed because a customer reported exactly the same symptoms the were fixed by Chucks patch some years ago.
The comment on your patch says (in part):
Also replace nfs_wb_nocommit() with a
call to filemap_write_and_wait(), which doesn't need to hold the inode->i_mutex.
It is certainly true that filemap_write_and_wait doesn't need to hold the mutex, but neither did nfs_wb_nocommit. The mutex is held to stop "suspend application writes temporarily" so no more pages get dirtied until all the current dirty pages have been written out. i.e. to stop generic_file_aio_write() from proceeding.
The particular test that shows the problem is a large write like
dd if=/dev/zero of=/mnt/nfs/somefile count=2000000
then in another window
ls -l /mnt/nfs
the "ls -l" will hang until the "dd" completes.
Can we put the mutex lock/unlock back please?
Thanks,
NeilBrown
Hi Neil-
Quoted 5 lines · NeilBrown
Hi Trond,
I just noticed commitacdc53b2146c7ee67feb1f02f7bc3020126514b8from 2010 reverts the effect commit28c494c5c8d425e15b7b82571e4df6d6bc34594dfrom Chunk in 2007.
I'm wondering if a subsequent commit changed filemap_write_and_wait().
There was some recent clean up that (very likely unintentionally) changed the serialization of some of these VFS utility functions.
Quoted 64 lines
Specifically it removes mutex lock/unlock in
nfs_getattr. Chuck added them:(quoted diff) +11
-2- /* Flush out writes to the server in order to update c/mtime */ - if (S_ISREG(inode->i_mode)) + /* + * Flush out writes to the server in order to update c/mtime. + * + * Hold the i_mutex to suspend application writes temporarily; + * this prevents long-running writing applications from blocking + * nfs_wb_nocommit. + */ + if (S_ISREG(inode->i_mode)) { + mutex_lock(&inode->i_mutex); nfs_wb_nocommit(inode); + mutex_unlock(&inode->i_mutex); + }You removed them.
(quoted diff) +4
-10- /* - * Flush out writes to the server in order to update c/mtime. - * - * Hold the i_mutex to suspend application writes temporarily; - * this prevents long-running writing applications from blocking - * nfs_wb_nocommit. - */ + /* Flush out writes to the server in order to update c/mtime. */ if (S_ISREG(inode->i_mode)) { - mutex_lock(&inode->i_mutex); - nfs_wb_nocommit(inode); - mutex_unlock(&inode->i_mutex); + err = filemap_write_and_wait(inode->i_mapping); + if (err) + goto out; } /*Do you recall why?
I noticed because a customer reported exactly the same symptoms the were fixed by Chucks patch some years ago.
The comment on your patch says (in part):
Also replace
nfs_wb_nocommit()with a
call tofilemap_write_and_wait(), which doesn't need to hold theinode->i_mutex.It is certainly true that
filemap_write_and_waitdoesn't need to hold the mutex, but neither didnfs_wb_nocommit. The mutex is held to stop "suspend application writes temporarily" so no more pages get dirtied until all the current dirty pages have been written out. i.e. to stopgeneric_file_aio_write()from proceeding.The particular test that shows the problem is a large write like
dd if=/dev/zero of=/mnt/nfs/somefile count=2000000
then in another window
ls -l/mnt/nfs
Quoted 6 lines
the "ls -l" will hang until the "dd" completes.
Can we put the mutex lock/unlock back please?
Thanks,
NeilBrown
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 11 lines · On Mon, 25 Nov 2013 09:59:39 -0500 Chuck Lever <[email protected]> wrote:
Hi Neil-
Quoted 5 lines · NeilBrown
Hi Trond,
I just noticed commitacdc53b2146c7ee67feb1f02f7bc3020126514b8from 2010 reverts the effect commit28c494c5c8d425e15b7b82571e4df6d6bc34594dfrom Chunk in 2007.I'm wondering if a subsequent commit changed
filemap_write_and_wait().
I hadn't thought of that possibility. I've just had a look at the differences between acdc53b2146c7ee6 and now and cannot find anything that could be related.
Thanks,
NeilBrown
Quoted 5 lines
There was some recent clean up that (very likely unintentionally) changed the serialization of some of these VFS utility functions.
Quoted 12 lines · NeilBrown
Quoted 10 lines · On Mon, 25 Nov 2013 09:59:39 -0500 Chuck Lever <[email protected]> wrote:
Hi Neil-
Quoted 5 lines · NeilBrown
Hi Trond,
I just noticed commitacdc53b2146c7ee67feb1f02f7bc3020126514b8from 2010 reverts the effect commit28c494c5c8d425e15b7b82571e4df6d6bc34594dfrom Chunk in 2007.
Quoted 5 lines
Quoted 1 line
I'm wondering if a subsequent commit changed
filemap_write_and_wait().I hadn't thought of that possibility. I've just had a look at the differences between
acdc53b2146c7ee6and now and cannot find anything that could be related.
To clarify a little: my understanding is that the current 2-pass code in write_cache_pages() is supposed to prevent livelock. Instead of chasing PAGECACHE_TAG_DIRTY tags (which are constantly being set if an application is actively writing), we call tag_pages_for_writeback() once in order to convert the current set of PAGECACHE_TAG_DIRTY tags into PAGECACHE_TAG_TOWRITE tags, and then we have a second pass write those pages back (and wait for completion).
IOW: the inode->i_mutex should be unnecessary here...
Now that said, we recently added in the call to nfs_inode_dio_wait(). If applications are using O_DIRECT, then _that_ could livelock. There is nothing currently preventing the applications from continuing to bump the inode->i_dio_count while we're waiting. Christoph has proposed some locking changes that should fix that problem. I'm still evaluating his patchset...
Cheers
Trond
Cheers
Trond
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
Quoted 19 lines · Trond
Quoted 12 lines · NeilBrown
Quoted 10 lines · On Mon, 25 Nov 2013 09:59:39 -0500 Chuck Lever <[email protected]> wrote:
Hi Neil-
Quoted 5 lines · NeilBrown
Hi Trond,
I just noticed commitacdc53b2146c7ee67feb1f02f7bc3020126514b8from 2010 reverts the effect commit28c494c5c8d425e15b7b82571e4df6d6bc34594dfrom Chunk in 2007.Quoted 5 lines
Quoted 1 line
I'm wondering if a subsequent commit changed
filemap_write_and_wait().I hadn't thought of that possibility. I've just had a look at the differences between
acdc53b2146c7ee6and now and cannot find anything that could be related.
Quoted 8 lines
To clarify a little: my understanding is that the current 2-pass code in
write_cache_pages()is supposed to prevent livelock. Instead of chasingPAGECACHE_TAG_DIRTYtags (which are constantly being set if an application is actively writing), we calltag_pages_for_writeback()once in order to convert the current set ofPAGECACHE_TAG_DIRTYtags intoPAGECACHE_TAG_TOWRITEtags, and then we have a second pass write those pages back (and wait for completion).
IOW: theinode->i_mutexshould be unnecessary here...
Thanks for that pointer. You are right, write_cache_pages shouldn't loop indefinitely any more.
And I also just noticed commit 72cb77f4a5ace37b12dcb47a0e8637a2c28ad881 and NFS_INO_FLUSHING which is an extra reason that the i_mutex isn't needed. Don't know how I missed that last night. Less haste, more speed I guess.
So it looks like I jumped the gun and there must be some other explanation. Sorry 'bout that.
Thanks,
NeilBrown
Quoted 14 lines
Now that said, we recently added in the call to
nfs_inode_dio_wait(). If applications are usingO_DIRECT, then _that_ could livelock. There is nothing currently preventing the applications from continuing to bump theinode->i_dio_countwhile we're waiting. Christoph has proposed some locking changes that should fix that problem. I'm still evaluating his patchset...Cheers
TrondCheers
Trond