Skip to contentSkip to conversation

Possible problem with commit a6305ddb080 : NFS: Fix a race with the new commit code

DISCUSSION Trond Myklebust 3 participants 4 messages Apr 27, 2010 – May 3, 2010 5 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw
Quoted 6 lines · Neil Brown

Hi Trond,
I think the above mentioned commit might have added a new race to replace
the old ....

I have report of a BUG in nfs_page_async_flush.

Quoted 8 lines

It isn't a vanilla upstream kernel - there are a bunch of SUSE patches in there - so quoting the line-number won't help you, but it is the

    BUG_ON(ret != 0);

after the call to nfs_set_page_writeback.
(https://bugzilla.novell.com/show_bug.cgi?id=599628)

This implies that nfs_find_and_lock_request got a new lock on the page, and then we found that it was already flagged for writeback.

That's odd. Callers such as write_cache_pages() should normally be doing a wait_on_page_writeback() after taking the page lock but prior to calling the filesystem.

Quoted 5 lines

The commit mentioned create just such an opportunity. It reorders things so that a page is unlocked before writeback is cleared, thus creating a window for that BUG to fire.

What is the race that you were trying to fix?

I want to ensure that the call to write_inode() immediately after filemap_fdatawait() in the function writeback_single_inode() works correctly. Prior to this fix, there was a race whereby the Pg_writeback flag could be cleared, but the nfs_page structure would still be locked. This again would result in the page being skipped by nfs_scan_list() and so it wouldn't be registered as COMMITed.

Cheers
Trond

A message that is not in the archive (sent off-list or not yet received): 1272369635.16814.52.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]

#4 lore raw
Quoted 13 lines · Trond Myklebust
Quoted 12 lines · Neil Brown

Hi Trond,
I think the above mentioned commit might have added a new race to replace
the old ....

I have report of a BUG in nfs_page_async_flush.

It isn't a vanilla upstream kernel - there are a bunch of SUSE patches in there - so quoting the line-number won't help you, but it is the

    BUG_ON(ret != 0);

after the call to nfs_set_page_writeback.
(https://bugzilla.novell.com/show_bug.cgi?id=599628)

Quoted 6 lines
Quoted 2 lines

This implies that nfs_find_and_lock_request got a new lock on the page, and then we found that it was already flagged for writeback.

That's odd. Callers such as write_cache_pages() should normally be doing a wait_on_page_writeback() after taking the page lock but prior to calling the filesystem.

The following patch ought to fix it. I suspect the same race exists in the ->readpage() path, so it makes sense to fix nfs_wb_page() rather than putting the wait_on_page_writeback call in nfs_try_to_update_request().

Cheers
Trond
------------------------------------------------------------------------------------------------
NFS: Ensure that nfs_wb_page() waits for Pg_writeback to clear

From: Trond Myklebust <[email protected]>

Neil Brown reports that he is seeing the BUG_ON(ret == 0) trigger in nfs_page_async_flush.

---

 fs/nfs/write.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
fs/nfs/write.c +3 -0
index ccde2ae..c700698 100644
@@ -1472,6 +1472,7 @@ int nfs_wb_page_cancel(struct inode *inode, struct page *page)
 
 	BUG_ON(!PageLocked(page));
 	for (;;) {
+		wait_on_page_writeback(page);
 		req = nfs_page_find_request(page);
 		if (req == NULL)
 			break;
@@ -1511,10 +1512,12 @@ int nfs_wb_page(struct inode *inode, struct page *page)
 	int ret;
 
 	while(PagePrivate(page)) {
+		wait_on_page_writeback(page);
 		if (clear_page_dirty_for_io(page)) {
 			ret = nfs_writepage_locked(page, &wbc);
 			if (ret < 0)
 				goto out_error;
+			continue;
 		}
 		req = nfs_find_and_lock_request(page);
 		if (!req)

A message that is not in the archive (sent off-list or not yet received): 1272406873.14667.6.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]

#6 lore raw
Quoted 17 lines · Trond Myklebust
Quoted 16 lines · Trond Myklebust
Quoted 14 lines · Neil Brown

Hi Trond,
I think the above mentioned commit might have added a new race to replace
the old ....

I have report of a BUG in nfs_page_async_flush.

It isn't a vanilla upstream kernel - there are a bunch of SUSE patches in there - so quoting the line-number won't help you, but it is the

    BUG_ON(ret != 0);

after the call to nfs_set_page_writeback.
(https://bugzilla.novell.com/show_bug.cgi?id=599628)

This implies that nfs_find_and_lock_request got a new lock on the page, and then we found that it was already flagged for writeback.

Quoted 8 lines
Quoted 3 lines

That's odd. Callers such as write_cache_pages() should normally be doing a wait_on_page_writeback() after taking the page lock but prior to calling the filesystem.

The following patch ought to fix it. I suspect the same race exists in the ->readpage() path, so it makes sense to fix nfs_wb_page() rather than putting the wait_on_page_writeback call in nfs_try_to_update_request().

Actually, this patch is even better since it cleans up nfs_wb_page() too.

Cheers
Trond
------------------------------------------------------------------------------------------
NFS: Ensure that nfs_wb_page() waits for Pg_writeback to clear

From: Trond Myklebust <[email protected]>

Neil Brown reports that he is seeing the BUG_ON(ret == 0) trigger in nfs_page_async_flush. According to the trace in

     https://bugzilla.novell.com/show_bug.cgi?id=599628

the problem appears to be due to nfs_wb_page() not waiting for the PG_writeback flag to clear.

There is a ditto problem in nfs_wb_page_cancel()

---

 fs/nfs/write.c |   19 ++++---------------
 1 files changed, 4 insertions(+), 15 deletions(-)
fs/nfs/write.c +4 -15
index ccde2ae..3aea3ca 100644
@@ -1472,6 +1472,7 @@ int nfs_wb_page_cancel(struct inode *inode, struct page *page)
 
 	BUG_ON(!PageLocked(page));
 	for (;;) {
+		wait_on_page_writeback(page);
 		req = nfs_page_find_request(page);
 		if (req == NULL)
 			break;
@@ -1506,30 +1507,18 @@ int nfs_wb_page(struct inode *inode, struct page *page)
 		.range_start = range_start,
 		.range_end = range_end,
 	};
-	struct nfs_page *req;
-	int need_commit;
 	int ret;
 
 	while(PagePrivate(page)) {
+		wait_on_page_writeback(page);
 		if (clear_page_dirty_for_io(page)) {
 			ret = nfs_writepage_locked(page, &wbc);
 			if (ret < 0)
 				goto out_error;
 		}
-		req = nfs_find_and_lock_request(page);
-		if (!req)
-			break;
-		if (IS_ERR(req)) {
-			ret = PTR_ERR(req);
+		ret = sync_inode(inode, &wbc);
+		if (ret < 0)
 			goto out_error;
-		}
-		need_commit = test_bit(PG_CLEAN, &req->wb_flags);
-		nfs_clear_page_tag_locked(req);
-		if (need_commit) {
-			ret = nfs_commit_inode(inode, FLUSH_SYNC);
-			if (ret < 0)
-				goto out_error;
-		}
 	}
 	return 0;
 out_error:

A message that is not in the archive (sent off-list or not yet received): 1272407756.14667.17.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]

#8 lore raw
Quoted 22 lines · On Tue, 27 Apr 2010 18:35:56 -0400 Trond Myklebust…
Quoted 21 lines · Trond Myklebust
Quoted 19 lines · Trond Myklebust
Quoted 6 lines · Neil Brown

Hi Trond,
I think the above mentioned commit might have added a new race to replace
the old ....

I have report of a BUG in nfs_page_async_flush.

Quoted 8 lines

It isn't a vanilla upstream kernel - there are a bunch of SUSE patches in there - so quoting the line-number won't help you, but it is the

    BUG_ON(ret != 0);

after the call to nfs_set_page_writeback.
(https://bugzilla.novell.com/show_bug.cgi?id=599628)

This implies that nfs_find_and_lock_request got a new lock on the page, and then we found that it was already flagged for writeback.

That's odd. Callers such as write_cache_pages() should normally be doing a wait_on_page_writeback() after taking the page lock but prior to calling the filesystem.

Quoted 7 lines
Quoted 4 lines

The following patch ought to fix it. I suspect the same race exists in the ->readpage() path, so it makes sense to fix nfs_wb_page() rather than putting the wait_on_page_writeback call in nfs_try_to_update_request().

Actually, this patch is even better since it cleans up nfs_wb_page() too.

Thanks Trond!
I won't pretend to completely understand it, but it certainly looks credible and removes some code, which is always nice!

I don't think the problem was easily reproducible so I cannot easily test if this fixes it, so I'll just assume it does and let you know if I hear otherwise.

Thanks,
NeilBrown

Quoted 74 lines

Cheers
Trond
------------------------------------------------------------------------------------------
NFS: Ensure that nfs_wb_page() waits for Pg_writeback to clear

From: Trond Myklebust <[email protected]>

Neil Brown reports that he is seeing the BUG_ON(ret == 0) trigger in nfs_page_async_flush. According to the trace in

     https://bugzilla.novell.com/show_bug.cgi?id=599628

the problem appears to be due to nfs_wb_page() not waiting for the PG_writeback flag to clear.

There is a ditto problem in nfs_wb_page_cancel()

---

 fs/nfs/write.c |   19 ++++---------------
 1 files changed, 4 insertions(+), 15 deletions(-)
fs/nfs/write.c +4 -16
index ccde2ae..3aea3ca 100644
@@ -1472,6 +1472,7 @@ int nfs_wb_page_cancel(struct inode *inode, struct page *page)
 
 	BUG_ON(!PageLocked(page));
 	for (;;) {
+		wait_on_page_writeback(page);
 		req = nfs_page_find_request(page);
 		if (req == NULL)
 			break;
@@ -1506,30 +1507,18 @@ int nfs_wb_page(struct inode *inode, struct page *page)
 		.range_start = range_start,
 		.range_end = range_end,
 	};
-	struct nfs_page *req;
-	int need_commit;
 	int ret;
 
 	while(PagePrivate(page)) {
+		wait_on_page_writeback(page);
 		if (clear_page_dirty_for_io(page)) {
 			ret = nfs_writepage_locked(page, &wbc);
 			if (ret < 0)
 				goto out_error;
 		}
-		req = nfs_find_and_lock_request(page);
-		if (!req)
-			break;
-		if (IS_ERR(req)) {
-			ret = PTR_ERR(req);
+		ret = sync_inode(inode, &wbc);
+		if (ret < 0)
 			goto out_error;
-		}
-		need_commit = test_bit(PG_CLEAN, &req->wb_flags);
-		nfs_clear_page_tag_locked(req);
-		if (need_commit) {
-			ret = nfs_commit_inode(inode, FLUSH_SYNC);
-			if (ret < 0)
-				goto out_error;
-		}
 	}
 	return 0;
 out_error:
 
--

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html