Skip to contentSkip to conversation

dquot: do full inode dirty in allocating space

PATCH In review Shaohua Li 2 participants 4 messages Aug 20, 2010 – Aug 24, 2010 Diskquota

lore
Share & tools
lore

Shaohua Li

author
#0 lore raw

PATCH 1/1 dquot: do full inode dirty in allocating space

Alex Shi found a regression when doing ffsb test. The test has several threads, and each thread creates a small file, write to it and then delete it. ffsb reports about 20% regression and Alex bisected it to 43d2932d88e4. The test will call __mark_inode_dirty 3 times. without this commit, we only take inode_lock one time, while with it, we take the lock 3 times with flags ( I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased too much. Below proposed patch fixes it.

fs is allocating blocks, which usually means file writes and the inode will be dirtied soon. We fully dirty the inode to reduce some inode_lock contention in several calls of __mark_inode_dirty.

include/linux/quotaops.h +1 -1
index d50ba85..8dd2cc3 100644
@@ -275,7 +275,7 @@ static inline int dquot_alloc_space(struct inode *inode, qsize_t nr)
 
 	ret = dquot_alloc_space_nodirty(inode, nr);
 	if (!ret)
-		mark_inode_dirty_sync(inode);
+		mark_inode_dirty(inode);
 	return ret;
 }
 
#1 lore raw

Hi,

Quoted 7 lines · Shaohua Li

Alex Shi found a regression when doing ffsb test. The test has several threads, and each thread creates a small file, write to it and then delete it. ffsb reports about 20% regression and Alex bisected it to 43d2932d88e4. The test will call __mark_inode_dirty 3 times. without this commit, we only take inode_lock one time, while with it, we take the lock 3 times with flags ( I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased too much. Below proposed patch fixes it.

Thanks for the analysis and the patch! With which filesystem have you
measured the results? And what kind of machine it was?

Quoted 3 lines

fs is allocating blocks, which usually means file writes and the inode will be dirtied soon. We fully dirty the inode to reduce some inode_lock contention in several calls of __mark_inode_dirty.

Well, this is rather a workaround for a poor handling of inode dirty
bits. BTW, I think Nick's VFS scalability patches address inode_lock contention as well so with them the contention you see should be reduced.
Anyway, I will take this patch for the time before inode_lock
contention improves and add a proper comment about this before mark_inode_dirty.

Honza

Quoted 17 lines
include/linux/quotaops.h +1 -1
index d50ba85..8dd2cc3 100644
@@ -275,7 +275,7 @@ static inline int dquot_alloc_space(struct inode *inode, qsize_t nr)
 
 	ret = dquot_alloc_space_nodirty(inode, nr);
 	if (!ret)
-		mark_inode_dirty_sync(inode);
+		mark_inode_dirty(inode);
 	return ret;
 }
 
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#2 lore raw
Quoted 16 lines · Jan Kara

Hi,

Quoted 7 lines · Shaohua Li

Alex Shi found a regression when doing ffsb test. The test has several threads, and each thread creates a small file, write to it and then delete it. ffsb reports about 20% regression and Alex bisected it to 43d2932d88e4. The test will call __mark_inode_dirty 3 times. without this commit, we only take inode_lock one time, while with it, we take the lock 3 times with flags ( I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased too much. Below proposed patch fixes it.

Thanks for the analysis and the patch! With which filesystem have you
measured the results? And what kind of machine it was?

Quoted 3 lines

fs is allocating blocks, which usually means file writes and the inode will be dirtied soon. We fully dirty the inode to reduce some inode_lock contention in several calls of __mark_inode_dirty.

Quoted 6 lines

Well, this is rather a workaround for a poor handling of inode dirty
bits. BTW, I think Nick's VFS scalability patches address inode_lock contention as well so with them the contention you see should be reduced.
Anyway, I will take this patch for the time before inode_lock
contention improves and add a proper comment about this before mark_inode_dirty.

Attached is a version of the patch that is in my tree now.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Quoted 1 line

From 2a7ad4df051b43a481862fe0afa50539bc7e6b78 Mon Sep 17 00:00:00 2001

From: Shaohua Li <[email protected]>
Date: Fri, 20 Aug 2010 16:49:43 +0800
Subject: [PATCH] dquot: do full inode dirty in allocating space

Alex Shi found a regression when doing ffsb test. The test has several threads, and each thread creates a small file, write to it and then delete it. ffsb reports about 20% regression and Alex bisected it to 43d2932d88e4. The test will call __mark_inode_dirty 3 times. without this commit, we only take inode_lock one time, while with it, we take the lock 3 times with flags ( I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased too much. Below proposed patch fixes it.

fs is allocating blocks, which usually means file writes and the inode will be dirtied soon. We fully dirty the inode to reduce some inode_lock contention in several calls of __mark_inode_dirty.

Jan Kara: Added comment.

---
 include/linux/quotaops.h |   10 ++++++++--
 1 files changed, 8 insertions(+), 2 deletions(-)
include/linux/quotaops.h +8 -2
index d50ba85..d1a9193 100644
@@ -274,8 +274,14 @@ static inline int dquot_alloc_space(struct inode *inode, qsize_t nr)
 	int ret;
 
 	ret = dquot_alloc_space_nodirty(inode, nr);
-	if (!ret)
-		mark_inode_dirty_sync(inode);
+	if (!ret) {
+		/*
+		 * Mark inode fully dirty. Since we are allocating blocks, inode
+		 * would become fully dirty soon anyway and it reportedly
+		 * reduces inode_lock contention.
+		 */
+		mark_inode_dirty(inode);
+	}
 	return ret;
 }
 
Signature · 1 line
1.6.4.2
author
#3 lore raw
In reply to Jan Kara
Quoted 10 lines · Jan Kara

Hi,

Quoted 7 lines · Shaohua Li

Alex Shi found a regression when doing ffsb test. The test has several threads, and each thread creates a small file, write to it and then delete it. ffsb reports about 20% regression and Alex bisected it to 43d2932d88e4. The test will call __mark_inode_dirty 3 times. without this commit, we only take inode_lock one time, while with it, we take the lock 3 times with flags ( I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased too much. Below proposed patch fixes it.

Quoted 2 lines

Thanks for the analysis and the patch! With which filesystem have you
measured the results? And what kind of machine it was?

it's ext3 and 24 CPU system with two sockets.

Quoted 6 lines
Quoted 3 lines

fs is allocating blocks, which usually means file writes and the inode will be dirtied soon. We fully dirty the inode to reduce some inode_lock contention in several calls of __mark_inode_dirty.

Well, this is rather a workaround for a poor handling of inode dirty
bits. BTW, I think Nick's VFS scalability patches address inode_lock contention as well so with them the contention you see should be reduced.

yep, the VFS scalability patch might reduce this.

Quoted 3 lines

Anyway, I will take this patch for the time before inode_lock
contention improves and add a proper comment about this before mark_inode_dirty.

Thanks.