Skip to contentSkip to conversation

prune back iprune_sem

RFC PATCH In review Christoph Hellwig 2 participants 5 messages Nov 2, 2010 – Feb 15, 2011 VFS

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 prune back iprune_sem

iprune_sem is continously giving us lockdep warnings because we do take it in read mode in the reclaim path, but we're also doing non-NOFS allocations under it taken in write mode.

Taking a bit deeper look at it I think it's fixable quite trivially:

- for invalidate_inodes we do not need iprune_sem at all. We have an active reference on the superblock, so the filesystem is not going away until it has finished.
- for evict_inodes we do need it, to make sure prune_icache has done it's work before we tear down the superblock. But there is no reason to hold it over the actual reclaim operation - it's enough to cycle through it after the actual reclaim to make sure we wait for any pending prune_icache to complete.

fs/inode.c +7 -5
index ae2727a..cfa7722 100644
@@ -492,8 +492,6 @@ void evict_inodes(struct super_block *sb)
 	struct inode *inode, *next;
 	LIST_HEAD(dispose);
 
-	down_write(&iprune_sem);
-
 	spin_lock(&inode_lock);
 	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
 		if (atomic_read(&inode->i_count))
@@ -518,6 +516,13 @@ void evict_inodes(struct super_block *sb)
 	spin_unlock(&inode_lock);
 
 	dispose_list(&dispose);
+
+	/*
+	 * Cycle through iprune_sem to make sure any inode that prune_icache
+	 * moved off the list before we took the lock has been fully torn
+	 * down.
+	 */
+	down_write(&iprune_sem);
 	up_write(&iprune_sem);
 }
 
@@ -534,8 +539,6 @@ int invalidate_inodes(struct super_block *sb)
 	struct inode *inode, *next;
 	LIST_HEAD(dispose);
 
-	down_write(&iprune_sem);
-
 	spin_lock(&inode_lock);
 	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
 		if (inode->i_state & (I_NEW | I_FREEING | I_WILL_FREE))
@@ -559,7 +562,6 @@ int invalidate_inodes(struct super_block *sb)
 	spin_unlock(&inode_lock);
 
 	dispose_list(&dispose);
-	up_write(&iprune_sem);
 
 	return busy;
 }
reviewer
#1 lore raw
Quoted 6 lines · Christoph Hellwig

iprune_sem is continously giving us lockdep warnings because we do take it in read mode in the reclaim path, but we're also doing non-NOFS allocations under it taken in write mode.

Taking a bit deeper look at it I think it's fixable quite trivially:

Quoted 8 lines

- for invalidate_inodes we do not need iprune_sem at all. We have an active reference on the superblock, so the filesystem is not going away until it has finished.
- for evict_inodes we do need it, to make sure prune_icache has done it's work before we tear down the superblock. But there is no reason to hold it over the actual reclaim operation - it's enough to cycle through it after the actual reclaim to make sure we wait for any pending prune_icache to complete.

The patch is OK but it's kind of subtle that evict_inodes() can now skip
some inode in the LRU list because prune_icache() is just processing it and so it has elevated i_count. Everything will work out fine because MS_ACTIVE is cleared and thus iput() will destroy the inode and prune_icache() will then just continue with the next inode in the inode_lru list. But as I said above it's subtle...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#2 lore raw
In reply to Christoph Hellwig

ping?

Quoted 54 lines · Christoph Hellwig

iprune_sem is continously giving us lockdep warnings because we do take it in read mode in the reclaim path, but we're also doing non-NOFS allocations under it taken in write mode.

Taking a bit deeper look at it I think it's fixable quite trivially:

- for invalidate_inodes we do not need iprune_sem at all. We have an active reference on the superblock, so the filesystem is not going away until it has finished.
- for evict_inodes we do need it, to make sure prune_icache has done it's work before we tear down the superblock. But there is no reason to hold it over the actual reclaim operation - it's enough to cycle through it after the actual reclaim to make sure we wait for any pending prune_icache to complete.

fs/inode.c +7 -4
index ae2727a..cfa7722 100644
@@ -492,8 +492,6 @@ void evict_inodes(struct super_block *sb)
 	struct inode *inode, *next;
 	LIST_HEAD(dispose);
 
-	down_write(&iprune_sem);
-
 	spin_lock(&inode_lock);
 	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
 		if (atomic_read(&inode->i_count))
@@ -518,6 +516,13 @@ void evict_inodes(struct super_block *sb)
 	spin_unlock(&inode_lock);
 
 	dispose_list(&dispose);
+
+	/*
+	 * Cycle through iprune_sem to make sure any inode that prune_icache
+	 * moved off the list before we took the lock has been fully torn
+	 * down.
+	 */
+	down_write(&iprune_sem);
 	up_write(&iprune_sem);
 }
 
@@ -534,8 +539,6 @@ int invalidate_inodes(struct super_block *sb)
 	struct inode *inode, *next;
 	LIST_HEAD(dispose);
 
-	down_write(&iprune_sem);
-
 	spin_lock(&inode_lock);
 	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
 		if (inode->i_state & (I_NEW | I_FREEING | I_WILL_FREE))

fs/inode.c int invalidate_inodes(struct super_block *sb) · line 567

Quoted 7 lines
fs/inode.c +0 -1
@@ -559,7 +562,6 @@ int invalidate_inodes(struct super_block *sb)
 	spin_unlock(&inode_lock);
 
 	dispose_list(&dispose);
-	up_write(&iprune_sem);
 
 	return busy;
 }

---end quoted text---

reviewer
#3 lore raw

Hi,

Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Christoph Hellwig

iprune_sem is continously giving us lockdep warnings because we do take it in read mode in the reclaim path, but we're also doing non-NOFS allocations under it taken in write mode.

Taking a bit deeper look at it I think it's fixable quite trivially:

Quoted 8 lines
Quoted 8 lines

- for invalidate_inodes we do not need iprune_sem at all. We have an active reference on the superblock, so the filesystem is not going away until it has finished.
- for evict_inodes we do need it, to make sure prune_icache has done it's work before we tear down the superblock. But there is no reason to hold it over the actual reclaim operation - it's enough to cycle through it after the actual reclaim to make sure we wait for any pending prune_icache to complete.

I just wonder: So with this change, evict_inodes() can start seeing
inodes, that are just being freed by prune_icache(). Thus we can trigger WARN_ON() in evict_inodes():

                if (inode->i_state & (I_NEW | I_FREEING | I_WILL_FREE)) {
                        WARN_ON(1);
                        continue;
                }

Otherwise, the change looks safe to me. BTW, the iprune_sem is now used
only so that evict_inodes() can wait for prune_icache() to finish so maybe we could have something simpler for that?

Honza

Quoted 52 lines
Quoted 40 lines
fs/inode.c +7 -4
index ae2727a..cfa7722 100644
@@ -492,8 +492,6 @@ void evict_inodes(struct super_block *sb)
 	struct inode *inode, *next;
 	LIST_HEAD(dispose);
 
-	down_write(&iprune_sem);
-
 	spin_lock(&inode_lock);
 	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
 		if (atomic_read(&inode->i_count))
@@ -518,6 +516,13 @@ void evict_inodes(struct super_block *sb)
 	spin_unlock(&inode_lock);
 
 	dispose_list(&dispose);
+
+	/*
+	 * Cycle through iprune_sem to make sure any inode that prune_icache
+	 * moved off the list before we took the lock has been fully torn
+	 * down.
+	 */
+	down_write(&iprune_sem);
 	up_write(&iprune_sem);
 }
 
@@ -534,8 +539,6 @@ int invalidate_inodes(struct super_block *sb)
 	struct inode *inode, *next;
 	LIST_HEAD(dispose);
 
-	down_write(&iprune_sem);
-
 	spin_lock(&inode_lock);
 	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
 		if (inode->i_state & (I_NEW | I_FREEING | I_WILL_FREE))

fs/inode.c int invalidate_inodes(struct super_block *sb) · line 567

Quoted 7 lines
fs/inode.c +0 -1
@@ -559,7 +562,6 @@ int invalidate_inodes(struct super_block *sb)
 	spin_unlock(&inode_lock);
 
 	dispose_list(&dispose);
-	up_write(&iprune_sem);
 
 	return busy;
 }

---end quoted text---
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#4 lore raw
Quoted 6 lines · Jan Kara

inodes, that are just being freed by prune_icache(). Thus we can trigger WARN_ON() in evict_inodes():

                if (inode->i_state & (I_NEW | I_FREEING | I_WILL_FREE)) {
                        WARN_ON(1);
                        continue;
                }

That WARN_ON didn't exist when I submitted the patch three month ago, but yes, it should be removed now.

Quoted 3 lines

Otherwise, the change looks safe to me. BTW, the iprune_sem is now used
only so that evict_inodes() can wait for prune_icache() to finish so maybe we could have something simpler for that?

I can't think of anything simple. The proper way to do it would be to make the inode lru per-sb just like the dentry lru list. That way we always hold a reference to the superblock while pruning inodes form the LRU and all associated issues go away. Dave had a patch for this as part of implementing a

	for_each_sb {
		prune dcache;
		prune icache;
		prune fs-specific cache;
	}

algorithm. I still think it's the right way to go, but it fell under the table and I really need a way to fix the lockdep warning / rare deadlock the current scheme causes for XFS.