Skip to contentSkip to conversation

Don't reset di_format in xfs_ifree()

PATCH In review Lachlan McIlroy 5 participants 11 messages Feb 11, 2009 – Feb 17, 2009 XFS filesystem

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Don't reset di_format in xfs_ifree()

I hit a panic while flushing a reclaimed inode that is fairly reproducible under load.

In xfs_iflush_fork() we're led to believe that there are extents on this inode but there aren't any. Actually the inode was a directory. I added some debugging to xfs_ifree() and found that di_format was XFS_DINODE_FMT_LOCAL and got reset to XFS_DINODE_FMT_EXTENTS and this has confused the code in xfs_iflush_fork().

[<ffffffffa0047de7>] xfs_iext_get_ext+0x6c/0xca [xfs]
[<ffffffffa004a3d4>] xfs_iflush_fork+0x1b0/0x3c6 [xfs]
[<ffffffffa004aa3f>] xfs_iflush_int+0x455/0x5a1 [xfs]
[<ffffffffa004b0da>] xfs_iflush+0x229/0x2d6 [xfs]
[<ffffffffa007283c>] xfs_reclaim_inode+0xd8/0x10f [xfs]
[<ffffffffa0072976>] xfs_reclaim_inodes_ag+0x103/0x13e [xfs]
[<ffffffffa00729f3>] xfs_reclaim_inodes+0x42/0x60 [xfs]
[<ffffffffa0072d1b>] xfs_sync_worker+0x30/0x8a [xfs]
[<ffffffffa0073416>] xfssyncd+0x14e/0x1a2 [xfs]
[<ffffffffa00732c8>] ? xfssyncd+0x0/0x1a2 [xfs]
[<ffffffff8104fc70>] kthread+0x49/0x79

I made this change and it passes the load test and XFSQA too. I'm not sure if this is indicative of a bigger problem though.

fs/xfs/xfs_inode.c +0 -2
Index: xfs-fix/fs/xfs/xfs_inode.c
===================================================================
@@ -2165,8 +2165,6 @@ xfs_ifree(
 	ip->i_d.di_forkoff = 0;		/* mark the attr fork not in use */
 	ip->i_df.if_ext_max =
 		XFS_IFORK_DSIZE(ip) / (uint)sizeof(xfs_bmbt_rec_t);
-	ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS;
-	ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS;
 	/*
 	 * Bump the generation count so no one will be confused
 	 * by reincarnations of this inode.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 2 lines · Lachlan McIlroy

I hit a panic while flushing a reclaimed inode that is fairly reproducible under load.

With what testcase?

Thanks,
-Eric

Quoted 36 lines

In xfs_iflush_fork() we're led to believe that there are extents on this inode but there aren't any. Actually the inode was a directory. I added some debugging to xfs_ifree() and found that di_format was XFS_DINODE_FMT_LOCAL and got reset to XFS_DINODE_FMT_EXTENTS and this has confused the code in xfs_iflush_fork().

[<ffffffffa0047de7>] xfs_iext_get_ext+0x6c/0xca [xfs]
[<ffffffffa004a3d4>] xfs_iflush_fork+0x1b0/0x3c6 [xfs]
[<ffffffffa004aa3f>] xfs_iflush_int+0x455/0x5a1 [xfs]
[<ffffffffa004b0da>] xfs_iflush+0x229/0x2d6 [xfs]
[<ffffffffa007283c>] xfs_reclaim_inode+0xd8/0x10f [xfs]
[<ffffffffa0072976>] xfs_reclaim_inodes_ag+0x103/0x13e [xfs]
[<ffffffffa00729f3>] xfs_reclaim_inodes+0x42/0x60 [xfs]
[<ffffffffa0072d1b>] xfs_sync_worker+0x30/0x8a [xfs]
[<ffffffffa0073416>] xfssyncd+0x14e/0x1a2 [xfs]
[<ffffffffa00732c8>] ? xfssyncd+0x0/0x1a2 [xfs]
[<ffffffff8104fc70>] kthread+0x49/0x79

I made this change and it passes the load test and XFSQA too. I'm not sure if this is indicative of a bigger problem though.

Index: xfs-fix/fs/xfs/xfs_inode.c
===================================================================

fs/xfs/xfs_inode.c +0 -2
@@ -2165,8 +2165,6 @@ xfs_ifree(
  	ip->i_d.di_forkoff = 0;		/* mark the attr fork not in use */
  	ip->i_df.if_ext_max =
  		XFS_IFORK_DSIZE(ip) / (uint)sizeof(xfs_bmbt_rec_t);
-	ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS;
-	ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS;
  	/*
  	 * Bump the generation count so no one will be confused
  	 * by reincarnations of this inode.
Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 5 lines · Eric Sandeen
Quoted 2 lines · Lachlan McIlroy

I hit a panic while flushing a reclaimed inode that is fairly reproducible under load.

With what testcase?

Initially with fsstress but then I wrote the extent.c program I attached to the next patch I posted.

Quoted 40 lines

Thanks,
-Eric

Quoted 36 lines

In xfs_iflush_fork() we're led to believe that there are extents on this inode but there aren't any. Actually the inode was a directory. I added some debugging to xfs_ifree() and found that di_format was XFS_DINODE_FMT_LOCAL and got reset to XFS_DINODE_FMT_EXTENTS and this has confused the code in xfs_iflush_fork().

[<ffffffffa0047de7>] xfs_iext_get_ext+0x6c/0xca [xfs]
[<ffffffffa004a3d4>] xfs_iflush_fork+0x1b0/0x3c6 [xfs]
[<ffffffffa004aa3f>] xfs_iflush_int+0x455/0x5a1 [xfs]
[<ffffffffa004b0da>] xfs_iflush+0x229/0x2d6 [xfs]
[<ffffffffa007283c>] xfs_reclaim_inode+0xd8/0x10f [xfs]
[<ffffffffa0072976>] xfs_reclaim_inodes_ag+0x103/0x13e [xfs]
[<ffffffffa00729f3>] xfs_reclaim_inodes+0x42/0x60 [xfs]
[<ffffffffa0072d1b>] xfs_sync_worker+0x30/0x8a [xfs]
[<ffffffffa0073416>] xfssyncd+0x14e/0x1a2 [xfs]
[<ffffffffa00732c8>] ? xfssyncd+0x0/0x1a2 [xfs]
[<ffffffff8104fc70>] kthread+0x49/0x79

I made this change and it passes the load test and XFSQA too. I'm not sure if this is indicative of a bigger problem though.

Index: xfs-fix/fs/xfs/xfs_inode.c
===================================================================

fs/xfs/xfs_inode.c +0 -2
@@ -2165,8 +2165,6 @@ xfs_ifree(
  	ip->i_d.di_forkoff = 0;		/* mark the attr fork not in use */
  	ip->i_df.if_ext_max =
  		XFS_IFORK_DSIZE(ip) / (uint)sizeof(xfs_bmbt_rec_t);
-	ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS;
-	ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS;
  	/*
  	 * Bump the generation count so no one will be confused
  	 * by reincarnations of this inode.
Quoted 6 lines
Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
Quoted 7 lines · Lachlan McIlroy
Quoted 4 lines · Eric Sandeen
Quoted 2 lines · Lachlan McIlroy

I hit a panic while flushing a reclaimed inode that is fairly reproducible under load.

With what testcase?

Initially with fsstress but then I wrote the extent.c program I attached to the next patch I posted.

Ah, ok, thanks.

-Eric

Quoted 2 lines
Quoted 2 lines

Thanks,
-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw
In reply to Lachlan McIlroy
Quoted 2 lines · Lachlan McIlroy

I hit a panic while flushing a reclaimed inode that is fairly reproducible under load.

What kind of panic was that? Where in xfs_iext_get_ext() did it panic?

Quoted 8 lines

In xfs_iflush_fork() we're led to believe that there are extents on this inode but there aren't any. Actually the inode was a directory. I added some debugging to xfs_ifree() and found that di_format was XFS_DINODE_FMT_LOCAL and got reset to XFS_DINODE_FMT_EXTENTS and this has confused the code in xfs_iflush_fork().

[<ffffffffa0047de7>] xfs_iext_get_ext+0x6c/0xca [xfs]

I assume you're running debug xfs, as I can see xfs_iext_get_ext() only in assert statements.

Quoted 19 lines
[<ffffffffa004a3d4>] xfs_iflush_fork+0x1b0/0x3c6 [xfs]
[<ffffffffa004aa3f>] xfs_iflush_int+0x455/0x5a1 [xfs]
[<ffffffffa004b0da>] xfs_iflush+0x229/0x2d6 [xfs]
[<ffffffffa007283c>] xfs_reclaim_inode+0xd8/0x10f [xfs]
[<ffffffffa0072976>] xfs_reclaim_inodes_ag+0x103/0x13e [xfs]
[<ffffffffa00729f3>] xfs_reclaim_inodes+0x42/0x60 [xfs]
[<ffffffffa0072d1b>] xfs_sync_worker+0x30/0x8a [xfs]
[<ffffffffa0073416>] xfssyncd+0x14e/0x1a2 [xfs]
[<ffffffffa00732c8>] ? xfssyncd+0x0/0x1a2 [xfs]
[<ffffffff8104fc70>] kthread+0x49/0x79

I made this change and it passes the load test and XFSQA too. I'm not sure if this is indicative of a bigger problem though.

Index: xfs-fix/fs/xfs/xfs_inode.c
===================================================================

fs/xfs/xfs_inode.c xfs_ifree( · line 2169

Quoted 5 lines
fs/xfs/xfs_inode.c +0 -2
@@ -2165,8 +2165,6 @@ xfs_ifree(
 	ip->i_d.di_forkoff = 0;		/* mark the attr fork not in use */
 	ip->i_df.if_ext_max =
 		XFS_IFORK_DSIZE(ip) / (uint)sizeof(xfs_bmbt_rec_t);
-	ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS;
-	ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS;

So, the idea here is to reset the ip->i_d. It seems strange to choose XFS_DINODE_FMT_EXTENTS as initializer, and even more strange how not changing di_aformat could affect your panic.

Just asking the questions at this time.
Felix

Quoted 8 lines
fs/xfs/xfs_inode.c +0 -0
 	/*
 	 * Bump the generation count so no one will be confused
 	 * by reincarnations of this inode.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 8 lines · Felix Blyakher
Quoted 2 lines · Lachlan McIlroy

I hit a panic while flushing a reclaimed inode that is fairly reproducible under load.

What kind of panic was that? Where in xfs_iext_get_ext() did it panic?

It was a bad address so I suspect that either ifp->if_u1.if_ext_irec or ifp->if_u1.if_extents was dereferenced when they have been freed (or not even allocated). The correct di_format was XFS_DINODE_FMT_LOCAL so ifp->if_u1.if_data should have been used.

Quoted 8 lines
Quoted 8 lines

In xfs_iflush_fork() we're led to believe that there are extents on this inode but there aren't any. Actually the inode was a directory. I added some debugging to xfs_ifree() and found that di_format was XFS_DINODE_FMT_LOCAL and got reset to XFS_DINODE_FMT_EXTENTS and this has confused the code in xfs_iflush_fork().

Quoted 4 lines
Quoted 1 line

[<ffffffffa0047de7>] xfs_iext_get_ext+0x6c/0xca [xfs]

I assume you're running debug xfs, as I can see xfs_iext_get_ext() only in assert statements.

Yes, debug.

Quoted 15 lines
Quoted 15 lines
[<ffffffffa004a3d4>] xfs_iflush_fork+0x1b0/0x3c6 [xfs]
[<ffffffffa004aa3f>] xfs_iflush_int+0x455/0x5a1 [xfs]
[<ffffffffa004b0da>] xfs_iflush+0x229/0x2d6 [xfs]
[<ffffffffa007283c>] xfs_reclaim_inode+0xd8/0x10f [xfs]
[<ffffffffa0072976>] xfs_reclaim_inodes_ag+0x103/0x13e [xfs]
[<ffffffffa00729f3>] xfs_reclaim_inodes+0x42/0x60 [xfs]
[<ffffffffa0072d1b>] xfs_sync_worker+0x30/0x8a [xfs]
[<ffffffffa0073416>] xfssyncd+0x14e/0x1a2 [xfs]
[<ffffffffa00732c8>] ? xfssyncd+0x0/0x1a2 [xfs]
[<ffffffff8104fc70>] kthread+0x49/0x79

I made this change and it passes the load test and XFSQA too. I'm not sure if this is indicative of a bigger problem though.

Quoted 14 lines

fs/xfs/xfs_inode.c xfs_ifree( · line 2169

Quoted 10 lines

Index: xfs-fix/fs/xfs/xfs_inode.c
===================================================================

fs/xfs/xfs_inode.c +0 -2
@@ -2165,8 +2165,6 @@ xfs_ifree(
     ip->i_d.di_forkoff = 0;        /* mark the attr fork not in use */
     ip->i_df.if_ext_max =
         XFS_IFORK_DSIZE(ip) / (uint)sizeof(xfs_bmbt_rec_t);
-    ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS;
-    ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS;

So, the idea here is to reset the ip->i_d. It seems strange to choose XFS_DINODE_FMT_EXTENTS as initializer, and even more strange how not changing di_aformat could affect your panic.

I figure if one (di_format) is wrong then the other probably is too.

Quoted 13 lines

Just asking the questions at this time.
Felix

Quoted 8 lines
fs/xfs/xfs_inode.c +0 -0
     /*
      * Bump the generation count so no one will be confused
      * by reincarnations of this inode.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
In reply to Lachlan McIlroy
Quoted 9 lines · Lachlan McIlroy

I hit a panic while flushing a reclaimed inode that is fairly reproducible under load.

In xfs_iflush_fork() we're led to believe that there are extents on this inode but there aren't any. Actually the inode was a directory. I added some debugging to xfs_ifree() and found that di_format was XFS_DINODE_FMT_LOCAL and got reset to XFS_DINODE_FMT_EXTENTS and this has confused the code in xfs_iflush_fork().

Wow. I wonder why we've never seen this before - it's not a new problem AFAICT.

A freed inode is supposed to have both forks in extent format with zero extents - it means the fork is empty. Changing it to local format means that it is not in the expected state for a subsequent create.

I think the problem may be that the size of the fork has not been reset to zero, not that format has been changed. If it was in local format, the truncates prior to freeing would not have done anything and the size of the data/attr fork would still be non-zero. Hence if the fork is then changed to extent format, xfs_iextents_copy() will be triggered from xfs_iflush_fork() and you'd see something like the confusion you are seeing.

Hence I think we should be ensuring the fork size is set to zero for both the attr/data fork when the format is changed, not removing the change of type....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw
Quoted 27 lines · Dave Chinner
Quoted 9 lines · Lachlan McIlroy

I hit a panic while flushing a reclaimed inode that is fairly reproducible under load.

In xfs_iflush_fork() we're led to believe that there are extents on this inode but there aren't any. Actually the inode was a directory. I added some debugging to xfs_ifree() and found that di_format was XFS_DINODE_FMT_LOCAL and got reset to XFS_DINODE_FMT_EXTENTS and this has confused the code in xfs_iflush_fork().

Wow. I wonder why we've never seen this before - it's not a new problem AFAICT.

A freed inode is supposed to have both forks in extent format with zero extents - it means the fork is empty. Changing it to local format means that it is not in the expected state for a subsequent create.

I think the problem may be that the size of the fork has not been reset to zero, not that format has been changed. If it was in local format, the truncates prior to freeing would not have done anything and the size of the data/attr fork would still be non-zero. Hence if the fork is then changed to extent format, xfs_iextents_copy() will be triggered from xfs_iflush_fork() and you'd see something like the confusion you are seeing.

Quoted 3 lines

Hence I think we should be ensuring the fork size is set to zero for both the attr/data fork when the format is changed, not removing the change of type....

Yes, I agree. I just don't have the time to hunt it down. I see there's a call to xfs_idestroy_fork() in xfs_ireclaim() for directories but xfs_ireclaim() gets called after xfs_iflush() in xfs_reclaim_inode(). Might also need something like:

@@ -2445,6 +2447,7 @@ xfs_idestroy_fork(

                        kmem_free(ifp->if_u1.if_data);
                        ifp->if_u1.if_data = NULL;
                        ifp->if_real_bytes = 0;
+                       ifp->if_bytes = 0;
                }
        } else if ((ifp->if_flags & XFS_IFEXTENTS) &&
                   ((ifp->if_flags & XFS_IFEXTIREC) ||

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw
Quoted 11 lines · Lachlan McIlroy
Quoted 10 lines · Dave Chinner
Quoted 9 lines · Lachlan McIlroy

I hit a panic while flushing a reclaimed inode that is fairly reproducible under load.

In xfs_iflush_fork() we're led to believe that there are extents on this inode but there aren't any. Actually the inode was a directory. I added some debugging to xfs_ifree() and found that di_format was XFS_DINODE_FMT_LOCAL and got reset to XFS_DINODE_FMT_EXTENTS and this has confused the code in xfs_iflush_fork().

....

Quoted 8 lines
Quoted 8 lines

I think the problem may be that the size of the fork has not been reset to zero, not that format has been changed. If it was in local format, the truncates prior to freeing would not have done anything and the size of the data/attr fork would still be non-zero. Hence if the fork is then changed to extent format, xfs_iextents_copy() will be triggered from xfs_iflush_fork() and you'd see something like the confusion you are seeing.

Quoted 7 lines
Quoted 3 lines

Hence I think we should be ensuring the fork size is set to zero for both the attr/data fork when the format is changed, not removing the change of type....

Yes, I agree. I just don't have the time to hunt it down. I see there's a call to xfs_idestroy_fork() in xfs_ireclaim() for directories but xfs_ireclaim() gets called after xfs_iflush() in xfs_reclaim_inode().

I suspect it should be in xfs_inactive() if we are in local format. This is what happens with the attribute fork. I think that is where we need something like:

	if ((ip->i_d.di_mode & S_IFMT) == S_IFDIR &&
	    ip->i_d.di_nextents == 0)
		xfs_idestroy_fork(ip, XFS_DATA_FORK);

Review xfs_idestroy_fork( · line 2453

Quoted 10 lines

Might also need something like:

(quoted diff) +1 -0
@@ -2445,6 +2447,7 @@ xfs_idestroy_fork(
                        kmem_free(ifp->if_u1.if_data);
                        ifp->if_u1.if_data = NULL;
                        ifp->if_real_bytes = 0;
+                       ifp->if_bytes = 0;
                }
        } else if ((ifp->if_flags & XFS_IFEXTENTS) &&
                   ((ifp->if_flags & XFS_IFEXTIREC) ||

Looking at that, the whole if (local) {} else if (extent/btree) code could probably be replaced with a single call to xfs_iext_destroy() as it does the cleanup correctly in both cases, anyway....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#9 lore raw
Quoted 11 lines · Dave Chinner
Quoted 3 lines

Yes, I agree. I just don't have the time to hunt it down. I see there's a call to xfs_idestroy_fork() in xfs_ireclaim() for directories but xfs_ireclaim() gets called after xfs_iflush() in xfs_reclaim_inode().

I suspect it should be in xfs_inactive() if we are in local format. This is what happens with the attribute fork. I think that is where we need something like:

	if ((ip->i_d.di_mode & S_IFMT) == S_IFDIR &&
	    ip->i_d.di_nextents == 0)
		xfs_idestroy_fork(ip, XFS_DATA_FORK);

Why would we special case directories?

Quoted 15 lines

Review xfs_idestroy_fork( · line 2453

Quoted 10 lines

Might also need something like:

(quoted diff) +1 -0
@@ -2445,6 +2447,7 @@ xfs_idestroy_fork(
                        kmem_free(ifp->if_u1.if_data);
                        ifp->if_u1.if_data = NULL;
                        ifp->if_real_bytes = 0;
+                       ifp->if_bytes = 0;
                }
        } else if ((ifp->if_flags & XFS_IFEXTENTS) &&
                   ((ifp->if_flags & XFS_IFEXTIREC) ||

Looking at that, the whole if (local) {} else if (extent/btree) code could probably be replaced with a single call to xfs_iext_destroy() as it does the cleanup correctly in both cases, anyway....

We could, but with the way the function name, comments and how the unions are set up it would be very confusing to the user.

Btw, I can't reproduce this issue with the extent.c program and the invocation from the next patch. Does it need other parameters to reproduce?

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Dave Chinner
Quoted 3 lines

Yes, I agree. I just don't have the time to hunt it down. I see there's a call to xfs_idestroy_fork() in xfs_ireclaim() for directories but xfs_ireclaim() gets called after xfs_iflush() in xfs_reclaim_inode().

I suspect it should be in xfs_inactive() if we are in local format. This is what happens with the attribute fork. I think that is where we need something like:

Quoted 5 lines
Quoted 3 lines
	if ((ip->i_d.di_mode & S_IFMT) == S_IFDIR &&
	    ip->i_d.di_nextents == 0)
		xfs_idestroy_fork(ip, XFS_DATA_FORK);

Why would we special case directories?

I guess we don't need to - I was thinking purely of local format inodes....

Quoted 3 lines

Btw, I can't reproduce this issue with the extent.c program and the invocation from the next patch. Does it need other parameters to reproduce?

No idea, I haven't been able to reproduce it either.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs