Skip to contentSkip to conversation

xfs: xfs_alloc_fix_minleft can underflow near ENOSPC

PATCH In review Dave Chinner 4 participants 10 messages Feb 12, 2015 – Feb 18, 2015 XFS filesystem

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xfs: xfs_alloc_fix_minleft can underflow near ENOSPC

From: Dave Chinner <[email protected]>

Test generic/224 is failing with a corruption being detected on one of Michael's test boxes. Debug that Michael added is indicating that the minleft trimming is resulting in an underflow:

.....

 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 before goto out_nominleft: rlen          1  args->len          0
 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 after fixup:               rlen          1  args->len          1
 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 after fixup:               rlen 4294967295  args->len 4294967295

XFS: Assertion failed: fs_is_ok, file: fs/xfs/libxfs/xfs_alloc.c, line: 1424

The "goto out_nominleft:" indicates that we are getting close to ENOSPC in the AG, and a couple of allocations later we underflow and the corruption check fires in xfs_alloc_ag_vextent_size().

The issue is that the extent length fixups comaprisons are done with variables of xfs_extlen_t types. These are unsigned so an underflow looks like a really big value and hence is not detected as being smaller than the minimum length allowed for the extent. Hence the corruption check fires as it is noticing that the returned length is longer than the original extent length passed in.

This can be easily fixed by ensuring we do the underflow test on signed values, the same way xfs_alloc_fix_len() prevents underflow. So we realise in future that these casts prevent underflows from going undetected, add comments to the code indicating this.

---
 fs/xfs/libxfs/xfs_alloc.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_alloc.c +3 -1
index 710554c..41e3630 100644
@@ -260,6 +260,7 @@ xfs_alloc_fix_len(
 		rlen = rlen - (k - args->mod);
 	else
 		rlen = rlen - args->prod + (args->mod - k);
+	/* casts to (int) catch length underflows */
 	if ((int)rlen < (int)args->minlen)
 		return;
 	ASSERT(rlen >= args->minlen && rlen <= args->maxlen);
@@ -286,7 +287,8 @@ xfs_alloc_fix_minleft(
 	if (diff >= 0)
 		return 1;
 	args->len += diff;		/* shrink the allocated space */
-	if (args->len >= args->minlen)
+	/* casts to (int) catch length underflows */
+	if ((int)args->len >= (int)args->minlen)
 		return 1;
 	args->agbno = NULLAGBLOCK;
 	return 0;
Signature · 6 lines
2.0.0

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw
Quoted 34 lines · Dave Chinner

From: Dave Chinner<[email protected]>

Test generic/224 is failing with a corruption being detected on one of Michael's test boxes. Debug that Michael added is indicating that the minleft trimming is resulting in an underflow:

.....

  before fixup:              rlen          1  args->len          0
  after xfs_alloc_fix_len  : rlen          1  args->len          1
  before goto out_nominleft: rlen          1  args->len          0
  before fixup:              rlen          1  args->len          0
  after xfs_alloc_fix_len  : rlen          1  args->len          1
  after fixup:               rlen          1  args->len          1
  before fixup:              rlen          1  args->len          0
  after xfs_alloc_fix_len  : rlen          1  args->len          1
  after fixup:               rlen 4294967295  args->len 4294967295

XFS: Assertion failed: fs_is_ok, file: fs/xfs/libxfs/xfs_alloc.c, line: 1424

The "goto out_nominleft:" indicates that we are getting close to ENOSPC in the AG, and a couple of allocations later we underflow and the corruption check fires in xfs_alloc_ag_vextent_size().

The issue is that the extent length fixups comaprisons are done with variables of xfs_extlen_t types. These are unsigned so an underflow looks like a really big value and hence is not detected as being smaller than the minimum length allowed for the extent. Hence the corruption check fires as it is noticing that the returned length is longer than the original extent length passed in.

This can be easily fixed by ensuring we do the underflow test on signed values, the same way xfs_alloc_fix_len() prevents underflow. So we realise in future that these casts prevent underflows from going undetected, add comments to the code indicating this.

Quoted 6 lines
---
  fs/xfs/libxfs/xfs_alloc.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

int diff = be32_to_cpu(agf->agf_freeblks)
- args->len - args->minleft;

Review xfs_alloc_fix_minleft( · line 288

Quoted 3 lines
(quoted diff) +0 -0
@@ -286,7 +287,8 @@ xfs_alloc_fix_minleft(
  	if (diff >= 0)
  		return 1;

If the diff math was done correctly, wouldn't it get caught here?

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 294

Quoted 7 lines
fs/xfs/libxfs/xfs_alloc.c +2 -1
  	args->len += diff;		/* shrink the allocated space */
-	if (args->len >= args->minlen)
+	/* casts to (int) catch length underflows */
+	if ((int)args->len >= (int)args->minlen)
  		return 1;
  	args->agbno = NULLAGBLOCK;
  	return 0;

We can and should fix the wrap in xfs_alloc_fix_minleft() but this also points to the fact that xfs_alloc_fix_freelist() is incorrectly choosing AGs that will later fail the allocation alignment, minlen, and minleft requirements.

You can connect the dots to see how this can lead to a deadlock with extent frees. We have seen them. I hacked the XFS code to lead to this situation.

Also bad is xfs_alloc_vextent() will temporally ignore the minleft for the xfs_alloc_fix_freelist() but makes the ag allocator enforce the minleft. I got this condition to ASSERT in xfs_alloc_ag_vextent_size(), but I never got a deadlock. It would require just the right sequence from xfs_alloc_vextent() and xfs_bmap_btalloc().

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 43 lines · Mark Tinguely
Quoted 34 lines · Dave Chinner

From: Dave Chinner<[email protected]>

Test generic/224 is failing with a corruption being detected on one of Michael's test boxes. Debug that Michael added is indicating that the minleft trimming is resulting in an underflow:

.....

 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 before goto out_nominleft: rlen          1  args->len          0
 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 after fixup:               rlen          1  args->len          1
 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 after fixup:               rlen 4294967295  args->len 4294967295

XFS: Assertion failed: fs_is_ok, file: fs/xfs/libxfs/xfs_alloc.c, line: 1424

The "goto out_nominleft:" indicates that we are getting close to ENOSPC in the AG, and a couple of allocations later we underflow and the corruption check fires in xfs_alloc_ag_vextent_size().

The issue is that the extent length fixups comaprisons are done with variables of xfs_extlen_t types. These are unsigned so an underflow looks like a really big value and hence is not detected as being smaller than the minimum length allowed for the extent. Hence the corruption check fires as it is noticing that the returned length is longer than the original extent length passed in.

This can be easily fixed by ensuring we do the underflow test on signed values, the same way xfs_alloc_fix_len() prevents underflow. So we realise in future that these casts prevent underflows from going undetected, add comments to the code indicating this.

Quoted 6 lines
---
 fs/xfs/libxfs/xfs_alloc.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 3 lines

int diff = be32_to_cpu(agf->agf_freeblks)
- args->len - args->minleft;

Preconditions:

	agf->agf_freeblks = 1
	args->len = 1
	args->minleft = 2

Therefore, diff = -2

Quoted 5 lines

Review xfs_alloc_fix_minleft( · line 288

Quoted 3 lines
(quoted diff) +0 -0
@@ -286,7 +287,8 @@ xfs_alloc_fix_minleft(
 	if (diff >= 0)
 		return 1;

If the diff math was done correctly, wouldn't it get caught here?

No, diff < 0.

Quoted 1 line

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 289

Quoted 1 line
fs/xfs/libxfs/xfs_alloc.c +0 -0
 	args->len += diff;		/* shrink the allocated space */

1 += -2
= -1

Quoted 1 line

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 289

Quoted 1 line
fs/xfs/libxfs/xfs_alloc.c +0 -1
-	if (args->len >= args->minlen)
	if (0xffffffff >= 1)

broken.

Quoted 2 lines

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 291

Quoted 2 lines
fs/xfs/libxfs/xfs_alloc.c +2 -0
+	/* casts to (int) catch length underflows */
+	if ((int)args->len >= (int)args->minlen)
	if (-1 >= 1)

correct.

Quoted 8 lines

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 294

Quoted 3 lines
fs/xfs/libxfs/xfs_alloc.c +0 -0
 		return 1;
 	args->agbno = NULLAGBLOCK;
 	return 0;

We can and should fix the wrap in xfs_alloc_fix_minleft() but this also points to the fact that xfs_alloc_fix_freelist() is incorrectly choosing AGs that will later fail the allocation alignment, minlen, and minleft requirements.

I don't think there's a problem there. At least, not the problem I think you trying to describe.

Quoted 3 lines

You can connect the dots to see how this can lead to a deadlock with extent frees. We have seen them. I hacked the XFS code to lead to this situation.

You should post the test cases and the patch that exposes the issues you are concerned about. Otherwise we have no real idea of what problems you are talking about, and certainly can't reproduce them.

Quoted 3 lines

Also bad is xfs_alloc_vextent() will temporally ignore the minleft for the xfs_alloc_fix_freelist() but makes the ag allocator enforce the minleft.

Hmmm - I suspect you haven't understood the underlying reason for setting minleft to zero for the call to xfs_alloc_fix_freelist(). There's are two places we do this: single AG constrainted allocation, and the "any AG, but near ENOSPC" allocation.

For single AG constrained allocation, minleft is something we can ignore because we must select that AG for allocation. minleft is applied against the total allocation requirement, not the minimum length of allocation that can be done. Hence we might be able to do a minlen allocation, but we'd reject it because we can't do a maxlen allocation due to minleft requirements.

Hence we switch off minleft in that case when doing freelist checks so that we don't reject allocations that could do a minlen allocation successfully. This requires the low level allocator to be able to trim back the selected extent from whatever length it finds to repesect minleft, and if it can't then fail the allocation. That is the function of xfs_alloc_fix_minleft() - constrain the allocated extent to the required minleft or fail the allocation.

And for the lowspace algorithm, the reasoning is similar. After two attempts to allocate in AGs that have enough space for a maxlen allocation, we switch off minleft so that we only constrain the AG search to AGs that can definitely satisfy a minlen allocation, hence improving the chance we can do an allocation successfully near ENOSPC.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 134 lines · Dave Chinner
Quoted 43 lines · Mark Tinguely
Quoted 34 lines · Dave Chinner

From: Dave Chinner<[email protected]>

Test generic/224 is failing with a corruption being detected on one of Michael's test boxes. Debug that Michael added is indicating that the minleft trimming is resulting in an underflow:

.....

 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 before goto out_nominleft: rlen          1  args->len          0
 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 after fixup:               rlen          1  args->len          1
 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 after fixup:               rlen 4294967295  args->len 4294967295

XFS: Assertion failed: fs_is_ok, file: fs/xfs/libxfs/xfs_alloc.c, line: 1424

The "goto out_nominleft:" indicates that we are getting close to ENOSPC in the AG, and a couple of allocations later we underflow and the corruption check fires in xfs_alloc_ag_vextent_size().

The issue is that the extent length fixups comaprisons are done with variables of xfs_extlen_t types. These are unsigned so an underflow looks like a really big value and hence is not detected as being smaller than the minimum length allowed for the extent. Hence the corruption check fires as it is noticing that the returned length is longer than the original extent length passed in.

This can be easily fixed by ensuring we do the underflow test on signed values, the same way xfs_alloc_fix_len() prevents underflow. So we realise in future that these casts prevent underflows from going undetected, add comments to the code indicating this.

Quoted 6 lines
---
 fs/xfs/libxfs/xfs_alloc.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 3 lines

int diff = be32_to_cpu(agf->agf_freeblks)
- args->len - args->minleft;

Preconditions:

	agf->agf_freeblks = 1
	args->len = 1
	args->minleft = 2

Therefore, diff = -2

Quoted 5 lines

Review xfs_alloc_fix_minleft( · line 288

Quoted 3 lines
(quoted diff) +0 -0
@@ -286,7 +287,8 @@ xfs_alloc_fix_minleft(
 	if (diff >= 0)
 		return 1;

If the diff math was done correctly, wouldn't it get caught here?

No, diff < 0.

Quoted 1 line

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 289

Quoted 1 line
fs/xfs/libxfs/xfs_alloc.c +0 -0
 	args->len += diff;		/* shrink the allocated space */

1 += -2
= -1

Quoted 1 line

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 289

Quoted 1 line
fs/xfs/libxfs/xfs_alloc.c +0 -1
-	if (args->len >= args->minlen)
	if (0xffffffff >= 1)

broken.

Quoted 2 lines

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 291

Quoted 2 lines
fs/xfs/libxfs/xfs_alloc.c +2 -0
+	/* casts to (int) catch length underflows */
+	if ((int)args->len >= (int)args->minlen)
	if (-1 >= 1)

correct.

Quoted 8 lines

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 294

Quoted 3 lines
fs/xfs/libxfs/xfs_alloc.c +0 -0
 		return 1;
 	args->agbno = NULLAGBLOCK;
 	return 0;

We can and should fix the wrap in xfs_alloc_fix_minleft() but this also points to the fact that xfs_alloc_fix_freelist() is incorrectly choosing AGs that will later fail the allocation alignment, minlen, and minleft requirements.

I don't think there's a problem there. At least, not the problem I think you trying to describe.

Quoted 3 lines

You can connect the dots to see how this can lead to a deadlock with extent frees. We have seen them. I hacked the XFS code to lead to this situation.

You should post the test cases and the patch that exposes the issues you are concerned about. Otherwise we have no real idea of what problems you are talking about, and certainly can't reproduce them.

Quoted 3 lines

Also bad is xfs_alloc_vextent() will temporally ignore the minleft for the xfs_alloc_fix_freelist() but makes the ag allocator enforce the minleft.

Hmmm - I suspect you haven't understood the underlying reason for setting minleft to zero for the call to xfs_alloc_fix_freelist(). There's are two places we do this: single AG constrainted allocation, and the "any AG, but near ENOSPC" allocation.

For single AG constrained allocation, minleft is something we can ignore because we must select that AG for allocation. minleft is applied against the total allocation requirement, not the minimum length of allocation that can be done. Hence we might be able to do a minlen allocation, but we'd reject it because we can't do a maxlen allocation due to minleft requirements.

Hence we switch off minleft in that case when doing freelist checks so that we don't reject allocations that could do a minlen allocation successfully. This requires the low level allocator to be able to trim back the selected extent from whatever length it finds to repesect minleft, and if it can't then fail the allocation. That is the function of xfs_alloc_fix_minleft() - constrain the allocated extent to the required minleft or fail the allocation.

And for the lowspace algorithm, the reasoning is similar. After two attempts to allocate in AGs that have enough space for a maxlen allocation, we switch off minleft so that we only constrain the AG search to AGs that can definitely satisfy a minlen allocation, hence improving the chance we can do an allocation successfully near ENOSPC.

Quoted 3 lines

Cheers,

Dave.

I took something out of Mark's argument. The original condition was hit using fstests generic/224 on RAID-0, while sizing up an old Core 2 PC for a new purpose. With Mark mentioning AGs, I looked around a little bit. The case can be hit more easily with this test on 16-GB $TEST_DEV and $SCRATCH_DEV:

while true; do MKFS_OPTIONS='-d agcount=16' ./check generic/224; done

That may not mean anything--a 50 MB/s Caviar SE on a SATA-1 controller may just make the original test too slow for my patience--but the extra AGs do make things seem rather repeatable.

I'll be happy to leave both the Core 2 PC and the i686 Pentium 4 PC in an unpatched state, should you or Mark want me to fetch more values via printk debugging. Everything else XFS is in pretty good shape for the "small or few" purposes here.

Thanks!

Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw
Quoted 57 lines · Michael L. Semon
Quoted 56 lines · Dave Chinner
Quoted 43 lines · Mark Tinguely
Quoted 34 lines · Dave Chinner

From: Dave Chinner<[email protected]>

Test generic/224 is failing with a corruption being detected on one of Michael's test boxes. Debug that Michael added is indicating that the minleft trimming is resulting in an underflow:

.....

  before fixup:              rlen          1  args->len          0
  after xfs_alloc_fix_len  : rlen          1  args->len          1
  before goto out_nominleft: rlen          1  args->len          0
  before fixup:              rlen          1  args->len          0
  after xfs_alloc_fix_len  : rlen          1  args->len          1
  after fixup:               rlen          1  args->len          1
  before fixup:              rlen          1  args->len          0
  after xfs_alloc_fix_len  : rlen          1  args->len          1
  after fixup:               rlen 4294967295  args->len 4294967295

XFS: Assertion failed: fs_is_ok, file: fs/xfs/libxfs/xfs_alloc.c, line: 1424

The "goto out_nominleft:" indicates that we are getting close to ENOSPC in the AG, and a couple of allocations later we underflow and the corruption check fires in xfs_alloc_ag_vextent_size().

The issue is that the extent length fixups comaprisons are done with variables of xfs_extlen_t types. These are unsigned so an underflow looks like a really big value and hence is not detected as being smaller than the minimum length allowed for the extent. Hence the corruption check fires as it is noticing that the returned length is longer than the original extent length passed in.

This can be easily fixed by ensuring we do the underflow test on signed values, the same way xfs_alloc_fix_len() prevents underflow. So we realise in future that these casts prevent underflows from going undetected, add comments to the code indicating this.

Quoted 6 lines
---
  fs/xfs/libxfs/xfs_alloc.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 3 lines

int diff = be32_to_cpu(agf->agf_freeblks)
- args->len - args->minleft;

Preconditions:

	agf->agf_freeblks = 1
	args->len = 1
	args->minleft = 2

Therefore, diff = -2

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Review xfs_alloc_fix_minleft( · line 288

Quoted 3 lines
(quoted diff) +0 -0
@@ -286,7 +287,8 @@ xfs_alloc_fix_minleft(
  	if (diff>= 0)
  		return 1;

If the diff math was done correctly, wouldn't it get caught here?

No, diff< 0.

Hee, what was I thinking? sdrawkcab ylsouivbO

Quoted 87 lines
Quoted 71 lines
Quoted 3 lines

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 289

Quoted 3 lines
fs/xfs/libxfs/xfs_alloc.c +0 -0
  	args->len += diff;		/* shrink the allocated space */

1 += -2
= -1

Quoted 1 line

Review xfs_alloc_fix_minleft( · line 289

Quoted 1 line
(quoted diff) +0 -1
-	if (args->len>= args->minlen)
	if (0xffffffff>= 1)

broken.

Quoted 2 lines

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 291

Quoted 2 lines
fs/xfs/libxfs/xfs_alloc.c +2 -0
+	/* casts to (int) catch length underflows */
+	if ((int)args->len >= (int)args->minlen)
	if (-1>= 1)

correct.

Quoted 8 lines

fs/xfs/libxfs/xfs_alloc.c xfs_alloc_fix_minleft( · line 294

Quoted 3 lines
fs/xfs/libxfs/xfs_alloc.c +0 -0
  		return 1;
  	args->agbno = NULLAGBLOCK;
  	return 0;

We can and should fix the wrap in xfs_alloc_fix_minleft() but this also points to the fact that xfs_alloc_fix_freelist() is incorrectly choosing AGs that will later fail the allocation alignment, minlen, and minleft requirements.

I don't think there's a problem there. At least, not the problem I think you trying to describe.

Quoted 3 lines

You can connect the dots to see how this can lead to a deadlock with extent frees. We have seen them. I hacked the XFS code to lead to this situation.

You should post the test cases and the patch that exposes the issues you are concerned about. Otherwise we have no real idea of what problems you are talking about, and certainly can't reproduce them.

Quoted 3 lines

Also bad is xfs_alloc_vextent() will temporally ignore the minleft for the xfs_alloc_fix_freelist() but makes the ag allocator enforce the minleft.

Hmmm - I suspect you haven't understood the underlying reason for setting minleft to zero for the call to xfs_alloc_fix_freelist(). There's are two places we do this: single AG constrainted allocation, and the "any AG, but near ENOSPC" allocation.

For single AG constrained allocation, minleft is something we can ignore because we must select that AG for allocation. minleft is applied against the total allocation requirement, not the minimum length of allocation that can be done. Hence we might be able to do a minlen allocation, but we'd reject it because we can't do a maxlen allocation due to minleft requirements.

Hence we switch off minleft in that case when doing freelist checks so that we don't reject allocations that could do a minlen allocation successfully. This requires the low level allocator to be able to trim back the selected extent from whatever length it finds to repesect minleft, and if it can't then fail the allocation. That is the function of xfs_alloc_fix_minleft() - constrain the allocated extent to the required minleft or fail the allocation.

And for the lowspace algorithm, the reasoning is similar. After two attempts to allocate in AGs that have enough space for a maxlen allocation, we switch off minleft so that we only constrain the AG search to AGs that can definitely satisfy a minlen allocation, hence improving the chance we can do an allocation successfully near ENOSPC.

Quoted 3 lines

Cheers,

Dave.

I took something out of Mark's argument. The original condition was hit using fstests generic/224 on RAID-0, while sizing up an old Core 2 PC for a new purpose. With Mark mentioning AGs, I looked around a little bit. The case can be hit more easily with this test on 16-GB $TEST_DEV and $SCRATCH_DEV:

while true; do MKFS_OPTIONS='-d agcount=16' ./check generic/224; done

That may not mean anything--a 50 MB/s Caviar SE on a SATA-1 controller may just make the original test too slow for my patience--but the extra AGs do make things seem rather repeatable.

Quoted 8 lines

I'll be happy to leave both the Core 2 PC and the i686 Pentium 4 PC in an unpatched state, should you or Mark want me to fetch more values via printk debugging. Everything else XFS is in pretty good shape for the "small or few" purposes here.

Thanks!

Michael

Thanks Michael, you don't need to hold your test box for me. I do have a way to recreate these ABBA AGF buffer allocation deadlocks and understand the whys and hows very well. I don't have a community way to make a xfstest for it but I think your test is getting close.

I am certain that with Dave's patch alone, and pushed long enough you will hit the ABBA AGF buffer deadlock. It needs to do that allocation to the last AG while freeing an extent to that AG.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 4 lines · Mark Tinguely

Thanks Michael, you don't need to hold your test box for me. I do have a way to recreate these ABBA AGF buffer allocation deadlocks and understand the whys and hows very well. I don't have a community way to make a xfstest for it but I think your test is getting close.

If you know what is causing them, then please explain how it occurs and how you think it needs to be fixed. Just telling us that you know something that we don't doesn't help us solve the problem. :(

In general, the use of the args->firstblock is supposed to avoid the ABBA locking order issues with multiple allocations in the one transaction by preventing AG selection loops from looping back into AGs with a lower index than the first allocation that was made.

So if you are seeing deadlocks, then it may be that we aren't following this constraint correctly in all locations....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#6 lore raw
Quoted 15 lines · Dave Chinner
Quoted 4 lines · Mark Tinguely

Thanks Michael, you don't need to hold your test box for me. I do have a way to recreate these ABBA AGF buffer allocation deadlocks and understand the whys and hows very well. I don't have a community way to make a xfstest for it but I think your test is getting close.

If you know what is causing them, then please explain how it occurs and how you think it needs to be fixed. Just telling us that you know something that we don't doesn't help us solve the problem. :(

In general, the use of the args->firstblock is supposed to avoid the ABBA locking order issues with multiple allocations in the one transaction by preventing AG selection loops from looping back into AGs with a lower index than the first allocation that was made.

Quoted 6 lines

So if you are seeing deadlocks, then it may be that we aren't following this constraint correctly in all locations....

Cheers,

Dave.

Will this be a classic deadlock that will cause problems when trying to kill processes and unmount filesystems? If so, then I was unable to use generic/224 to trigger a deadlock. If not, then I'll need a better way of looking at the problem.

The longest generic/224 loop lasted only 3-1/2 hours, though. The fstests enospc group was given some consideration as well.

If this issue does not require a lot of files, I might see if fio can be helpful here.

Hints on whether to us a fast kernel or a miserably slow kernel would be rather helpful.

My test setup is torn because most of the recent warning messages are coming from the CONFIG_XFS_WARN kernels. The i686 Pentium 4 box will be left that way. However, the Core 2 box was configured per Documentation/SubmitChecklist from the kernel source, adding debug XFS and locktorture. The locktorture settings are in flux, exercising spinlocks at present. There was a mild halt in I/O for generic/017, but that was XFS waiting on kmem-something waiting on a kmemleak function. kmemleak was removed, and I'll continue from there.

Thanks!

Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
Quoted 16 lines · Michael L. Semon
Quoted 15 lines · Dave Chinner
Quoted 4 lines · Mark Tinguely

Thanks Michael, you don't need to hold your test box for me. I do have a way to recreate these ABBA AGF buffer allocation deadlocks and understand the whys and hows very well. I don't have a community way to make a xfstest for it but I think your test is getting close.

If you know what is causing them, then please explain how it occurs and how you think it needs to be fixed. Just telling us that you know something that we don't doesn't help us solve the problem. :(

In general, the use of the args->firstblock is supposed to avoid the ABBA locking order issues with multiple allocations in the one transaction by preventing AG selection loops from looping back into AGs with a lower index than the first allocation that was made.

Quoted 7 lines
Quoted 2 lines

So if you are seeing deadlocks, then it may be that we aren't following this constraint correctly in all locations....

Will this be a classic deadlock that will cause problems when trying to kill processes and unmount filesystems? If so, then I was unable to use generic/224 to trigger a deadlock. If not, then I'll need a better way of looking at the problem.

Yes, it will hang the filesystem.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw
Quoted 25 lines · Dave Chinner
Quoted 16 lines · Michael L. Semon
Quoted 15 lines · Dave Chinner
Quoted 4 lines · Mark Tinguely

Thanks Michael, you don't need to hold your test box for me. I do have a way to recreate these ABBA AGF buffer allocation deadlocks and understand the whys and hows very well. I don't have a community way to make a xfstest for it but I think your test is getting close.

If you know what is causing them, then please explain how it occurs and how you think it needs to be fixed. Just telling us that you know something that we don't doesn't help us solve the problem. :(

In general, the use of the args->firstblock is supposed to avoid the ABBA locking order issues with multiple allocations in the one transaction by preventing AG selection loops from looping back into AGs with a lower index than the first allocation that was made.

Quoted 7 lines
Quoted 2 lines

So if you are seeing deadlocks, then it may be that we aren't following this constraint correctly in all locations....

Will this be a classic deadlock that will cause problems when trying to kill processes and unmount filesystems? If so, then I was unable to use generic/224 to trigger a deadlock. If not, then I'll need a better way of looking at the problem.

Quoted 5 lines

Yes, it will hang the filesystem.

Cheers,

Dave.

Thanks. I'll try again tonight.

Last night's attempt was a combination of fio, fsstress, and a shell loop of xfs_io's fcollapse command, all at once on an SSD. At the end of the night, XFS was laughing at me. Therefore, I added the same test on the 3-partition RAID-0 side. This morning, XFS is still laughing at me, but the RAID-0 test is still running.

Thanks!

Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by on v1 1/1
#9 lore raw
In reply to Dave Chinner
Quoted 34 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Test generic/224 is failing with a corruption being detected on one of Michael's test boxes. Debug that Michael added is indicating that the minleft trimming is resulting in an underflow:

.....

 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 before goto out_nominleft: rlen          1  args->len          0
 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 after fixup:               rlen          1  args->len          1
 before fixup:              rlen          1  args->len          0
 after xfs_alloc_fix_len  : rlen          1  args->len          1
 after fixup:               rlen 4294967295  args->len 4294967295

XFS: Assertion failed: fs_is_ok, file: fs/xfs/libxfs/xfs_alloc.c, line: 1424

The "goto out_nominleft:" indicates that we are getting close to ENOSPC in the AG, and a couple of allocations later we underflow and the corruption check fires in xfs_alloc_ag_vextent_size().

The issue is that the extent length fixups comaprisons are done with variables of xfs_extlen_t types. These are unsigned so an underflow looks like a really big value and hence is not detected as being smaller than the minimum length allowed for the extent. Hence the corruption check fires as it is noticing that the returned length is longer than the original extent length passed in.

This can be easily fixed by ensuring we do the underflow test on signed values, the same way xfs_alloc_fix_len() prevents underflow. So we realise in future that these casts prevent underflows from going undetected, add comments to the code indicating this.

Quoted 4 lines

---

Looks fine to me:

Quoted 32 lines
 fs/xfs/libxfs/xfs_alloc.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_alloc.c +3 -1
index 710554c..41e3630 100644
@@ -260,6 +260,7 @@ xfs_alloc_fix_len(
 		rlen = rlen - (k - args->mod);
 	else
 		rlen = rlen - args->prod + (args->mod - k);
+	/* casts to (int) catch length underflows */
 	if ((int)rlen < (int)args->minlen)
 		return;
 	ASSERT(rlen >= args->minlen && rlen <= args->maxlen);
@@ -286,7 +287,8 @@ xfs_alloc_fix_minleft(
 	if (diff >= 0)
 		return 1;
 	args->len += diff;		/* shrink the allocated space */
-	if (args->len >= args->minlen)
+	/* casts to (int) catch length underflows */
+	if ((int)args->len >= (int)args->minlen)
 		return 1;
 	args->agbno = NULLAGBLOCK;
 	return 0;
Signature · 6 lines
2.0.0

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs