Skip to contentSkip to conversation

xfsdump SGI_FS_BULKSTAT errno = 22, how could this IRIX bug get into Ubuntu 10.04 Lucid between kernels 2.6.32-27 and 2.6.32-26?

DISCUSSION Michael Lueck 4 participants 23 messages Feb 2, 2011 – Apr 22, 2011

lore
Share & tools
lore
#0 lore raw

Greetings,

Somehow a reported IRIX bug with XFS got into Ubuntu 10.04 (Lucid) starting with kernel 2.6.32-27.

I am cross posting to this list in order to receive details as to what bug got in Ubuntu which has been solved in IRIX, hoping with more details Ubuntu might make the same fix.

Ubuntu has since updated their kernel to 2.6.32-28 and someone already verified at the bug report that the problem persists with that kernel version.

"Regression between 2.6.32-27 and 2.6.32-26 xfsdump SGI_FS_BULKSTAT errno = 22"
https://bugs.launchpad.net/bugs/692848

On our Ubuntu 10.04 LTS server running x86 code, this evening a kernel update was ready for installation. I updated the kernel, rebooted (IPL'ed), and proceeded with the backup which utilized xfsdump as we use the xfs filesystem. Four of the xfsdump received a never before seen
error: SGI_FS_BULKSTAT errno = 22

Output as follows:
using file dump (drive_simple) strategy
version 3.0.4 (dump format 3.0) - Running single-threaded
level 0 dump of ldslnx01:/srv
dump date: Mon Dec 20 21:59:33 2010
session id: f98f8cc0-963f-41a6-9a19-a89192502bf0
session label: "data"
ino map phase 1: constructing initial dump list
ino map phase 2: skipping (no pruning necessary)
ino map phase 3: skipping (only one dump stream)
ino map construction complete
estimated dump size: 100739707392 bytes
WARNING: no media label specified
creating dump session media file 0 (media 0, file 0)
dumping ino map
dumping directories
SGI_FS_BULKSTAT failed: Invalid argument (22)
dump size (non-dir files) : 0 bytes
NOTE: dump interrupted: 79 seconds elapsed: may resume later using -R option Dump Status: INTERRUPT

This backup file did have some size to it. The other three, backing up smaller amounts of data, were all zero (0) length dump files.

I rebooted to the prior kernel: $ uname -a
Linux ldslnx01 2.6.32-26-generic-pae #48-Ubuntu SMP Wed Nov 24 10:31:20 UTC 2010 i686 GNU/Linux

And the same backup gets to 100% success.

Reboot to the new kernel, same failure.

I think that fairly well illustrates that the problem exists only with the kernel update installed this evening.

<><><><>

I did come across a reference to this problem on the SGI website:

http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=relnotes&fname=/usr/relnotes/eoe
1.6.17 Bugs fixed in IRIX 6.5.13
+ 816457: xfsdump SGI_FS_BULKSTAT errno = 22 cxfs

So evidently it is something that has been seen and corrected in IRIX.

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw
Quoted 6 lines · Michael Lueck

Greetings,

Somehow a reported IRIX bug with XFS got into Ubuntu 10.04 (Lucid) starting with kernel 2.6.32-27.

I am cross posting to this list in order to receive details as to what bug got in Ubuntu which has been solved in IRIX, hoping with more details Ubuntu might make the same fix.

Aside from the fact that the errno is the same, there's nothing to suggest that the Ubuntu problem is the the same bug. The IRIX bug is quite old.

Quoted 6 lines

Ubuntu has since updated their kernel to 2.6.32-28 and someone already verified at the bug report that the problem persists with that kernel version.

"Regression between 2.6.32-27 and 2.6.32-26 xfsdump SGI_FS_BULKSTAT errno = 22"
https://bugs.launchpad.net/bugs/692848

Between 2.6.32-26 and 2.6.32-27, Ubuntu backported 4 XFS commits from 2.6.35/2.6.36. All are part of a bulkstat security fix.

% git log Ubuntu-2.6.32-26.48..Ubuntu-2.6.32-27.49 -- fs/xfs | grep commit commit 52d2a4cfbc852da8c3d3b9fa0cac2a07b12f5cfd

    (cherry picked from commit 4536f2ad8b330453d7ebec0746c4374eadd649b1)
commit eb5ab28c8a5e4bb3f1ce05eba166c12175f6c701
    (backported from commit 7b6259e7a83647948fa33a736cc832310c8d85aa)
commit 5f8e8c6ab416bbd58d4f5df512c119a888ff923c
    (cherry picked from commit 1920779e67cbf5ea8afef317777c5bf2b8096188)
commit 52e0d703745f7110f1ecbe83c02cf06a83da82e8
    (backported from commit 7124fe0a5b619d65b739477b3b55a20bf805b06d)

I'm not aware of a similar problem upstream, so it would appear to be a problem with Ubuntu's backport of these commits.

Bill

Quoted 43 lines

On our Ubuntu 10.04 LTS server running x86 code, this evening a kernel update was ready for installation. I updated the kernel, rebooted (IPL'ed), and proceeded with the backup which utilized xfsdump as we use the xfs filesystem. Four of the xfsdump received a never before seen
error: SGI_FS_BULKSTAT errno = 22

Output as follows:
using file dump (drive_simple) strategy
version 3.0.4 (dump format 3.0) - Running single-threaded
level 0 dump of ldslnx01:/srv
dump date: Mon Dec 20 21:59:33 2010
session id: f98f8cc0-963f-41a6-9a19-a89192502bf0
session label: "data"
ino map phase 1: constructing initial dump list
ino map phase 2: skipping (no pruning necessary)
ino map phase 3: skipping (only one dump stream)
ino map construction complete
estimated dump size: 100739707392 bytes
WARNING: no media label specified
creating dump session media file 0 (media 0, file 0)
dumping ino map
dumping directories
SGI_FS_BULKSTAT failed: Invalid argument (22)
dump size (non-dir files) : 0 bytes
NOTE: dump interrupted: 79 seconds elapsed: may resume later using -R option Dump Status: INTERRUPT

This backup file did have some size to it. The other three, backing up smaller amounts of data, were all zero (0) length dump files.

I rebooted to the prior kernel: $ uname -a
Linux ldslnx01 2.6.32-26-generic-pae #48-Ubuntu SMP Wed Nov 24 10:31:20 UTC 2010 i686 GNU/Linux

And the same backup gets to 100% success.

Reboot to the new kernel, same failure.

I think that fairly well illustrates that the problem exists only with the kernel update installed this evening.

<><><><>

Quoted 8 lines

I did come across a reference to this problem on the SGI website:

http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=relnotes&fname=/usr/relnotes/eoe
1.6.17 Bugs fixed in IRIX 6.5.13
+ 816457: xfsdump SGI_FS_BULKSTAT errno = 22 cxfs

So evidently it is something that has been seen and corrected in IRIX.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 2 lines · Bill Kendall

I'm not aware of a similar problem upstream, so it would appear to be a problem with Ubuntu's backport of these commits.

Thank you SO much, Bill! :-) I will post your comments back to the Ubuntu bug report.

Sincerely,

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
In reply to Bill Kendall
Quoted 31 lines · Bill Kendall
Quoted 6 lines · Michael Lueck

Greetings,

Somehow a reported IRIX bug with XFS got into Ubuntu 10.04 (Lucid) starting with kernel 2.6.32-27.

I am cross posting to this list in order to receive details as to what bug got in Ubuntu which has been solved in IRIX, hoping with more details Ubuntu might make the same fix.

Aside from the fact that the errno is the same, there's nothing to suggest that the Ubuntu problem is the the same bug. The IRIX bug is quite old.

Quoted 6 lines

Ubuntu has since updated their kernel to 2.6.32-28 and someone already verified at the bug report that the problem persists with that kernel version.

"Regression between 2.6.32-27 and 2.6.32-26 xfsdump SGI_FS_BULKSTAT errno = 22"
https://bugs.launchpad.net/bugs/692848

Between 2.6.32-26 and 2.6.32-27, Ubuntu backported 4 XFS commits from 2.6.35/2.6.36. All are part of a bulkstat security fix.

% git log Ubuntu-2.6.32-26.48..Ubuntu-2.6.32-27.49 -- fs/xfs | grep commit commit 52d2a4cfbc852da8c3d3b9fa0cac2a07b12f5cfd

    (cherry picked from commit 4536f2ad8b330453d7ebec0746c4374eadd649b1)
commit eb5ab28c8a5e4bb3f1ce05eba166c12175f6c701
    (backported from commit 7b6259e7a83647948fa33a736cc832310c8d85aa)
commit 5f8e8c6ab416bbd58d4f5df512c119a888ff923c
    (cherry picked from commit 1920779e67cbf5ea8afef317777c5bf2b8096188)
commit 52e0d703745f7110f1ecbe83c02cf06a83da82e8
    (backported from commit 7124fe0a5b619d65b739477b3b55a20bf805b06d)
Quoted 2 lines

I'm not aware of a similar problem upstream, so it would appear to be a problem with Ubuntu's backport of these commits.

<sigh>

That'll be the untrusted inode lookup fixes.

Quoted 30 lines

Bill

Quoted 27 lines

On our Ubuntu 10.04 LTS server running x86 code, this evening a kernel update was ready for installation. I updated the kernel, rebooted (IPL'ed), and proceeded with the backup which utilized xfsdump as we use the xfs filesystem. Four of the xfsdump received a never before seen
error: SGI_FS_BULKSTAT errno = 22

Output as follows:
using file dump (drive_simple) strategy
version 3.0.4 (dump format 3.0) - Running single-threaded
level 0 dump of ldslnx01:/srv
dump date: Mon Dec 20 21:59:33 2010
session id: f98f8cc0-963f-41a6-9a19-a89192502bf0
session label: "data"
ino map phase 1: constructing initial dump list
ino map phase 2: skipping (no pruning necessary)
ino map phase 3: skipping (only one dump stream)
ino map construction complete
estimated dump size: 100739707392 bytes
WARNING: no media label specified
creating dump session media file 0 (media 0, file 0)
dumping ino map
dumping directories
SGI_FS_BULKSTAT failed: Invalid argument (22)
dump size (non-dir files) : 0 bytes
NOTE: dump interrupted: 79 seconds elapsed: may resume later using -R option Dump Status: INTERRUPT

So bulkstat got EINVAL returned for and inode that it was looking up. That implies that it was racing with an unlink, which is what the above commits catch and prevent. Can you run xfsdump with full debug output (-v 5) so we can see what inode is being operated on when this failure occurs?

Dann, I'm not sure whether this means there was a bug in your backport or whether it's just xfsdump not handling a failure gracefully....

Quoted 15 lines
Quoted 15 lines

This backup file did have some size to it. The other three, backing up smaller amounts of data, were all zero (0) length dump files.

I rebooted to the prior kernel: $ uname -a
Linux ldslnx01 2.6.32-26-generic-pae #48-Ubuntu SMP Wed Nov 24 10:31:20 UTC 2010 i686 GNU/Linux

And the same backup gets to 100% success.

Reboot to the new kernel, same failure.

I think that fairly well illustrates that the problem exists only with the kernel update installed this evening.

<><><><>

Quoted 7 lines
Quoted 7 lines

I did come across a reference to this problem on the SGI website:

http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=relnotes&fname=/usr/relnotes/eoe
1.6.17 Bugs fixed in IRIX 6.5.13
+ 816457: xfsdump SGI_FS_BULKSTAT errno = 22 cxfs

So evidently it is something that has been seen and corrected in IRIX.

Oh, the curse of Google.

Irix 6.5.13 was released in 2001, so I don't think this is at all relevant for a regression reported for the latest and greatest Ubuntu kernel....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
Quoted 5 lines · Dave Chinner

So bulkstat got EINVAL returned for and inode that it was looking up. That implies that it was racing with an unlink, which is what the above commits catch and prevent. Can you run xfsdump with full debug output (-v 5) so we can see what inode is being operated on when this failure occurs?

Thank you so much Dave!

Please find the trace output here in zipped format:

http://www.lueckdatasystems.com/pub/ldsbackup.trace.log.zip

I chose to trace the "data" backup that actually had some size to it.

Sincerely,

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw
Quoted 7 lines · Michael Lueck
Quoted 5 lines · Dave Chinner

So bulkstat got EINVAL returned for and inode that it was looking up. That implies that it was racing with an unlink, which is what the above commits catch and prevent. Can you run xfsdump with full debug output (-v 5) so we can see what inode is being operated on when this failure occurs?

Quoted 5 lines

Thank you so much Dave!

Please find the trace output here in zipped format:

http://www.lueckdatasystems.com/pub/ldsbackup.trace.log.zip

Ok, so xfsdump i seeing a short bulkstat, then an EINVAL returned from the next bulkstat. That's not a race condition, and makes me think you have some kind of on-disk corruption. The inode it is starting at when it returns EINVAL is 80508397. Can you firstly post the output of:

# xfs_db -c "inode 80508397" -c p <dev>

And can you also run 'xfs_repair -n <dev>' on the filesystem and post the output as well so we can see what state the filesytem is in?

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#6 lore raw
Quoted 3 lines · Dave Chinner

Ok, so xfsdump i seeing a short bulkstat, then an EINVAL returned from the next bulkstat. That's not a race condition, and makes me think you have some kind of on-disk corruption.

Very odd that some kind of on-disk corruption is suddenly causing xfsdump problems starting with Ubuntu 10.04 (Lucid) kernel 2.6.32-27 and persisting in 2.6.32-28.

And there is one other person who confirmed this xfsdump problem running Lucid with kernel 2.6.32-28. They reported their "me too" in the Ubuntu bug tracker.

Could it be that 2.6.32-26 and prior managed to write something to disk corrupted, and the newer code is tripping on it?

I shall reboot the server to the 2.6.32-28 kernel and perform the tests you requested.

Sincerely,

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw
Quoted 8 lines · Michael Lueck
Quoted 3 lines · Dave Chinner

Ok, so xfsdump i seeing a short bulkstat, then an EINVAL returned from the next bulkstat. That's not a race condition, and makes me think you have some kind of on-disk corruption.

Very odd that some kind of on-disk corruption is suddenly causing xfsdump problems starting with Ubuntu 10.04 (Lucid) kernel 2.6.32-27 and persisting in 2.6.32-28.

Not really. The newer kernels have code in them that does more validity checks than previous kernels, so older kernels would have erroneously and silently returned unlinked files to xfsdump and have them backed up. IOWs, you'd never notice such a corruption with xfsdump. On the new kernel, xfsdump gets an EINVAL error to such occurrences, which it should have in the first place.

Quoted 6 lines

And there is one other person who confirmed this xfsdump problem running Lucid with kernel 2.6.32-28. They reported their "me too" in the Ubuntu bug tracker.

Could it be that 2.6.32-26 and prior managed to write something to disk corrupted, and the newer code is tripping on it?

That's what I'm trying to find out. Or it could be something as simple as your disk has had an undetected bit error that has flipped a bit in the inode allocation btree.

Quoted 2 lines

I shall reboot the server to the 2.6.32-28 kernel and perform the tests you requested.

No need to change kernels to run xfs_repair....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw
Quoted 21 lines · Dave Chinner
Quoted 8 lines · Michael Lueck
Quoted 3 lines · Dave Chinner

Ok, so xfsdump i seeing a short bulkstat, then an EINVAL returned from the next bulkstat. That's not a race condition, and makes me think you have some kind of on-disk corruption.

Very odd that some kind of on-disk corruption is suddenly causing xfsdump problems starting with Ubuntu 10.04 (Lucid) kernel 2.6.32-27 and persisting in 2.6.32-28.

Not really. The newer kernels have code in them that does more validity checks than previous kernels, so older kernels would have erroneously and silently returned unlinked files to xfsdump and have them backed up. IOWs, you'd never notice such a corruption with xfsdump. On the new kernel, xfsdump gets an EINVAL error to such occurrences, which it should have in the first place.

Quoted 4 lines

And there is one other person who confirmed this xfsdump problem running Lucid with kernel 2.6.32-28. They reported their "me too" in the Ubuntu bug tracker.

Quoted 7 lines
Quoted 2 lines

Could it be that 2.6.32-26 and prior managed to write something to disk corrupted, and the newer code is tripping on it?

That's what I'm trying to find out. Or it could be something as simple as your disk has had an undetected bit error that has flipped a bit in the inode allocation btree.

Hi Dave,

I am able to reproduce this on a system running Ubuntu 10.4 (2.6.32-28). I took a metadump of the filesystem and moved it to a system running 10.10 (2.6.35-25), and was able to successfully dump it there. Likewise it dumps fine on 2.6.38-rc1. So this suggests an issue with the Ubuntu 10.4 kernel.

Bill

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#9 lore raw
Quoted 30 lines · Bill Kendall
Quoted 21 lines · Dave Chinner
Quoted 8 lines · Michael Lueck
Quoted 3 lines · Dave Chinner

Ok, so xfsdump i seeing a short bulkstat, then an EINVAL returned from the next bulkstat. That's not a race condition, and makes me think you have some kind of on-disk corruption.

Very odd that some kind of on-disk corruption is suddenly causing xfsdump problems starting with Ubuntu 10.04 (Lucid) kernel 2.6.32-27 and persisting in 2.6.32-28.

Not really. The newer kernels have code in them that does more validity checks than previous kernels, so older kernels would have erroneously and silently returned unlinked files to xfsdump and have them backed up. IOWs, you'd never notice such a corruption with xfsdump. On the new kernel, xfsdump gets an EINVAL error to such occurrences, which it should have in the first place.

Quoted 4 lines

And there is one other person who confirmed this xfsdump problem running Lucid with kernel 2.6.32-28. They reported their "me too" in the Ubuntu bug tracker.

Quoted 7 lines
Quoted 2 lines

Could it be that 2.6.32-26 and prior managed to write something to disk corrupted, and the newer code is tripping on it?

That's what I'm trying to find out. Or it could be something as simple as your disk has had an undetected bit error that has flipped a bit in the inode allocation btree.

Quoted 7 lines

Hi Dave,

I am able to reproduce this on a system running Ubuntu 10.4 (2.6.32-28). I took a metadump of the filesystem and moved it to a system running 10.10 (2.6.35-25), and was able to successfully dump it there. Likewise it dumps fine on 2.6.38-rc1. So this suggests an issue with the Ubuntu 10.4 kernel.

2.6.35 hasn't had the untrusted inode lookup patches back ported to it, so it's no surprise that it isn't having problems - it's just like the older 2.6.32 kernels.

Hmmm, can you find out if there is any specific pattern to the inode numbers that are returning EINVAL? Maybe the inode allocbt freespace record checks aren't quite correct in the backport (like the original bogus alignment assumption I made).

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#10 lore raw
Quoted 39 lines · Dave Chinner
Quoted 30 lines · Bill Kendall
Quoted 21 lines · Dave Chinner
Quoted 8 lines · Michael Lueck
Quoted 3 lines · Dave Chinner

Ok, so xfsdump i seeing a short bulkstat, then an EINVAL returned from the next bulkstat. That's not a race condition, and makes me think you have some kind of on-disk corruption.

Very odd that some kind of on-disk corruption is suddenly causing xfsdump problems starting with Ubuntu 10.04 (Lucid) kernel 2.6.32-27 and persisting in 2.6.32-28.

Not really. The newer kernels have code in them that does more validity checks than previous kernels, so older kernels would have erroneously and silently returned unlinked files to xfsdump and have them backed up. IOWs, you'd never notice such a corruption with xfsdump. On the new kernel, xfsdump gets an EINVAL error to such occurrences, which it should have in the first place.

Quoted 4 lines

And there is one other person who confirmed this xfsdump problem running Lucid with kernel 2.6.32-28. They reported their "me too" in the Ubuntu bug tracker.

Quoted 7 lines
Quoted 2 lines

Could it be that 2.6.32-26 and prior managed to write something to disk corrupted, and the newer code is tripping on it?

That's what I'm trying to find out. Or it could be something as simple as your disk has had an undetected bit error that has flipped a bit in the inode allocation btree.

Quoted 7 lines

Hi Dave,

I am able to reproduce this on a system running Ubuntu 10.4 (2.6.32-28). I took a metadump of the filesystem and moved it to a system running 10.10 (2.6.35-25), and was able to successfully dump it there. Likewise it dumps fine on 2.6.38-rc1. So this suggests an issue with the Ubuntu 10.4 kernel.

Quoted 3 lines

2.6.35 hasn't had the untrusted inode lookup patches back ported to it, so it's no surprise that it isn't having problems - it's just like the older 2.6.32 kernels.

I thought it landed in 2.6.35 and then a regression was fixed in 2.6.36. The untrusted inode lookup changes are referenced here:
http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.35

Quoted 5 lines

Hmmm, can you find out if there is any specific pattern to the inode numbers that are returning EINVAL? Maybe the inode allocbt freespace record checks aren't quite correct in the backport (like the original bogus alignment assumption I made).

I'll take a look.

Bill

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#11 lore raw
Quoted 40 lines · Bill Kendall
Quoted 39 lines · Dave Chinner
Quoted 37 lines · Bill Kendall
Quoted 21 lines · Dave Chinner
Quoted 8 lines · Michael Lueck
Quoted 3 lines · Dave Chinner

Ok, so xfsdump i seeing a short bulkstat, then an EINVAL returned from the next bulkstat. That's not a race condition, and makes me think you have some kind of on-disk corruption.

Very odd that some kind of on-disk corruption is suddenly causing xfsdump problems starting with Ubuntu 10.04 (Lucid) kernel 2.6.32-27 and persisting in 2.6.32-28.

Not really. The newer kernels have code in them that does more validity checks than previous kernels, so older kernels would have erroneously and silently returned unlinked files to xfsdump and have them backed up. IOWs, you'd never notice such a corruption with xfsdump. On the new kernel, xfsdump gets an EINVAL error to such occurrences, which it should have in the first place.

Quoted 4 lines

And there is one other person who confirmed this xfsdump problem running Lucid with kernel 2.6.32-28. They reported their "me too" in the Ubuntu bug tracker.

Quoted 7 lines
Quoted 2 lines

Could it be that 2.6.32-26 and prior managed to write something to disk corrupted, and the newer code is tripping on it?

That's what I'm trying to find out. Or it could be something as simple as your disk has had an undetected bit error that has flipped a bit in the inode allocation btree.

Hi Dave,

I am able to reproduce this on a system running Ubuntu 10.4 (2.6.32-28). I took a metadump of the filesystem and moved it to a system running 10.10 (2.6.35-25), and was able to successfully dump it there. Likewise it dumps fine on 2.6.38-rc1. So this suggests an issue with the Ubuntu 10.4 kernel.

Quoted 7 lines
Quoted 3 lines

2.6.35 hasn't had the untrusted inode lookup patches back ported to it, so it's no surprise that it isn't having problems - it's just like the older 2.6.32 kernels.

I thought it landed in 2.6.35 and then a regression was fixed in 2.6.36. The untrusted inode lookup changes are referenced here:
http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.35

My bad, I just checked the regression fix. I have no idea if it got back ported to 2.6.35-stable or not - it probably didn't judging by your results.....

Quoted 6 lines
Quoted 4 lines

Hmmm, can you find out if there is any specific pattern to the inode numbers that are returning EINVAL? Maybe the inode allocbt freespace record checks aren't quite correct in the backport (like the original bogus alignment assumption I made).

I'll take a look.

Thanks.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#12 lore raw
Quoted 45 lines · Dave Chinner
Quoted 44 lines · Bill Kendall
Quoted 42 lines · Dave Chinner
Quoted 30 lines · Bill Kendall
Quoted 21 lines · Dave Chinner
Quoted 8 lines · Michael Lueck
Quoted 3 lines · Dave Chinner

Ok, so xfsdump i seeing a short bulkstat, then an EINVAL returned from the next bulkstat. That's not a race condition, and makes me think you have some kind of on-disk corruption.

Very odd that some kind of on-disk corruption is suddenly causing xfsdump problems starting with Ubuntu 10.04 (Lucid) kernel 2.6.32-27 and persisting in 2.6.32-28.

Not really. The newer kernels have code in them that does more validity checks than previous kernels, so older kernels would have erroneously and silently returned unlinked files to xfsdump and have them backed up. IOWs, you'd never notice such a corruption with xfsdump. On the new kernel, xfsdump gets an EINVAL error to such occurrences, which it should have in the first place.

Quoted 4 lines

And there is one other person who confirmed this xfsdump problem running Lucid with kernel 2.6.32-28. They reported their "me too" in the Ubuntu bug tracker.

Quoted 7 lines
Quoted 2 lines

Could it be that 2.6.32-26 and prior managed to write something to disk corrupted, and the newer code is tripping on it?

That's what I'm trying to find out. Or it could be something as simple as your disk has had an undetected bit error that has flipped a bit in the inode allocation btree.

Quoted 7 lines

Hi Dave,

I am able to reproduce this on a system running Ubuntu 10.4 (2.6.32-28). I took a metadump of the filesystem and moved it to a system running 10.10 (2.6.35-25), and was able to successfully dump it there. Likewise it dumps fine on 2.6.38-rc1. So this suggests an issue with the Ubuntu 10.4 kernel.

2.6.35 hasn't had the untrusted inode lookup patches back ported to it, so it's no surprise that it isn't having problems - it's just like the older 2.6.32 kernels.

Quoted 14 lines
Quoted 3 lines

I thought it landed in 2.6.35 and then a regression was fixed in 2.6.36. The untrusted inode lookup changes are referenced here:
http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.35

My bad, I just checked the regression fix. I have no idea if it got back ported to 2.6.35-stable or not - it probably didn't judging by your results.....

Quoted 6 lines
Quoted 4 lines

Hmmm, can you find out if there is any specific pattern to the inode numbers that are returning EINVAL? Maybe the inode allocbt freespace record checks aren't quite correct in the backport (like the original bogus alignment assumption I made).

I'll take a look.

The failing bulkstats, at least the ones I've checked so far, are hitting this path in xfs_bulkstat():

/*
* Skip if this inode is free.
*/

if (XFS_INOBT_MASK(chunkidx) & irbp->ir_free) {
	lastino = ino;
	continue;
}

The backport of the 4 untrusted inode lookup commits looks okay to me, however I think they depend on commit 7dce11dbac54fce777eea0f5fb25b2694ccd7900 (xfs: always use iget in bulkstat), which was checked in shortly before the untrusted inode lookup changes. When that commit is added to the Ubuntu 2.6.32-28 kernel, xfsdump runs fine on the 2 filesystems of mine that were exhibiting the problem.

Bill

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#13 lore raw
In reply to Dave Chinner

Greetings Dave,

I am just finally getting back to this. Rebooted to the latest 10.04 kernel.

$ uname -a
Linux ldslnx01 2.6.32-28-generic-pae #55-Ubuntu SMP Mon Jan 10 22:34:08 UTC 2011 i686 GNU/Linux
Quoted 3 lines · Dave Chinner

Can you firstly post the output of:

# xfs_db -c "inode 80508397" -c p<dev>

Command seems grumpy...

$ sudo xfs_db -c "inode 80508397" -c p <dev>
-bash: syntax error near unexpected token `newline'

I even tried unmounting the /srv partition. I am having difficulty finding docs about the usage of xfs_db / xfs_repair.

rrrrrr?????

Quoted 3 lines

And can you also run 'xfs_repair -n<dev>' on the filesystem and post the output as well so we can see what state the filesytem is in?

This one as well...

/srv$ sudo xfs_repair -n<dev>
-bash: syntax error near unexpected token `newline'

Sincerely,

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#14 lore raw
Quoted 10 lines · Michael Lueck

Greetings Dave,

I am just finally getting back to this. Rebooted to the latest 10.04 kernel.

$ uname -a
Linux ldslnx01 2.6.32-28-generic-pae #55-Ubuntu SMP Mon Jan 10 22:34:08 UTC 2011 i686 GNU/Linux
Quoted 2 lines · Dave Chinner

Can you firstly post the output of:

Quoted 8 lines
Quoted 1 line
# xfs_db -c "inode 80508397" -c p<dev>

Command seems grumpy...

$ sudo xfs_db -c "inode 80508397" -c p <dev>
-bash: syntax error near unexpected token `newline'

I even tried unmounting the /srv partition. I am having difficulty finding docs about the usage of xfs_db / xfs_repair.

The man pages, perhaps? i.e. 'man xfs_db' and 'man xfs_repair'

Quoted 8 lines
Quoted 3 lines

And can you also run 'xfs_repair -n<dev>' on the filesystem and post the output as well so we can see what state the filesytem is in?

This one as well...

/srv$ sudo xfs_repair -n<dev>
-bash: syntax error near unexpected token `newline'

Substitute /dev/sda, /dev/sdb or whatever your filesystem is on for "<dev>" in the above commands.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#15 lore raw
Quoted 1 line · Dave Chinner

The man pages, perhaps? i.e. 'man xfs_db' and 'man xfs_repair'

I did, and oddly enough I did not find "<dev>" in the example syntax...

Quoted 2 lines

Substitute /dev/sda, /dev/sdb or whatever your filesystem is on for "<dev>" in the above commands.

aaaahhh, THAT explains the curve ball. ;-)

I will IPL the server to the latest kernel and run the tests immediately.

Thank you so much for the explanation!

Sincerely,

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#16 lore raw
Quoted 1 line · Michael Lueck

I will IPL the server to the latest kernel and run the tests immediately.

Here they are attached...

Sincerely,

Signature · 3 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/
# sudo xfs_db -c "inode 80508397" -c p /dev/sda9 2>&1 | tee xfs_db-inode_80508397.log
core.magic = 0x494e
core.mode = 040777
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 2
core.onlink = 0
core.projid = 0
core.uid = 1000
core.gid = 1000
core.flushiter = 10
core.atime.sec = Sun Feb  6 16:59:44 2011
core.atime.nsec = 088915516
core.mtime.sec = Tue Jan 11 13:50:51 2011
core.mtime.nsec = 950662167
core.ctime.sec = Tue Jan 11 13:50:51 2011
core.ctime.nsec = 950662167
core.size = 4096
core.nblocks = 1
core.extsize = 0
core.nextents = 1
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 3978985788
next_unlinked = null
u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,5052169,1,0]
# sudo xfs_repair -n /dev/sda9 2>&1 | tee xfs_repair-srv.log
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#17 lore raw
Quoted 41 lines · Michael Lueck
Quoted 1 line · Michael Lueck

I will IPL the server to the latest kernel and run the tests immediately.

Here they are attached...

# sudo xfs_db -c "inode 80508397" -c p /dev/sda9 2>&1 | tee xfs_db-inode_80508397.log
core.magic = 0x494e
core.mode = 040777
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 2
core.onlink = 0
core.projid = 0
core.uid = 1000
core.gid = 1000
core.flushiter = 10
core.atime.sec = Sun Feb  6 16:59:44 2011
core.atime.nsec = 088915516
core.mtime.sec = Tue Jan 11 13:50:51 2011
core.mtime.nsec = 950662167
core.ctime.sec = Tue Jan 11 13:50:51 2011
core.ctime.nsec = 950662167
core.size = 4096
core.nblocks = 1
core.extsize = 0
core.nextents = 1
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
Quoted 8 lines
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 3978985788
next_unlinked = null
u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,5052169,1,0]

Nothing wrong with that inode....

Quoted 19 lines
# sudo xfs_repair -n /dev/sda9 2>&1 | tee xfs_repair-srv.log
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
Quoted 8 lines
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

And no errors detected in the filesystem. Can you run the xfsdump again to confirm that it fails on the same inode? If it doesņ then it definitely seems like an alignment problem in the untrusted inode lookup patches....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#18 lore raw

Greetings Dave,

Quoted 4 lines · Dave Chinner

And no errors detected in the filesystem. Can you run the xfsdump again to confirm that it fails on the same inode? If it doesņ then it definitely seems like an alignment problem in the untrusted inode lookup patches....

I got around to doing the recreate test just now. A larger trace file this time, but still crashed. Rebooted off of the -26 kernel and pulling a backup with "old faithful" kernel.

Please find the trace output here in zipped format:

http://www.lueckdatasystems.com/pub/ldsbackup.trace2.log.zip

I suppose this means that the "untrusted inode lookup patches" are further confirmed as missing? If so, any idea on how long it will take to receive an updated kernel with the xfs code corrections?

Sincerely,

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#19 lore raw
In reply to Dave Chinner

Greetings Dave,

I am just finally getting back to this. Rebooted to the latest 10.04 kernel.

$ uname -a
Linux ldslnx01 2.6.32-28-generic-pae #55-Ubuntu SMP Mon Jan 10 22:34:08 UTC 2011 i686 GNU/Linux
Quoted 3 lines · Dave Chinner

Can you firstly post the output of:

# xfs_db -c "inode 80508397" -c p<dev>

Command seems grumpy...

$ sudo xfs_db -c "inode 80508397" -c p <dev>
-bash: syntax error near unexpected token `newline'

I even tried unmounting the /srv partition. I am having difficulty finding docs about the usage of xfs_db / xfs_repair.

rrrrrr?????

Quoted 3 lines

And can you also run 'xfs_repair -n<dev>' on the filesystem and post the output as well so we can see what state the filesytem is in?

This one as well...

/srv$ sudo xfs_repair -n<dev>
-bash: syntax error near unexpected token `newline'

Sincerely,

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#20 lore raw
In reply to Dave Chinner

Greetings Dave-

BTW: I ended up testing with the latest 10.04 kernel.

The size diffrerence of both kernel packages is disturbing!

-rw-r--r-- 1 root root 31608926 2011-01-11 13:09 linux-image-2.6.32-28-generic-pae_2.6.32-28.55_i386.deb
-rw-r--r-- 1 root root 4160 2010-11-08 05:05 linux-image-generic-pae_2.6.32.26.28_i386.deb

Perhaps that is due to the only change-log entry that came through apticron to me that "bump kernel number in prep for xyz" type of a change comment.

Sincerely,

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#21 lore raw
In reply to Dave Chinner
Quoted 76 lines · Dave Chinner
Quoted 31 lines · Bill Kendall
Quoted 6 lines · Michael Lueck

Greetings,

Somehow a reported IRIX bug with XFS got into Ubuntu 10.04 (Lucid) starting with kernel 2.6.32-27.

I am cross posting to this list in order to receive details as to what bug got in Ubuntu which has been solved in IRIX, hoping with more details Ubuntu might make the same fix.

Aside from the fact that the errno is the same, there's nothing to suggest that the Ubuntu problem is the the same bug. The IRIX bug is quite old.

Quoted 6 lines

Ubuntu has since updated their kernel to 2.6.32-28 and someone already verified at the bug report that the problem persists with that kernel version.

"Regression between 2.6.32-27 and 2.6.32-26 xfsdump SGI_FS_BULKSTAT errno = 22"
https://bugs.launchpad.net/bugs/692848

Between 2.6.32-26 and 2.6.32-27, Ubuntu backported 4 XFS commits from 2.6.35/2.6.36. All are part of a bulkstat security fix.

% git log Ubuntu-2.6.32-26.48..Ubuntu-2.6.32-27.49 -- fs/xfs | grep commit commit 52d2a4cfbc852da8c3d3b9fa0cac2a07b12f5cfd

    (cherry picked from commit 4536f2ad8b330453d7ebec0746c4374eadd649b1)
commit eb5ab28c8a5e4bb3f1ce05eba166c12175f6c701
    (backported from commit 7b6259e7a83647948fa33a736cc832310c8d85aa)
commit 5f8e8c6ab416bbd58d4f5df512c119a888ff923c
    (cherry picked from commit 1920779e67cbf5ea8afef317777c5bf2b8096188)
commit 52e0d703745f7110f1ecbe83c02cf06a83da82e8
    (backported from commit 7124fe0a5b619d65b739477b3b55a20bf805b06d)
Quoted 2 lines

I'm not aware of a similar problem upstream, so it would appear to be a problem with Ubuntu's backport of these commits.

<sigh>

That'll be the untrusted inode lookup fixes.

Quoted 30 lines

Bill

Quoted 27 lines

On our Ubuntu 10.04 LTS server running x86 code, this evening a kernel update was ready for installation. I updated the kernel, rebooted (IPL'ed), and proceeded with the backup which utilized xfsdump as we use the xfs filesystem. Four of the xfsdump received a never before seen
error: SGI_FS_BULKSTAT errno = 22

Output as follows:
using file dump (drive_simple) strategy
version 3.0.4 (dump format 3.0) - Running single-threaded
level 0 dump of ldslnx01:/srv
dump date: Mon Dec 20 21:59:33 2010
session id: f98f8cc0-963f-41a6-9a19-a89192502bf0
session label: "data"
ino map phase 1: constructing initial dump list
ino map phase 2: skipping (no pruning necessary)
ino map phase 3: skipping (only one dump stream)
ino map construction complete
estimated dump size: 100739707392 bytes
WARNING: no media label specified
creating dump session media file 0 (media 0, file 0)
dumping ino map
dumping directories
SGI_FS_BULKSTAT failed: Invalid argument (22)
dump size (non-dir files) : 0 bytes
NOTE: dump interrupted: 79 seconds elapsed: may resume later using -R option Dump Status: INTERRUPT

So bulkstat got EINVAL returned for and inode that it was looking up. That implies that it was racing with an unlink, which is what the above commits catch and prevent. Can you run xfsdump with full debug output (-v 5) so we can see what inode is being operated on when this failure occurs?

Quoted 3 lines

Dann, I'm not sure whether this means there was a bug in your backport or whether it's just xfsdump not handling a failure gracefully....

Yeah, I'm not sure either. I've attempted to reproduce it, but have so far not been successful. I'll try to get a 32-bit test system setup in case that makes a difference.

Quoted 25 lines
Quoted 15 lines
Quoted 15 lines

This backup file did have some size to it. The other three, backing up smaller amounts of data, were all zero (0) length dump files.

I rebooted to the prior kernel: $ uname -a
Linux ldslnx01 2.6.32-26-generic-pae #48-Ubuntu SMP Wed Nov 24 10:31:20 UTC 2010 i686 GNU/Linux

And the same backup gets to 100% success.

Reboot to the new kernel, same failure.

I think that fairly well illustrates that the problem exists only with the kernel update installed this evening.

<><><><>

Quoted 7 lines
Quoted 7 lines

I did come across a reference to this problem on the SGI website:

http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=relnotes&fname=/usr/relnotes/eoe
1.6.17 Bugs fixed in IRIX 6.5.13
+ 816457: xfsdump SGI_FS_BULKSTAT errno = 22 cxfs

So evidently it is something that has been seen and corrected in IRIX.

Oh, the curse of Google.

Quoted 7 lines

Irix 6.5.13 was released in 2001, so I don't think this is at all relevant for a regression reported for the latest and greatest Ubuntu kernel....

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#22 lore raw
In reply to Michael Lueck

Greetings,

Yesterday I received via production Ubuntu 10.04 Lucid updates the new kernel addressing this issue.

Thank you so much Bill, Dave, and Dann for your assistance in getting this diagnosed and resolved so quickly! I really appreciate it!

Sincerely,

Signature · 8 lines
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs