Skip to contentSkip to conversation

csum failure messages

DISCUSSION Russell Coker 7 participants 12 messages Nov 5, 2013 – Nov 14, 2013 Btrfs file system

lore
Share & tools
lore
#0 lore raw

The below messages are from dmesg on a system where "btrfs balance" just aborted. It's running kernel 3.11.6 (the latest Debian package).

This seems to be telling me that Inode 388 is involved, but there are over 300 subvols on that system which could contain such an Inode.

I think that more information is needed for such log messages. We need to at least be able to identify the subvol (is it possible to extract this from the numbers in the log messages?). Ideally we would be able to identify the file name as well.

[10751.637517] BTRFS info (device sda3): csum failed ino 388 off 23191552 csum 
2566472073 private 3193692311
[10751.646390] BTRFS info (device sda3): csum failed ino 388 off 24104960 csum 
5219137 private 2264608335
[10751.654472] BTRFS info (device sda3): csum failed ino 388 off 24154112 csum 
4084831521 private 1792217768
[10751.731830] BTRFS info (device sda3): csum failed ino 388 off 23191552 csum 

2566472073 private 3193692311

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#1 lore raw

As you were in the process of a rebalance these errors may actually be caused by this serious bug "Btrfs: relocate csums properly with prealloc extents".

I hit that myself with several preallocated files made by rtorrent during a rebalance and I lost several huge files as a consequence. The only way I could rebalance without large scale corruptions was to manually patch the 3.11.6 kernel with the small patch that fixes the issue.
For some reason this patch is not pushed upstream yet. I think that is strange as it leads to corruption and actual data loss and it is 100% reproducible with preallocated files. Only systemd logs is mentioned in the bug reports, but in my case it was actually hitting several terabytes of files created by rtorrent.

Mvh

Hans-Kristian Bakke

Quoted 28 lines · Russell Coker

The below messages are from dmesg on a system where "btrfs balance" just aborted. It's running kernel 3.11.6 (the latest Debian package).

This seems to be telling me that Inode 388 is involved, but there are over 300 subvols on that system which could contain such an Inode.

I think that more information is needed for such log messages. We need to at least be able to identify the subvol (is it possible to extract this from the numbers in the log messages?). Ideally we would be able to identify the file name as well.

[10751.637517] BTRFS info (device sda3): csum failed ino 388 off 23191552 csum
2566472073 private 3193692311
[10751.646390] BTRFS info (device sda3): csum failed ino 388 off 24104960 csum
5219137 private 2264608335
[10751.654472] BTRFS info (device sda3): csum failed ino 388 off 24154112 csum
4084831521 private 1792217768
[10751.731830] BTRFS info (device sda3): csum failed ino 388 off 23191552 csum

2566472073 private 3193692311

--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 4 lines · Hans-Kristian Bakke

As you were in the process of a rebalance these errors may actually be caused by this serious bug "Btrfs: relocate csums properly with prealloc extents".

Quoted 10 lines

I hit that myself with several preallocated files made by rtorrent during a rebalance and I lost several huge files as a consequence. The only way I could rebalance without large scale corruptions was to manually patch the 3.11.6 kernel with the small patch that fixes the issue.
For some reason this patch is not pushed upstream yet. I think that is strange as it leads to corruption and actual data loss and it is 100% reproducible with preallocated files. Only systemd logs is mentioned in the bug reports, but in my case it was actually hitting several terabytes of files created by rtorrent.

Thanks for the summary. There's no doubt that this is serious.

Chris, please can you somehow get the patch into stable sooner than it gets to the 3.13 queue? The merge window will start in 1 week and based on previous pull request schedule, the patch will be merged in ~2 weeks from now. That's kind of long time for for a unfixed corruption bug that reportedly affects common installations (with systemd) or usecases (torrent) in combination with balance.

david

#3 lore raw
In reply to Hans-Kristian Bakke
Quoted 4 lines · Hans-Kristian Bakke

As you were in the process of a rebalance these errors may actually be caused by this serious bug "Btrfs: relocate csums properly with prealloc extents".

Quoted 10 lines

I hit that myself with several preallocated files made by rtorrent during a rebalance and I lost several huge files as a consequence. The only way I could rebalance without large scale corruptions was to manually patch the 3.11.6 kernel with the small patch that fixes the issue.
For some reason this patch is not pushed upstream yet. I think that is strange as it leads to corruption and actual data loss and it is 100% reproducible with preallocated files. Only systemd logs is mentioned in the bug reports, but in my case it was actually hitting several terabytes of files created by rtorrent.

I run systemd to I guess it's the systemd logs. That's fortunate as such logs aren't important to me. Thanks for providing this information.

I've just run a scrub and I saw the following output. There was nothing useful or apparently relevant in the kernel message log either. So scrub is just telling me that there are 57 errors without giving me a clue as to which files might need to be restored from backup.

# btrfs scrub start -B /
scrub done for c55218a6-abb5-4e35-9a20-33fb1fa05879
        scrub started at Tue Nov  5 11:32:03 2013 and finished after 6762 
seconds
        total bytes scrubbed: 140.06GB with 57 errors
        error details: csum=57
        corrected errors: 0, uncorrectable errors: 57, unverified errors: 0

I can imagine a balance operation being unable to conveniently display all the data that one might desire. But a scrub really should go through everything and should know where the inconsistencies are. In this case the scrub gave me less information than the balance.

I presume that my filesystem is still corrupt.

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#4 lore raw

I gave up on getting the filesystem to a concistent state, but my corruption was much more severe than yours. Several 100 000's. As the fs was still usable and mountable I just moved all the files to another filesystem, patched the kernel recreated the original btrfs fs and ran a rebalance. This time without issues because of the patch. As the corrupt files were rtorrent files in my case I could just rehash the torrents and make rtorrent redownload the corrupt blocks. Very lucky indeed. The other files I could verify against backup.

Luckily the reason for the rebalance in the first place was to add another 16TB of disk to the RAID10 array, so I just happened to have enough temporary storage lying around. After patching the kernel and rebalance I now have a 32TB btrfs RAID10 volume.
Mvh

Hans-Kristian Bakke

Quoted 42 lines · Russell Coker
Quoted 4 lines · Hans-Kristian Bakke

As you were in the process of a rebalance these errors may actually be caused by this serious bug "Btrfs: relocate csums properly with prealloc extents".

Quoted 10 lines

I hit that myself with several preallocated files made by rtorrent during a rebalance and I lost several huge files as a consequence. The only way I could rebalance without large scale corruptions was to manually patch the 3.11.6 kernel with the small patch that fixes the issue.
For some reason this patch is not pushed upstream yet. I think that is strange as it leads to corruption and actual data loss and it is 100% reproducible with preallocated files. Only systemd logs is mentioned in the bug reports, but in my case it was actually hitting several terabytes of files created by rtorrent.

I run systemd to I guess it's the systemd logs. That's fortunate as such logs aren't important to me. Thanks for providing this information.

I've just run a scrub and I saw the following output. There was nothing useful or apparently relevant in the kernel message log either. So scrub is just telling me that there are 57 errors without giving me a clue as to which files might need to be restored from backup.

# btrfs scrub start -B /
scrub done for c55218a6-abb5-4e35-9a20-33fb1fa05879
        scrub started at Tue Nov  5 11:32:03 2013 and finished after 6762
seconds
        total bytes scrubbed: 140.06GB with 57 errors
        error details: csum=57
        corrected errors: 0, uncorrectable errors: 57, unverified errors: 0

I can imagine a balance operation being unable to conveniently display all the data that one might desire. But a scrub really should go through everything and should know where the inconsistencies are. In this case the scrub gave me less information than the balance.

I presume that my filesystem is still corrupt.

--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/

#5 lore raw
In reply to Russell Coker
Quoted 2 lines · Russell Coker

I presume that my filesystem is still corrupt.

I'm the original reporter of the bug. The file system itself isn't corrupt, but the affected files probably are. In my case, systemd journal files were reported as corrupt by systemd following a balance, as well as btrfs scrub. Upon scrub, dmesg contains a path for each affected file. Upon deleting those files, subsequent scrubs come up clean.

Fedora merged the fix for this bug with: 3.11.5-302.fc20, 3.11.6-200.fc19, 3.11.6-101.fc18. I thought for sure it was marked for stable to go into mainline 3.11.6, so somehow it's been missed.

Chris Murphy

#6 lore raw
Quoted 8 lines · Chris Murphy
Quoted 2 lines · Russell Coker

I presume that my filesystem is still corrupt.

I'm the original reporter of the bug. The file system itself isn't corrupt, but the affected files probably are. In my case, systemd journal files were reported as corrupt by systemd following a balance, as well as btrfs scrub. Upon scrub, dmesg contains a path for each affected file. Upon deleting those files, subsequent scrubs come up clean.

Fedora merged the fix for this bug with: 3.11.5-302.fc20, 3.11.6-200.fc19, 3.11.6-101.fc18. I thought for sure it was marked for stable to go into mainline 3.11.6, so somehow it's been missed.

Someone else tripped over it on IRC last night, and I was surprised
to discover it hadn't made it upstream yet. :(

Hugo.

Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- Two things came out of Berkeley in the 1960s: LSD and Unix. ---   
                       This is not a coincidence.                        
#7 lore raw
Quoted 4 lines · Hugo Mills
Quoted 3 lines · Chris Murphy
Quoted 1 line · Russell Coker
Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I presume that my filesystem is still corrupt.

I'm the original reporter of the bug. The file system itself isn't corrupt, but the affected files probably are. In my case, systemd journal files were reported as corrupt by systemd following a balance, as well as btrfs scrub. Upon scrub, dmesg contains a path for each affected file. Upon deleting those files, subsequent scrubs come up clean.

Fedora merged the fix for this bug with: 3.11.5-302.fc20, 3.11.6-200.fc19, 3.11.6-101.fc18. I thought for sure it was marked for stable to go into mainline 3.11.6, so somehow it's been missed.

Someone else tripped over it on IRC last night, and I was surprised
to discover it hadn't made it upstream yet. :(

Is there now a verification test that could detect an issue like this? It seems like the sort of thing that needs to be added to automated testing.

#8 lore raw

John Williams posted on Tue, 05 Nov 2013 16:20:58 -0800 as excerpted:

Quoted 3 lines

Is there now a verification test that could detect an issue like this? It seems like the sort of thing that needs to be added to automated testing.

[Your question is general enough, not mentioning xfs-tests, simply asking about general automated testing, I'm assuming a general answer is appropriate.]

I haven't tracked this specific issue, but in general, the btrfs devs are pretty strict with adding an xfs-tests (NOT used for just xfs, at least btrfs and ext4 use it too) package test for any regressions they find, and people ARE regularly running those tests on new code, so past issues don't happen again.

If you watch the list you'll see occasional patch rejections due to failed xfs-tests, as well as regular new xfs-tests patches adding new tests, as well as review discussion requesting an xfs-test be added as appropriate.

So I'd be /very/ surprised if this bugfix didn't already have a corresponding new xfs-tests test.

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#10 lore raw
In reply to Hugo Mills
Quoted 4 lines · Hugo Mills
Quoted 3 lines · Chris Murphy
Quoted 1 line · Russell Coker
Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I presume that my filesystem is still corrupt.

I'm the original reporter of the bug. The file system itself isn't corrupt, but the affected files probably are. In my case, systemd journal files were reported as corrupt by systemd following a balance, as well as btrfs scrub. Upon scrub, dmesg contains a path for each affected file. Upon deleting those files, subsequent scrubs come up clean.

Fedora merged the fix for this bug with: 3.11.5-302.fc20, 3.11.6-200.fc19, 3.11.6-101.fc18. I thought for sure it was marked for stable to go into mainline 3.11.6, so somehow it's been missed.

Someone else tripped over it on IRC last night, and I was surprised
to discover it hadn't made it upstream yet. :(

I just checked kernel.org changelogs and I'm not seeing this fixed in either 3.11.7 or 3.11.8. It is listed in today's git pull by Chris for 3.12 as

Btrfs: relocate csums properly with prealloc extents

Chris Murphy

#11 lore raw
Quoted 13 lines · Chris Murphy
Quoted 10 lines · Hugo Mills
Quoted 8 lines · Chris Murphy
Quoted 2 lines · Russell Coker

I presume that my filesystem is still corrupt.

I'm the original reporter of the bug. The file system itself isn't corrupt, but the affected files probably are. In my case, systemd journal files were reported as corrupt by systemd following a balance, as well as btrfs scrub. Upon scrub, dmesg contains a path for each affected file. Upon deleting those files, subsequent scrubs come up clean.

Fedora merged the fix for this bug with: 3.11.5-302.fc20, 3.11.6-200.fc19, 3.11.6-101.fc18. I thought for sure it was marked for stable to go into mainline 3.11.6, so somehow it's been missed.

Quoted 8 lines
Quoted 2 lines

Someone else tripped over it on IRC last night, and I was surprised
to discover it hadn't made it upstream yet. :(

I just checked kernel.org changelogs and I'm not seeing this fixed in either 3.11.7 or 3.11.8. It is listed in today's git pull by Chris for 3.12 as

Btrfs: relocate csums properly with prealloc extents

The stable tree process does not normally accept patches that are not in Linus' tree first. The patch has yet to be submitted to stable right after today's pull is merged.

david