3.16.3: fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empty

PATCH In review Marc MERLIN 4 participants 10 messages Dec 28, 2014 – Jan 2, 2015 Btrfs file system

lore

Share & tools

lore

Marc MERLIN

author

2014-12-28 19:26 UTC #0 lore raw

Not sure if it's useful to anyone, but there you go. This happened after a forced power cycle:

BTRFS info (device dm-1): disk space caching is enabled

------------[ cut here ]------------
WARNING: CPU: 2 PID: 778 at fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empty+0x32/0x34()
Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek snd_hda_codec_generic microcode snd_cmipci gameport snd_hda_intel kvm_intel snd_hda_controller kvm snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi battery snd_rawmidi snd_hwdep sparse_keymap rfkill snd_pcm snd_seq_device tpm_infineon snd_timer tpm_tis rc_ati_x10 asix coretemp tpm i2c_i801 snd processor wmi pl2303 kl5kusb105 libphy ati_remote parport_pc rc_core xhci_hcd intel_rapl keyspan ftdi_sio evdev usbnet soundcore pcspkr parport lpc_ich intel_powerclamp ezusb usbserial x86_pkg_temp_thermal xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32c_intel crc32_pclmul sata_sil24 thermal crct10dif_pclmul ehci_pci ehci_hcd ghash_clmulni_intel r8169 cryptd fan mii usbcore usb_common sata_mv
CPU: 2 PID: 778 Comm: btrfs-transacti Tainted: G        W     3.16.7-amd64-i915-volpreempt-20141114 #1
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
 0000000000000000 ffff8802117abdc8 ffffffff816295db 0000000000000000
 ffff8802117abe00 ffffffff81051e2d ffffffff8127038d ffff880211534000
 ffff880212639980 0000000000000000 ffff8802137eaf00 ffff8802117abe10
Call Trace:
 [<ffffffff816295db>] dump_stack+0x45/0x56
 [<ffffffff81051e2d>] warn_slowpath_common+0x7f/0x98
 [<ffffffff8127038d>] ? btrfs_assert_delayed_root_empty+0x32/0x34
 [<ffffffff81051ef4>] warn_slowpath_null+0x1a/0x1c
 [<ffffffff8127038d>] btrfs_assert_delayed_root_empty+0x32/0x34
 [<ffffffff8122db95>] btrfs_commit_transaction+0x37f/0x867
 [<ffffffff8122a2f1>] transaction_kthread+0xec/0x19f
 [<ffffffff8122a205>] ? btrfs_cleanup_transaction+0x3f3/0x3f3
 [<ffffffff8106cd8f>] kthread+0xae/0xb6
 [<ffffffff8106cce1>] ? __kthread_parkme+0x61/0x61
 [<ffffffff8163007c>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106cce1>] ? __kthread_parkme+0x61/0x61
---[ end trace 32de13ca415f14fa ]---

I'm now getting this on interval as kernel spam.

It doesn't say which of my 4 btrfs volumes this is linked to, which doesn't make life easier.

Any idea what I should do from here?

Will btrfs scrub, even if it takes about 24H to run for me, tell me which FS is affected and if so do I run btrfs repair?

Thanks,
Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Roman Mamedov

2014-12-28 20:00 UTC #1 lore raw

Quoted 7 lines · On Sun, 28 Dec 2014 11:26:14 -0800 Marc MERLIN <[email protected]> wrote:

Not sure if it's useful to anyone, but there you go. This happened after a forced power cycle:

BTRFS info (device dm-1): disk space caching is enabled

------------[ cut here ]------------
WARNING: CPU: 2 PID: 778 at fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empty+0x32/0x34()
Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek snd_hda_codec_generic microcode snd_cmipci gameport snd_hda_intel kvm_intel snd_hda_controller kvm snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi battery snd_rawmidi snd_hwdep sparse_keymap rfkill snd_pcm snd_seq_device tpm_infineon snd_timer tpm_tis rc_ati_x10 asix coretemp tpm i2c_i801 snd processor wmi pl2303 kl5kusb105 libphy ati_remote parport_pc rc_core xhci_hcd intel_rapl keyspan ftdi_sio evdev usbnet soundcore pcspkr parport lpc_ich intel_powerclamp ezusb usbserial x86_pkg_temp_thermal xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32c_intel crc32_pclmul sata_sil24 thermal crct10dif_pclmul ehci_pci eh

Quoted 23 lines

_hcd ghash_clmulni_intel r8169 cryptd fan mii usbcore usb_common sata_mv

CPU: 2 PID: 778 Comm: btrfs-transacti Tainted: G        W     3.16.7-amd64-i915-volpreempt-20141114 #1
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
 0000000000000000 ffff8802117abdc8 ffffffff816295db 0000000000000000
 ffff8802117abe00 ffffffff81051e2d ffffffff8127038d ffff880211534000
 ffff880212639980 0000000000000000 ffff8802137eaf00 ffff8802117abe10
Call Trace:
 [<ffffffff816295db>] dump_stack+0x45/0x56
 [<ffffffff81051e2d>] warn_slowpath_common+0x7f/0x98
 [<ffffffff8127038d>] ? btrfs_assert_delayed_root_empty+0x32/0x34
 [<ffffffff81051ef4>] warn_slowpath_null+0x1a/0x1c
 [<ffffffff8127038d>] btrfs_assert_delayed_root_empty+0x32/0x34
 [<ffffffff8122db95>] btrfs_commit_transaction+0x37f/0x867
 [<ffffffff8122a2f1>] transaction_kthread+0xec/0x19f
 [<ffffffff8122a205>] ? btrfs_cleanup_transaction+0x3f3/0x3f3
 [<ffffffff8106cd8f>] kthread+0xae/0xb6
 [<ffffffff8106cce1>] ? __kthread_parkme+0x61/0x61
 [<ffffffff8163007c>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106cce1>] ? __kthread_parkme+0x61/0x61
---[ end trace 32de13ca415f14fa ]---

I'm now getting this on interval as kernel spam.

Quoted 7 lines

It doesn't say which of my 4 btrfs volumes this is linked to, which doesn't make life easier.
Any idea what I should do from here?
Will btrfs scrub, even if it takes about 24H to run for me, tell me which FS is affected and if so do I run btrfs repair?

I had this: http://www.spinics.net/lists/linux-btrfs/msg40586.html

1) I determined which btrfs of the multiple ones that I have is the culprit, by unmounting them one by one and seeing if the dmesg spam disappears;

2) Surprisingly(#1), it was not the one that was heavily operated on in the fashion described in that message;

3) After that, I ran btrfsck (it did found some errors that looked like this, repeated dozens of times, with different "root nnnnn" numbers):

root 22730 inode 97339 errors 200, dir isize wrong
root 22730 inode 4044171 errors 200, dir isize wrong
root 22730 inode 4478553 errors 200, dir isize wrong
root 22730 inode 6236418 errors 2000, link count wrong
unresolved ref dir 105512 index 586340 namelen 48 name [redacted].dat.bak filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6325949 errors 2000, link count wrong
unresolved ref dir 105512 index 586348 namelen 48 name [redacted].dat.bak filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326136 errors 2000, link count wrong
unresolved ref dir 105512 index 586344 namelen 48 name [redacted].dat.bak filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326291 errors 2000, link count wrong
unresolved ref dir 104979 index 192292 namelen 16 name downloads.config filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326292 errors 2000, link count wrong
unresolved ref dir 4376855 index 19522 namelen 15 name xfce4-panel.xml filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326296 errors 2000, link count wrong
unresolved ref dir 104979 index 192295 namelen 18 name azureus.statistics filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326380 errors 2000, link count wrong
unresolved ref dir 4478552 index 45107 namelen 11 name Local State filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326402 errors 2000, link count wrong
unresolved ref dir 105469 index 233238 namelen 11 name diverse.dat filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326405 errors 2000, link count wrong
unresolved ref dir 4478553 index 914792 namelen 17 name TransportSecurity filetype 0 errors 3, no dir item, no dir index

4) then btrfsck --repair;

5) The latter, after three hours flat of 100% CPU usage on a 4 GHz AMD FX, proceeding very slowly with increasing the "root nnnnn" number and the same error descriptions, still didn't manage to fix of them; so I terminated it as I had other work to do on the machine, rather than sitting around with its key FSes unmounted;

6) Surprisingly(#2), despite apparently not all of the errors having been fixed, the btrfs_assert_delayed_root_empty messages no longer appear in dmesg.

The current versions of files mentioned (xfce4-panel.xml and parts of the Chromium profile) were of course corrupted, but I already noticed that and restored them from an earlier snapshot even before starting the fsck (yes I also had backups, but didn't need them as snapshotted versions were fine).

Signature · 2 lines

With respect,
Roman

Marc MERLIN

author

2014-12-28 21:36 UTC #2 lore raw

Quoted 7 lines · Roman Mamedov

Quoted 2 lines
Will btrfs scrub, even if it takes about 24H to run for me, tell me which FS is affected and if so do I run btrfs repair?
I had this: http://www.spinics.net/lists/linux-btrfs/msg40586.html
1) I determined which btrfs of the multiple ones that I have is the culprit, by unmounting them one by one and seeing if the dmesg spam disappears;

And of course it's the root filesystem on a remote server which I can't service remotely :-/

Quoted 2 lines

3) After that, I ran btrfsck (it did found some errors that looked like this, repeated dozens of times, with different "root nnnnn" numbers):

For the archives, one should use btrfs check --repair directly, btrfsck is dead.

Quoted 7 lines

6) Surprisingly(#2), despite apparently not all of the errors having been fixed, the btrfs_assert_delayed_root_empty messages no longer appear in dmesg.
The current versions of files mentioned (xfce4-panel.xml and parts of the Chromium profile) were of course corrupted, but I already noticed that and restored them from an earlier snapshot even before starting the fsck (yes I also had backups, but didn't need them as snapshotted versions were fine).

Thanks for the info. I think for now I'll be forced to leave the broken FS run as is and will deal with it when I get home.

Dear btrfs-devs: this is one more example of btrfs having a problem with a non consistent state that ended up on disk.

I got there this way:
- btrfs on top of dmcrypt on top of md raid1 (sorry too many raid bugs in btrfs, so I went back to mdadm at the time)
- kernel bug in a serial driver was causing a loop, so I was forced to cycle power remotely
- btrfs got broken as per this mail.
- please please please, all warnings and bugs should still be fixed to output what device they happened on. Making the admin guess by trying filesystem one by one isn't really a good way.

Anyway, assuming there isn't a core bug in the btrfs "always consistent state on disk" code, dmcrypt or mdadm prevented a consistent state from reaching the disks.

Separately, I wish I could just fix this while the filesystem is online. btrfs scrub ran totally clean with no errors :(
scrub device /dev/mapper/cryptroot (id 1) done
scrub started at Sun Dec 28 12:07:55 2014 and finished after 512 seconds total bytes scrubbed: 25.95GiB with 0 errors

Thankfully the filesystem is still running for now, so it could be worse.

Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Chris Mason

2014-12-29 15:17 UTC #3 lore raw

Quoted 56 lines · Marc MERLIN

Quoted 10 lines · Roman Mamedov
Quoted 1 line
Will btrfs scrub, even if it takes about 24H to run for me, tell
me
Quoted 1 line
which FS is affected and if so do I run btrfs repair?
I had this:
https://urldefense.proofpoint.com/v1/url?u=http://www.spinics.net/lists/linux-btrfs/msg40586.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=6%2FL0lzzDhu0Y1hL9xm%2BQyA%3D%3D%0A&m=yBJylKLQ0wXzMPYXMMCJaXZfTMrX%2FbRGSoF3t%2FRZsUU%3D%0A&s=9d08d8fb169b6429b819fb9a0c2fda816b4b6c031ee4c5e6ca5a53bb04e3c067
1) I determined which btrfs of the multiple ones that I have is the
culprit, by
unmounting them one by one and seeing if the dmesg spam disappears;
And of course it's the root filesystem on a remote server which I can't service remotely :-/
Quoted 3 lines
3) After that, I ran btrfsck (it did found some errors that looked
like this,
repeated dozens of times, with different "root nnnnn" numbers):
For the archives, one should use btrfs check --repair directly, btrfsck is dead.
Quoted 5 lines
6) Surprisingly(#2), despite apparently not all of the errors
having been
fixed, the btrfs_assert_delayed_root_empty messages no longer
appear in dmesg.
Quoted 7 lines
The current versions of files mentioned (xfce4-panel.xml and parts
of the Chromium profile)
were of course corrupted, but I already noticed that and restored
them from an earlier snapshot
even before starting the fsck (yes I also had backups, but didn't
need them as snapshotted versions
were fine).
Thanks for the info. I think for now I'll be forced to leave the broken FS run as is and will deal with it when I get home.
Dear btrfs-devs: this is one more example of btrfs having a problem with a non consistent state that ended up on disk.
I got there this way:
- btrfs on top of dmcrypt on top of md raid1 (sorry too many raid bugs in btrfs, so I went back to mdadm at the time)
- kernel bug in a serial driver was causing a loop, so I was forced to cycle power remotely
- btrfs got broken as per this mail.
- please please please, all warnings and bugs should still be fixed to output what device they happened on. Making the admin guess by
trying
filesystem one by one isn't really a good way.
Anyway, assuming there isn't a core bug in the btrfs "always consistent state on disk" code, dmcrypt or mdadm prevented a consistent state from reaching the disks.

Quoted 8 lines

Separately, I wish I could just fix this while the filesystem is online. btrfs scrub ran totally clean with no errors :(
scrub device /dev/mapper/cryptroot (id 1) done
scrub started at Sun Dec 28 12:07:55 2014 and finished after
512 seconds
total bytes scrubbed: 25.95GiB with 0 errors
Thankfully the filesystem is still running for now, so it could be worse.

I've hit this recently on my laptop, and haven't yet been able to recreate it on a machine where I can debug things. The messages are an error in the log tree replay code, and I don't think they are actually related to any corruptions. Trying to nail it down today.

-chris

Marc MERLIN

author

2014-12-29 15:41 UTC #4 lore raw

Quoted 4 lines · Chris Mason

I've hit this recently on my laptop, and haven't yet been able to recreate it on a machine where I can debug things. The messages are an error in the log tree replay code, and I don't think they are actually related to any corruptions. Trying to nail it down today.

Thanks for the update and looking at it.

Just to rule things out for me, on your laptop, are you running btrfs directly on disk, or do you have layers like dmcrypt in the middle? (having 2 other layers myself, I never know if it's btrfs that could be to blame, or the other 2 layers not passing data through in atomic bits like they're supposed to)

Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Chris Mason

2014-12-29 15:57 UTC #5 lore raw

Quoted 6 lines · Marc MERLIN

Quoted 4 lines · Chris Mason
I've hit this recently on my laptop, and haven't yet been able to recreate it on a machine where I can debug things. The messages are an error in the log tree replay code, and I don't think they are actually related to any corruptions. Trying to nail it down today.

Quoted 7 lines

Thanks for the update and looking at it.
Just to rule things out for me, on your laptop, are you running btrfs directly on disk, or do you have layers like dmcrypt in the middle? (having 2 other layers myself, I never know if it's btrfs that could be to blame, or the other 2 layers not passing data through in atomic bits like they're supposed to)

I do have dmcrypt, but I really think this is only in btrfs.

-chris

Qu Wenruo

2014-12-30 01:07 UTC #6 lore raw

In reply to Roman Mamedov

-------- Original Message --------
Subject: Re: 3.16.3: fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empty
From: Roman Mamedov <[email protected]>
To: Marc MERLIN <[email protected]>
Date: 2014年12月29日 04:00

Quoted 66 lines

Quoted 7 lines · On Sun, 28 Dec 2014 11:26:14 -0800 Marc MERLIN <[email protected]> wrote:
Not sure if it's useful to anyone, but there you go. This happened after a forced power cycle:
BTRFS info (device dm-1): disk space caching is enabled
------------[ cut here ]------------
WARNING: CPU: 2 PID: 778 at fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empty+0x32/0x34()
Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek snd_hda_codec_generic microcode snd_cmipci gameport snd_hda_intel kvm_intel snd_hda_controller kvm snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi battery snd_rawmidi snd_hwdep sparse_keymap rfkill snd_pcm snd_seq_device tpm_infineon snd_timer tpm_tis rc_ati_x10 asix coretemp tpm i2c_i801 snd processor wmi pl2303 kl5kusb105 libphy ati_remote parport_pc rc_core xhci_hcd intel_rapl keyspan ftdi_sio evdev usbnet soundcore pcspkr parport lpc_ich intel_powerclamp ezusb usbserial x86_pkg_temp_thermal xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32c_intel crc32_pclmul sata_sil24 thermal crct10dif_pclmul eh!
ci_pci eh
ci
Quoted 23 lines
_hcd ghash_clmulni_intel r8169 cryptd fan mii usbcore usb_common sata_mv
CPU: 2 PID: 778 Comm: btrfs-transacti Tainted: G        W     3.16.7-amd64-i915-volpreempt-20141114 #1
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
  0000000000000000 ffff8802117abdc8 ffffffff816295db 0000000000000000
  ffff8802117abe00 ffffffff81051e2d ffffffff8127038d ffff880211534000
  ffff880212639980 0000000000000000 ffff8802137eaf00 ffff8802117abe10
Call Trace:
  [<ffffffff816295db>] dump_stack+0x45/0x56
  [<ffffffff81051e2d>] warn_slowpath_common+0x7f/0x98
  [<ffffffff8127038d>] ? btrfs_assert_delayed_root_empty+0x32/0x34
  [<ffffffff81051ef4>] warn_slowpath_null+0x1a/0x1c
  [<ffffffff8127038d>] btrfs_assert_delayed_root_empty+0x32/0x34
  [<ffffffff8122db95>] btrfs_commit_transaction+0x37f/0x867
  [<ffffffff8122a2f1>] transaction_kthread+0xec/0x19f
  [<ffffffff8122a205>] ? btrfs_cleanup_transaction+0x3f3/0x3f3
  [<ffffffff8106cd8f>] kthread+0xae/0xb6
  [<ffffffff8106cce1>] ? __kthread_parkme+0x61/0x61
  [<ffffffff8163007c>] ret_from_fork+0x7c/0xb0
  [<ffffffff8106cce1>] ? __kthread_parkme+0x61/0x61
---[ end trace 32de13ca415f14fa ]---
I'm now getting this on interval as kernel spam.
Quoted 7 lines
It doesn't say which of my 4 btrfs volumes this is linked to, which doesn't make life easier.
Any idea what I should do from here?
Will btrfs scrub, even if it takes about 24H to run for me, tell me which FS is affected and if so do I run btrfs repair?
I had this: http://www.spinics.net/lists/linux-btrfs/msg40586.html
1) I determined which btrfs of the multiple ones that I have is the culprit, by unmounting them one by one and seeing if the dmesg spam disappears;
2) Surprisingly(#1), it was not the one that was heavily operated on in the fashion described in that message;
3) After that, I ran btrfsck (it did found some errors that looked like this, repeated dozens of times, with different "root nnnnn" numbers):
root 22730 inode 97339 errors 200, dir isize wrong
root 22730 inode 4044171 errors 200, dir isize wrong
root 22730 inode 4478553 errors 200, dir isize wrong
root 22730 inode 6236418 errors 2000, link count wrong
unresolved ref dir 105512 index 586340 namelen 48 name [redacted].dat.bak filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6325949 errors 2000, link count wrong
unresolved ref dir 105512 index 586348 namelen 48 name [redacted].dat.bak filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326136 errors 2000, link count wrong
unresolved ref dir 105512 index 586344 namelen 48 name [redacted].dat.bak filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326291 errors 2000, link count wrong
unresolved ref dir 104979 index 192292 namelen 16 name downloads.config filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326292 errors 2000, link count wrong
unresolved ref dir 4376855 index 19522 namelen 15 name xfce4-panel.xml filetype 0 errors 3, no dir item, no dir index

Quoted 8 lines

root 22730 inode 6326296 errors 2000, link count wrong
unresolved ref dir 104979 index 192295 namelen 18 name azureus.statistics filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326380 errors 2000, link count wrong
unresolved ref dir 4478552 index 45107 namelen 11 name Local State filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326402 errors 2000, link count wrong
unresolved ref dir 105469 index 233238 namelen 11 name diverse.dat filetype 0 errors 3, no dir item, no dir index
root 22730 inode 6326405 errors 2000, link count wrong
unresolved ref dir 4478553 index 914792 namelen 17 name TransportSecurity filetype 0 errors 3, no dir item, no dir index

According to the btrfsck, it seems that at least 3 other users have already hit the problem, and it seems to be the cause of the nlink problem.

Thanks,
Qu

Quoted 16 lines

4) then btrfsck --repair;
5) The latter, after three hours flat of 100% CPU usage on a 4 GHz AMD FX, proceeding very slowly with increasing the "root nnnnn" number and the same error descriptions, still didn't manage to fix of them; so I terminated it as I had other work to do on the machine, rather than sitting around with its key FSes unmounted;
6) Surprisingly(#2), despite apparently not all of the errors having been fixed, the btrfs_assert_delayed_root_empty messages no longer appear in dmesg.
The current versions of files mentioned (xfce4-panel.xml and parts of the Chromium profile) were of course corrupted, but I already noticed that and restored them from an earlier snapshot even before starting the fsck (yes I also had backups, but didn't need them as snapshotted versions were fine).

Chris Mason

2014-12-31 18:30 UTC #7 lore raw

PATCH 1/1 Btrfs: don't delay inode ref updates during log replay

Commit 1d52c78afbb (Btrfs: try not to ENOSPC on log replay) added a check to skip delayed inode updates during log replay because it confuses the enospc code. But the delayed processing will end up skipping delayed refs from log replay because the inode itself wasn't put through the delayed code.

This can end up triggering a warning at commit time:

WARNING: CPU: 2 PID: 778 at fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empty+0x32/0x34()

Which is repeated for each commit because we never process the delayed ref.

The fix used here is to change btrfs_delayed_delete_inode_ref to return an error if we're current in log replay. The caller will do the ref deletion immediately and everything will work properly.

This bug can cause lost files, whick fsck will find. --repair on btrfs-progs 3.18 will fix them

Signed-off-by Chris Mason <[email protected]>

cc [email protected] # v3.18 and any stable series that picked 1d52c78afbbf80b58299e076a159617d6b42fe3c

fs/btrfs/delayed-inode.c +8 -0

index 054577b..de4e70f 100644
@@ -1857,6 +1857,14 @@ int btrfs_delayed_delete_inode_ref(struct inode *inode)
 {
 	struct btrfs_delayed_node *delayed_node;
 
+	/*
+	 * we don't do delayed inode updates during log recovery because it
+	 * leads to enospc problems.  This means we also can't do
+	 * delayed inode refs
+	 */
+	if (BTRFS_I(inode)->root->fs_info->log_root_recovering)
+		return -EAGAIN;
+
 	delayed_node = btrfs_get_or_create_delayed_node(inode);
 	if (IS_ERR(delayed_node))
 		return PTR_ERR(delayed_node);

Marc MERLIN

author Tested-by on v1 1/1

2015-01-01 22:59 UTC #8 lore raw

Re: [PATCH] Btrfs: don't delay inode ref updates during log replay

Chris, you rule, I applied this to 3.16.7 and my problem went away.

Tested-by Marc MERLIN <[email protected]>

Happy new year! :)

Marc

Quoted 46 lines · Chris Mason

Commit 1d52c78afbb (Btrfs: try not to ENOSPC on log replay) added a check to skip delayed inode updates during log replay because it confuses the enospc code. But the delayed processing will end up skipping delayed refs from log replay because the inode itself wasn't put through the delayed code.
This can end up triggering a warning at commit time:
WARNING: CPU: 2 PID: 778 at fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empty+0x32/0x34()
Which is repeated for each commit because we never process the delayed ref.
The fix used here is to change btrfs_delayed_delete_inode_ref to return an error if we're current in log replay. The caller will do the ref deletion immediately and everything will work properly.
This bug can cause lost files, whick fsck will find. --repair on btrfs-progs 3.18 will fix them
Signed-off-by Chris Mason <[email protected]>
cc [email protected] # v3.18 and any stable series that picked 1d52c78afbbf80b58299e076a159617d6b42fe3c
fs/btrfs/delayed-inode.c +8 -1
index 054577b..de4e70f 100644
@@ -1857,6 +1857,14 @@ int btrfs_delayed_delete_inode_ref(struct inode *inode)
 {
 	struct btrfs_delayed_node *delayed_node;
 
+	/*
+	 * we don't do delayed inode updates during log recovery because it
+	 * leads to enospc problems.  This means we also can't do
+	 * delayed inode refs
+	 */
+	if (BTRFS_I(inode)->root->fs_info->log_root_recovering)
+		return -EAGAIN;
+
 	delayed_node = btrfs_get_or_create_delayed_node(inode);
 	if (IS_ERR(delayed_node))
 		return PTR_ERR(delayed_node);
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Qu Wenruo

2015-01-02 00:44 UTC #9 lore raw

Re: [PATCH] Btrfs: don't delay inode ref updates during log replay

Great! The root cause of missing file is fixed!

Thanks a lot!
Qu
-------- Original Message --------
Subject: Re: [PATCH] Btrfs: don't delay inode ref updates during log replay
From: Marc MERLIN <[email protected]>
To: Chris Mason <[email protected]>, Qu Wenruo <[email protected]>, Roman Mamedov <[email protected]>, Btrfs BTRFS <[email protected]>
Date: 2015年01月02日 06:58

Quoted 55 lines

Chris, you rule, I applied this to 3.16.7 and my problem went away.
Tested-by Marc MERLIN <[email protected]>
Happy new year! :)
Marc
Quoted 46 lines · Chris Mason
Commit 1d52c78afbb (Btrfs: try not to ENOSPC on log replay) added a check to skip delayed inode updates during log replay because it confuses the enospc code. But the delayed processing will end up skipping delayed refs from log replay because the inode itself wasn't put through the delayed code.
This can end up triggering a warning at commit time:
WARNING: CPU: 2 PID: 778 at fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empty+0x32/0x34()
Which is repeated for each commit because we never process the delayed ref.
The fix used here is to change btrfs_delayed_delete_inode_ref to return an error if we're current in log replay. The caller will do the ref deletion immediately and everything will work properly.
This bug can cause lost files, whick fsck will find. --repair on btrfs-progs 3.18 will fix them
Signed-off-by Chris Mason <[email protected]>
cc [email protected] # v3.18 and any stable series that picked 1d52c78afbbf80b58299e076a159617d6b42fe3c
fs/btrfs/delayed-inode.c +8 -1
index 054577b..de4e70f 100644
@@ -1857,6 +1857,14 @@ int btrfs_delayed_delete_inode_ref(struct inode *inode)
  {
  	struct btrfs_delayed_node *delayed_node;
 
+	/*
+	 * we don't do delayed inode updates during log recovery because it
+	 * leads to enospc problems.  This means we also can't do
+	 * delayed inode refs
+	 */
+	if (BTRFS_I(inode)->root->fs_info->log_root_recovering)
+		return -EAGAIN;
+
  	delayed_node = btrfs_get_or_create_delayed_node(inode);
  	if (IS_ERR(delayed_node))
  		return PTR_ERR(delayed_node);
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html