Skip to contentSkip to conversation

Debian/Jessie 3.16.7-ckt2-1 kernel error

DISCUSSION Russell Coker 3 participants 5 messages Dec 24, 2014 – Jan 5, 2015 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I've attached the kernel message log that I get after booting kernel 3.16.7 from Debian/Unstable. This is the kernel branch that will go into Debian/Jessie so it's important to get it fixed.

Below has the start of the errors, the attached file has everything from boot. I've got similar issues on another system, would it help if I collect the logs from multiple systems?

[    6.618809] Btrfs loaded
[    6.670878] BTRFS: device fsid a2d7cbbc-23be-4d97-b2bb-de99b0c58c7d devid 1 
t
ransid 548614 /dev/mapper/root-crypt
[    6.706907] BTRFS info (device dm-0): disk space caching is enabled
[    6.798762] BTRFS: detected SSD devices, enabling SSD mode
[    6.881272] ------------[ cut here ]------------
[    6.881358] WARNING: CPU: 3 PID: 198 at /build/linux-CMiYW9/linux-3.16.7-

ckt2
/fs/btrfs/delayed-inode.c:1410 btrfs_commit_transaction+0x38a/0x9c0 [btrfs]()

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#1 lore raw

Hi Russel and all,

Quoted 16 lines · Russell Coker

I've attached the kernel message log that I get after booting kernel 3.16.7 from Debian/Unstable. This is the kernel branch that will go into Debian/Jessie so it's important to get it fixed.

Below has the start of the errors, the attached file has everything from boot. I've got similar issues on another system, would it help if I collect the logs from multiple systems?

[    6.618809] Btrfs loaded
[    6.670878] BTRFS: device fsid a2d7cbbc-23be-4d97-b2bb-de99b0c58c7d devid 1
t
ransid 548614 /dev/mapper/root-crypt
[    6.706907] BTRFS info (device dm-0): disk space caching is enabled
[    6.798762] BTRFS: detected SSD devices, enabling SSD mode
[    6.881272] ------------[ cut here ]------------
[    6.881358] WARNING: CPU: 3 PID: 198 at /build/linux-CMiYW9/linux-3.16.7-
Quoted 3 lines

ckt2
/fs/btrfs/delayed-inode.c:1410 btrfs_commit_transaction+0x38a/0x9c0 [btrfs]()

This problem is once reported by Petr and would not be fixed yet. In addition, there is no Btrfs related error/warning messages.

Here are my questions.

1. Is your system "btrfs scrub" clean?
2. Is this message shown every boot time?
3. Is this message shown only in boot?
4. When this message is started to be shown?
5. Do you have any trouble, change your operation or configuration just before the answer of Q4 ?

*** The result of my brief analysis (a bit long). ***

Your kernel traces are as follows > for developers

====

[    6.881272] ------------[ cut here ]------------
[    6.881358] WARNING: CPU: 3 PID: 198 at /build/linux-CMiYW9/linux-3.16.7-ckt2/fs/btrfs/delayed-inode.c:1410 btrfs_commit_transaction+0x38a/0x9c0 [btrfs]()
[    6.881453] Modules linked in: btrfs xor raid6_pq sha256_ssse3 sha256_generic ecb cbc algif_skcipher af_alg hid_generic usbhid hid dm_crypt dm_mod sd_mod crc_t10dif crct10dif_generic sg sr_mod crct10dif_common cdrom crc32c_intel r8169 mii ehci_pci ata_generic ehci_hcd ata_piix libata e1000e usbcore ptp scsi_mod usb_common pps_core
[    6.883373] CPU: 3 PID: 198 Comm: exe Not tainted 3.16.0-4-amd64 #1 Debian 3.16.7-ckt2-1
[    6.883455] Hardware name:                  /DH55HC, BIOS TCIBX10H.86A.0036.2010.0525.1620 05/25/2010
[    6.884169]  0000000000000009 ffffffff81507263 0000000000000000 ffffffff81065847
[    6.884449]  ffff8802366ba980 ffff8800b67fb000 ffff8802366bbe10 0000000000000000
[    6.884723]  0000000000000000 ffffffffa022727a cc9e8800b750eec8 006c000000000029
[    6.885057] Call Trace:
[    6.885127]  [<ffffffff81507263>] ? dump_stack+0x41/0x51
[    6.885199]  [<ffffffff81065847>] ? warn_slowpath_common+0x77/0x90
[    6.885277]  [<ffffffffa022727a>] ? btrfs_commit_transaction+0x38a/0x9c0 [btrfs]
[    6.885369]  [<ffffffffa026668a>] ? btrfs_recover_log_trees+0x38a/0x440 [btrfs]
[    6.885460]  [<ffffffffa021de50>] ? free_root_pointers+0x60/0x60 [btrfs]
[    6.885538]  [<ffffffffa0263e20>] ? replay_one_extent+0x6c0/0x6c0 [btrfs]
[    6.885613]  [<ffffffffa02249af>] ? open_ctree+0x176f/0x1f20 [btrfs]
[    6.885688]  [<ffffffffa01fb93f>] ? btrfs_mount+0x61f/0x8b0 [btrfs]
[    6.885758]  [<ffffffff81237547>] ? selinux_sb_copy_data+0x157/0x220
[    6.885831]  [<ffffffff811a9534>] ? mount_fs+0x34/0x1a0
[    6.885902]  [<ffffffff811c2d72>] ? vfs_kern_mount+0x62/0x110
[    6.885970]  [<ffffffff811c55ca>] ? do_mount+0x23a/0xac0
[    6.886041]  [<ffffffff811c52b6>] ? copy_mount_options+0xb6/0x150
[    6.886109]  [<ffffffff811c6141>] ? SyS_mount+0x81/0xc0
[    6.886180]  [<ffffffff8150d32d>] ? system_call_fast_compare_end+0x10/0x15
[    6.886252] ---[ end trace 9b7dbfb04527c9ee ]---

====

Your system hits the same following WARN_ON() many times at the boot time.

fs/btrfs/delayed-inode.c:
===============================================================================
void btrfs_assert_delayed_root_empty(struct btrfs_root *root)

{
        struct btrfs_delayed_root *delayed_root;
        delayed_root = btrfs_get_delayed_root(root);
        WARN_ON(btrfs_first_delayed_node(delayed_root));
}

===============================================================================

There is no other Btrfs related error/warning message.

It seems the same/similar problem of what Petr (CC'ed) reported.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/39721

The WARN_ON() is the same and his system also hit this problem at the boot time.

Although I briefly look at the fix for this problem by "git log --grep=btrfs_commit_transaction v3.16... fs/btrfs", I couldn't find it (Of course, I would be wrong). In addition, Petr hit this problem with 3.18-rc1. So I suspect this problem is not fixed yet.

I don't know why it happens now. Any idea? >developers.

Thanks,
Satoru

#2 lore raw

Hi Russel, Petr and all

# Add Timofey to the Cc list.

Quoted 24 lines · Russell Coker
Quoted 16 lines

I've attached the kernel message log that I get after booting kernel 3.16.7 from Debian/Unstable. This is the kernel branch that will go into Debian/Jessie so it's important to get it fixed.

Below has the start of the errors, the attached file has everything from boot. I've got similar issues on another system, would it help if I collect the logs from multiple systems?

[    6.618809] Btrfs loaded
[    6.670878] BTRFS: device fsid a2d7cbbc-23be-4d97-b2bb-de99b0c58c7d devid 1
t
ransid 548614 /dev/mapper/root-crypt
[    6.706907] BTRFS info (device dm-0): disk space caching is enabled
[    6.798762] BTRFS: detected SSD devices, enabling SSD mode
[    6.881272] ------------[ cut here ]------------
[    6.881358] WARNING: CPU: 3 PID: 198 at /build/linux-CMiYW9/linux-3.16.7-
Quoted 3 lines

ckt2
/fs/btrfs/delayed-inode.c:1410 btrfs_commit_transaction+0x38a/0x9c0 [btrfs]()

This problem is once reported by Petr and would not be fixed yet. In addition, there is no Btrfs related error/warning messages.

Quoted 8 lines

Here are my questions.

1. Is your system "btrfs scrub" clean?
2. Is this message shown every boot time?
3. Is this message shown only in boot?
4. When this message is started to be shown?
5. Do you have any trouble, change your operation or configuration just before the answer of Q4 ?

Additional questions.
Q5. Could you give me your kernel configuration?
At least, could you tell me whether your kernel enabled CONFIG_PREEMPT or not?

The first reporter, Timofey, used preemptive kernel.

> [$] uname -a
> Linux beplan 3.18.0-rc1-next-20141023-ARCH-dirty #1 SMP PREEMPT Sat > Oct 25 22:19:01 FET 2014 x86_64 GNU/Linux

# Please fix me if I'm wrong > Timofey.

I've never got this problem and I've not used preemptive kernel. This config may be correspond to this problem.

Q6. If the answer of Q1 is correct, please give me the
file system image which can be captured by the following command.

# btrfs-image <device contains Btrfs> <dest>

* The Additional result of my analyze # for developers.

The first warning message is displayed at boot time and it's happen in replay_one_extent(). The following warning messages are displayed once a 30 secs. It means these happen probably with every periodic transaction commit. So I suspect your Btrfs file system has something wrong.

Thanks,
Satoru

Quoted 69 lines

*** The result of my brief analysis (a bit long). ***

Your kernel traces are as follows > for developers

====

[    6.881272] ------------[ cut here ]------------
[    6.881358] WARNING: CPU: 3 PID: 198 at /build/linux-CMiYW9/linux-3.16.7-ckt2/fs/btrfs/delayed-inode.c:1410 btrfs_commit_transaction+0x38a/0x9c0 [btrfs]()
[    6.881453] Modules linked in: btrfs xor raid6_pq sha256_ssse3 sha256_generic ecb cbc algif_skcipher af_alg hid_generic usbhid hid dm_crypt dm_mod sd_mod crc_t10dif crct10dif_generic sg sr_mod crct10dif_common cdrom crc32c_intel r8169 mii ehci_pci ata_generic ehci_hcd ata_piix libata e1000e usbcore ptp scsi_mod usb_common pps_core
[    6.883373] CPU: 3 PID: 198 Comm: exe Not tainted 3.16.0-4-amd64 #1 Debian 3.16.7-ckt2-1
[    6.883455] Hardware name:                  /DH55HC, BIOS TCIBX10H.86A.0036.2010.0525.1620 05/25/2010
[    6.884169]  0000000000000009 ffffffff81507263 0000000000000000 ffffffff81065847
[    6.884449]  ffff8802366ba980 ffff8800b67fb000 ffff8802366bbe10 0000000000000000
[    6.884723]  0000000000000000 ffffffffa022727a cc9e8800b750eec8 006c000000000029
[    6.885057] Call Trace:
[    6.885127]  [<ffffffff81507263>] ? dump_stack+0x41/0x51
[    6.885199]  [<ffffffff81065847>] ? warn_slowpath_common+0x77/0x90
[    6.885277]  [<ffffffffa022727a>] ? btrfs_commit_transaction+0x38a/0x9c0 [btrfs]
[    6.885369]  [<ffffffffa026668a>] ? btrfs_recover_log_trees+0x38a/0x440 [btrfs]
[    6.885460]  [<ffffffffa021de50>] ? free_root_pointers+0x60/0x60 [btrfs]
[    6.885538]  [<ffffffffa0263e20>] ? replay_one_extent+0x6c0/0x6c0 [btrfs]
[    6.885613]  [<ffffffffa02249af>] ? open_ctree+0x176f/0x1f20 [btrfs]
[    6.885688]  [<ffffffffa01fb93f>] ? btrfs_mount+0x61f/0x8b0 [btrfs]
[    6.885758]  [<ffffffff81237547>] ? selinux_sb_copy_data+0x157/0x220
[    6.885831]  [<ffffffff811a9534>] ? mount_fs+0x34/0x1a0
[    6.885902]  [<ffffffff811c2d72>] ? vfs_kern_mount+0x62/0x110
[    6.885970]  [<ffffffff811c55ca>] ? do_mount+0x23a/0xac0
[    6.886041]  [<ffffffff811c52b6>] ? copy_mount_options+0xb6/0x150
[    6.886109]  [<ffffffff811c6141>] ? SyS_mount+0x81/0xc0
[    6.886180]  [<ffffffff8150d32d>] ? system_call_fast_compare_end+0x10/0x15
[    6.886252] ---[ end trace 9b7dbfb04527c9ee ]---

====

Your system hits the same following WARN_ON() many times at the boot time.

fs/btrfs/delayed-inode.c:
===============================================================================
void btrfs_assert_delayed_root_empty(struct btrfs_root *root)

{
         struct btrfs_delayed_root *delayed_root;
         delayed_root = btrfs_get_delayed_root(root);
         WARN_ON(btrfs_first_delayed_node(delayed_root));
}

===============================================================================

There is no other Btrfs related error/warning message.

It seems the same/similar problem of what Petr (CC'ed) reported.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/39721

The WARN_ON() is the same and his system also hit this problem at the boot time.

Although I briefly look at the fix for this problem by "git log --grep=btrfs_commit_transaction v3.16... fs/btrfs", I couldn't find it (Of course, I would be wrong). In addition, Petr hit this problem with 3.18-rc1. So I suspect this problem is not fixed yet.

I don't know why it happens now. Any idea? >developers.

Thanks,
Satoru

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw

Hello Satoru and all,

that Oct. report was the only time I've experienced the error, so I
don't have much to add. I can try to answer your questions:

Quoted 3 lines
Quoted 3 lines

Here are my questions.

1. Is your system "btrfs scrub" clean?

yes,

Quoted 1 line
Quoted 1 line

2. Is this message shown every boot time?

no, I have seen them only during one boot

Quoted 1 line
Quoted 1 line

3. Is this message shown only in boot?

As in my Oct. email
http://permalink.gmane.org/gmane.comp.file-systems.btrfs/39721
I've seen a similar one after creating a subvolume on a new fs. But it was during the same boot.

Quoted 3 lines
Quoted 3 lines

4. When this message is started to be shown?
5. Do you have any trouble, change your operation or configuration just before the answer of Q4 ?

a disk was added to the fs and balance has been run. The balance
crashed, as in https://bugzilla.kernel.org/show_bug.cgi?id=64961
(probably unrelated). After reboot, I've seen the messages.

Quoted 4 lines

Additional questions.
Q5. Could you give me your kernel configuration?
At least, could you tell me whether your kernel enabled CONFIG_PREEMPT or not?

CONFIG_PREEMPT_VOLUNTARY=y

Quoted 2 lines

Q6. If the answer of Q1 is correct, please give me the
file system image which can be captured by the following command.

Sorry, the fs's are long gone. I continued to run similar workloads on
that test box, but these errors never appeared again.

Regards,

Petr

#4 lore raw

Hi Petr,

Quoted 35 lines · Petr Janecek

Hello Satoru and all,

that Oct. report was the only time I've experienced the error, so I
don't have much to add. I can try to answer your questions:

Quoted 3 lines
Quoted 3 lines

Here are my questions.

1. Is your system "btrfs scrub" clean?

yes,

Quoted 1 line
Quoted 1 line

2. Is this message shown every boot time?

no, I have seen them only during one boot

Quoted 1 line
Quoted 1 line

3. Is this message shown only in boot?

As in my Oct. email
http://permalink.gmane.org/gmane.comp.file-systems.btrfs/39721
I've seen a similar one after creating a subvolume on a new fs. But it was during the same boot.

Quoted 3 lines
Quoted 3 lines

4. When this message is started to be shown?
5. Do you have any trouble, change your operation or configuration just before the answer of Q4 ?

a disk was added to the fs and balance has been run. The balance
crashed, as in https://bugzilla.kernel.org/show_bug.cgi?id=64961
(probably unrelated). After reboot, I've seen the messages.

Quoted 4 lines

Additional questions.
Q5. Could you give me your kernel configuration?
At least, could you tell me whether your kernel enabled CONFIG_PREEMPT or not?

Quoted 7 lines

CONFIG_PREEMPT_VOLUNTARY=y

Quoted 2 lines

Q6. If the answer of Q1 is correct, please give me the
file system image which can be captured by the following command.

Sorry, the fs's are long gone. I continued to run similar workloads on
that test box, but these errors never appeared again.

Thank you for giving me information.

So, further investigation of this problem seems to be hard. Please give us the above-mentioned information if this problem happens again.

Thanks,
Satoru

Quoted 10 lines

Regards,

Petr
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html