btrfs panic in 3.5.0

DISCUSSION Marc MERLIN 4 participants 8 messages Aug 7, 2012 – Aug 9, 2012 Btrfs file system

lore

Share & tools

lore

Marc MERLIN

author

2012-08-07 17:40 UTC #0 lore raw

Unfortunately I only have a screenshot.

Apparently the panic was in
btrfs_set_lock_blocking_rw
with a RIP in btrfs_cow_block

Screenshot here:
http://marc.merlins.org/tmp/btrfs_oops.jpg

Because the display looks a bit messed up, I can't tell if the ata error happened before or after the oops.

System rebooted ok.

Was there a better way to get this ooops if I didn't have serial console?

Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/

Arne Jansen

2012-08-07 18:14 UTC #1 lore raw

Quoted 6 lines · Marc MERLIN

Unfortunately I only have a screenshot.
Apparently the panic was in
btrfs_set_lock_blocking_rw
with a RIP in btrfs_cow_block

Can you please resolve btrfs_cow_block+0x3b to a line number?

gdb btrfs.ko
(gdb) info line *btrfs_cow_block+0x3b

Thanks,
Arne

Quoted 12 lines

Screenshot here:
http://marc.merlins.org/tmp/btrfs_oops.jpg
Because the display looks a bit messed up, I can't tell if the ata error happened before or after the oops.
System rebooted ok.
Was there a better way to get this ooops if I didn't have serial console?
Marc

Marc MERLIN

author

2012-08-07 18:47 UTC #2 lore raw

Quoted 11 lines · Arne Jansen

Quoted 5 lines · Marc MERLIN
Unfortunately I only have a screenshot.
Apparently the panic was in
btrfs_set_lock_blocking_rw
with a RIP in btrfs_cow_block
Can you please resolve btrfs_cow_block+0x3b to a line number?
gdb btrfs.ko
(gdb) info line *btrfs_cow_block+0x3b

So, I'm not very good at this, sorry if I'm doing it wrong:

gandalfthegreat:~# gdb /lib/modules/3.5.0-amd64-preempt-noide-20120410/kernel/fs/btrfs/btrfs.ko
Reading symbols from /lib/modules/3.5.0-amd64-preempt-noide-20120410/kernel/fs/btrfs/btrfs.ko...(no debugging symbols found)...done.
(gdb) info line *btrfs_cow_block+0x3b
No line number information available for address 0x9a6e

Mmmh, it seems that I'm missing a kernel option that adds symbols in modules?

I can add it for my next kernel compile. Do you have the config option name off hand?

I put my module here if that helps:
http://marc.merlins.org/tmp/btrfs.ko

Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/

Marc MERLIN

author

2012-08-09 02:52 UTC #3 lore raw

Quoted 19 lines · Marc MERLIN

Quoted 11 lines · Arne Jansen
Quoted 5 lines · Marc MERLIN
Unfortunately I only have a screenshot.
Apparently the panic was in
btrfs_set_lock_blocking_rw
with a RIP in btrfs_cow_block
Can you please resolve btrfs_cow_block+0x3b to a line number?
gdb btrfs.ko
(gdb) info line *btrfs_cow_block+0x3b
So, I'm not very good at this, sorry if I'm doing it wrong:
gandalfthegreat:~# gdb /lib/modules/3.5.0-amd64-preempt-noide-20120410/kernel/fs/btrfs/btrfs.ko
Reading symbols from /lib/modules/3.5.0-amd64-preempt-noide-20120410/kernel/fs/btrfs/btrfs.ko...(no debugging symbols found)...done.
(gdb) info line *btrfs_cow_block+0x3b
No line number information available for address 0x9a6e

Quoted 7 lines

Mmmh, it seems that I'm missing a kernel option that adds symbols in modules?
I can add it for my next kernel compile. Do you have the config option name off hand?
I put my module here if that helps:
http://marc.merlins.org/tmp/btrfs.ko

I felt bad for having a kernel without debug symbols it seems, so I looked at my kernel config and I do have:
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set

Any idea what else I'm missing to provide better debug info if I have a problem again?

And is it reasonably easy to take the .ko apparently without line numbers, like the one I gave you, and infer the line of code for a function offset?

Thanks,
Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/

Arne Jansen

2012-08-09 06:42 UTC #4 lore raw

Quoted 34 lines · Marc MERLIN

Quoted 19 lines · Marc MERLIN
Quoted 11 lines · Arne Jansen
Quoted 5 lines · Marc MERLIN
Unfortunately I only have a screenshot.
Apparently the panic was in
btrfs_set_lock_blocking_rw
with a RIP in btrfs_cow_block
Can you please resolve btrfs_cow_block+0x3b to a line number?
gdb btrfs.ko
(gdb) info line *btrfs_cow_block+0x3b
So, I'm not very good at this, sorry if I'm doing it wrong:
gandalfthegreat:~# gdb /lib/modules/3.5.0-amd64-preempt-noide-20120410/kernel/fs/btrfs/btrfs.ko
Reading symbols from /lib/modules/3.5.0-amd64-preempt-noide-20120410/kernel/fs/btrfs/btrfs.ko...(no debugging symbols found)...done.
(gdb) info line *btrfs_cow_block+0x3b
No line number information available for address 0x9a6e
Quoted 7 lines
Mmmh, it seems that I'm missing a kernel option that adds symbols in modules?
I can add it for my next kernel compile. Do you have the config option name off hand?
I put my module here if that helps:
http://marc.merlins.org/tmp/btrfs.ko
I felt bad for having a kernel without debug symbols it seems, so I looked at my kernel config and I do have:
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set

Quoted 5 lines

Any idea what else I'm missing to provide better debug info if I have a problem again?
And is it reasonably easy to take the .ko apparently without line numbers, like the one I gave you, and infer the line of code for a function offset?

The .ko is fine. It crashes here:

noinline int btrfs_cow_block(struct btrfs_trans_handle *trans,
struct btrfs_root *root, struct extent_buffer *buf, struct extent_buffer *parent, int parent_slot, struct extent_buffer **cow_ret)

{
        u64 search_start;
        int ret;

        if (trans->transaction != root->fs_info->running_transaction) {
                printk(KERN_CRIT "trans %llu running %llu\n",
                       (unsigned long long)trans->transid,
                       (unsigned long long)
                       root->fs_info->running_transaction->transid);
                                                          ^^

                WARN_ON(1);
        }

fs_info->running_transaction is probably NULL.

Quoted 3 lines

Thanks,
Marc

Jan Schmidt

2012-08-09 06:43 UTC #5 lore raw

Quoted 61 lines · Arne Jansen

Quoted 34 lines · Marc MERLIN
Quoted 19 lines · Marc MERLIN
Quoted 11 lines · Arne Jansen
Quoted 5 lines · Marc MERLIN
Unfortunately I only have a screenshot.
Apparently the panic was in
btrfs_set_lock_blocking_rw
with a RIP in btrfs_cow_block
Can you please resolve btrfs_cow_block+0x3b to a line number?
gdb btrfs.ko
(gdb) info line *btrfs_cow_block+0x3b
So, I'm not very good at this, sorry if I'm doing it wrong:
gandalfthegreat:~# gdb /lib/modules/3.5.0-amd64-preempt-noide-20120410/kernel/fs/btrfs/btrfs.ko
Reading symbols from /lib/modules/3.5.0-amd64-preempt-noide-20120410/kernel/fs/btrfs/btrfs.ko...(no debugging symbols found)...done.
(gdb) info line *btrfs_cow_block+0x3b
No line number information available for address 0x9a6e
Quoted 7 lines
Mmmh, it seems that I'm missing a kernel option that adds symbols in modules?
I can add it for my next kernel compile. Do you have the config option name off hand?
I put my module here if that helps:
http://marc.merlins.org/tmp/btrfs.ko
I felt bad for having a kernel without debug symbols it seems, so I looked at my kernel config and I do have:
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
Quoted 5 lines
Any idea what else I'm missing to provide better debug info if I have a problem again?
And is it reasonably easy to take the .ko apparently without line numbers, like the one I gave you, and infer the line of code for a function offset?
The .ko is fine. It crashes here:
noinline int btrfs_cow_block(struct btrfs_trans_handle *trans,
struct btrfs_root *root, struct extent_buffer *buf, struct extent_buffer *parent, int parent_slot, struct extent_buffer **cow_ret)
{
        u64 search_start;
        int ret;

        if (trans->transaction != root->fs_info->running_transaction) {
                printk(KERN_CRIT "trans %llu running %llu\n",
                       (unsigned long long)trans->transid,
                       (unsigned long long)
                       root->fs_info->running_transaction->transid);
                                                          ^^

                WARN_ON(1);
        }

Quoted 1 line

fs_info->running_transaction is probably NULL.

Agreed. Which means, that we probably came through btrfs_cleanup_transaction, which explicitly sets it to NULL.

-Jan

Jérôme Poulin

2012-08-08 02:38 UTC #6 lore raw

In reply to Marc MERLIN

Quoted 2 lines · Marc MERLIN

System rebooted ok.

I just want to be sure that you are aware that your hard drive is currently killing itself. Those READ FPDMA QUEUED mean that your hard disk is relocatting bad sectors and has problem reading those.

Marc MERLIN

author

2012-08-08 02:47 UTC #7 lore raw

Quoted 7 lines · Jérôme Poulin

Quoted 2 lines · Marc MERLIN
System rebooted ok.
I just want to be sure that you are aware that your hard drive is currently killing itself. Those READ FPDMA QUEUED mean that your hard disk is relocatting bad sectors and has problem reading those.

Yeah, I saw that, so it's actually an SSD (the wretched samsung one I've been posting about), and I'm just about to return it.

What's interesting is that smart shows no such error:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       132
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       19
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       29
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   051   040   000    Old_age   Always       -       49
195 Hardware_ECC_Recovered  0x001a   200   200   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   253   253   000    Old_age   Always       -       2
235 Unknown_Attribute       0x0012   099   099   000    Old_age   Always       -       6
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       627681656

I'm not saying nothing is wrong with the drive, but that it's not a magnetic bad sector.

Either way, I'm going to get a different SSD soon, although I guess this faliure mode was useful in finding a bug in the btrfs code in the meantime :)

Thanks for the heads up
Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/