Skip to contentSkip to conversation

BTRFS BUG at insert_inline_extent_backref+0xe3/0xf0 while rebalancing

DISCUSSION Erkki Seppala 3 participants 5 messages Oct 22, 2015 – Oct 30, 2015 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hello,

Recently I added daily rebalancing to my cron.d (after finding myself in the no-space-situation), and not long after that, I found my PC had crashed over night. Having no sign in the logs anywhere (not even over network even though there should be) I had nothing to go on, but this night it crashed again after starting the rebalance, and this time there was some information on the kernel log.

Kernel version: 4.2.3 (package linux-image-4.2.0-1-amd64 version 4.2.3-1 from Debian Unstable)

The dump is available at:

http://www.modeemi.fi/~flux/btrfs/btrfs-BUG-2015-10-55.txt

The log is available as well (stripped some unrelated USB- and firewall logging, showing that last evening there was some kernel task hung for 120 seconds; but it's in another btrfs filesystem and is another story):

http://www.modeemi.fi/~flux/btrfs/btrfs-2015-10-55.txt

I'm not quite sure which of the btrfs balance commands caused the issue. But there is my script:

#!/bin/sh
fs="$1"
if [ -z "$fs" ]; then
echo usage: btrfs-balance / 0 1 5 10 20 50
exit 1
fi
fs="$1"
shift
for usage in d m; do for a in "$@"; do date; /bin/btrfs balance start "$fs" -v -${usage}usage=$a; done; done

And it was started at 07:30 with:

/usr/local/sbin/btrfs-balance / 0 1 2 5 10 20 30 50 70

I should add that the filesystem in question is backed by MD RAID10 and that is backed by four SSDs, so it's reasonably fast in IO, if that affects anything. There should have been no much competing IO at the time of the occurrence.

Before Duncan asks ;-), I only have a moderate number of subvolumes and snapshots, ie. one subvolume for each of /, /var/log/journal and /home, 24 snapshots of / and /home plus <10 snapshots of /.

Before that balance there was another balance on a another BTRFS RAID10, but given the time stamp I think I can easily say it wasn't the cause.

I don't really have other 'solutions' than disabling the rebalancing for the time being, and only use it as-needed as I had earlier done..

Cheers,

Signature · 4 lines
  _____________________________________________________________________
     / __// /__ ____  __               http://www.modeemi.fi/~flux/\   \
    / /_ / // // /\ \/ /                                            \  /
   /_/  /_/ \___/ /_/\_\@modeemi.fi                                  \/
#1 lore raw
Quoted 49 lines · Erkki Seppala

Hello,

Recently I added daily rebalancing to my cron.d (after finding myself in the no-space-situation), and not long after that, I found my PC had crashed over night. Having no sign in the logs anywhere (not even over network even though there should be) I had nothing to go on, but this night it crashed again after starting the rebalance, and this time there was some information on the kernel log.

Kernel version: 4.2.3 (package linux-image-4.2.0-1-amd64 version 4.2.3-1 from Debian Unstable)

The dump is available at:

http://www.modeemi.fi/~flux/btrfs/btrfs-BUG-2015-10-55.txt

The log is available as well (stripped some unrelated USB- and firewall logging, showing that last evening there was some kernel task hung for 120 seconds; but it's in another btrfs filesystem and is another story):

http://www.modeemi.fi/~flux/btrfs/btrfs-2015-10-55.txt

I'm not quite sure which of the btrfs balance commands caused the issue. But there is my script:

#!/bin/sh
fs="$1"
if [ -z "$fs" ]; then
echo usage: btrfs-balance / 0 1 5 10 20 50
exit 1
fi
fs="$1"
shift
for usage in d m; do for a in "$@"; do date; /bin/btrfs balance start "$fs" -v -${usage}usage=$a; done; done

And it was started at 07:30 with:

/usr/local/sbin/btrfs-balance / 0 1 2 5 10 20 30 50 70

I should add that the filesystem in question is backed by MD RAID10 and that is backed by four SSDs, so it's reasonably fast in IO, if that affects anything. There should have been no much competing IO at the time of the occurrence.

Before Duncan asks ;-), I only have a moderate number of subvolumes and snapshots, ie. one subvolume for each of /, /var/log/journal and /home, 24 snapshots of / and /home plus <10 snapshots of /.

Quoted 5 lines

Before that balance there was another balance on a another BTRFS RAID10, but given the time stamp I think I can easily say it wasn't the cause.

I don't really have other 'solutions' than disabling the rebalancing for the time being, and only use it as-needed as I had earlier done..

Try this (just sent a few minutes ago):
https://patchwork.kernel.org/patch/7463161/

thanks

Quoted 13 lines

Cheers,

--
_____________________________________________________________________
/ __// /__ ____ __ http://www.modeemi.fi/~flux/\ \
/ /_ / // // /\ \/ / \ /
/_/ /_/ \___/ /_/\_\@modeemi.fi \/

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."
#2 lore raw

Le 2015-10-22 10:53, Filipe Manana a écrit :

Quoted 50 lines · Erkki Seppala
Quoted 49 lines

Hello,

Recently I added daily rebalancing to my cron.d (after finding myself in the no-space-situation), and not long after that, I found my PC had crashed over night. Having no sign in the logs anywhere (not even over network even though there should be) I had nothing to go on, but this night it crashed again after starting the rebalance, and this time there was some information on the kernel log.

Kernel version: 4.2.3 (package linux-image-4.2.0-1-amd64 version 4.2.3-1 from Debian Unstable)

The dump is available at:

http://www.modeemi.fi/~flux/btrfs/btrfs-BUG-2015-10-55.txt

The log is available as well (stripped some unrelated USB- and firewall logging, showing that last evening there was some kernel task hung for 120 seconds; but it's in another btrfs filesystem and is another story):

http://www.modeemi.fi/~flux/btrfs/btrfs-2015-10-55.txt

I'm not quite sure which of the btrfs balance commands caused the issue. But there is my script:

#!/bin/sh
fs="$1"
if [ -z "$fs" ]; then
echo usage: btrfs-balance / 0 1 5 10 20 50
exit 1
fi
fs="$1"
shift
for usage in d m; do for a in "$@"; do date; /bin/btrfs balance start "$fs" -v -${usage}usage=$a; done; done

And it was started at 07:30 with:

/usr/local/sbin/btrfs-balance / 0 1 2 5 10 20 30 50 70

I should add that the filesystem in question is backed by MD RAID10 and that is backed by four SSDs, so it's reasonably fast in IO, if that affects anything. There should have been no much competing IO at the time of the occurrence.

Before Duncan asks ;-), I only have a moderate number of subvolumes and snapshots, ie. one subvolume for each of /, /var/log/journal and /home, 24 snapshots of / and /home plus <10 snapshots of /.

Quoted 8 lines
Quoted 5 lines

Before that balance there was another balance on a another BTRFS RAID10, but given the time stamp I think I can easily say it wasn't the cause.

I don't really have other 'solutions' than disabling the rebalancing for the time being, and only use it as-needed as I had earlier done..

Try this (just sent a few minutes ago):
https://patchwork.kernel.org/patch/7463161/

Awesome, I'll also try it right now under 4.3.0-rc6. My system is currently hit so hard by this bug that it no longer survives a balance for longer than a few minutes.

Will keep you posted on the outcome.

Thanks,

Signature · 1 line
Stéphane.
#3 lore raw
In reply to Filipe Manana

Hello,

Thanks for the super-fast response :).

I've installed the patch and shall be waiting. The effects should be visible within a week given daily rebalances of two filesystems.

Signature · 4 lines
  _____________________________________________________________________
     / __// /__ ____  __               http://www.modeemi.fi/~flux/\   \
    / /_ / // // /\ \/ /                                            \  /
   /_/  /_/ \___/ /_/\_\@modeemi.fi                                  \/
#4 lore raw
In reply to Filipe Manana
Quoted 2 lines · Filipe Manana

Try this (just sent a few minutes ago):
https://patchwork.kernel.org/patch/7463161/

I've been using this patch for a week now, doing two rebalances a day (one per file system) - no problem so far. Thanks!

Probably unrelated to this I did experience one reboot without any trace, possibly because I had enabled panic = 10 and panic_on_oops = 1, but that event did not happen anytime near a balance was happening. I wonder if the hang detector could trigger that configuration to reboot?

Thanks again for the great work, your detective work is always impressive :).

Signature · 4 lines
  _____________________________________________________________________
     / __// /__ ____  __               http://www.modeemi.fi/~flux/\   \
    / /_ / // // /\ \/ /                                            \  /
   /_/  /_/ \___/ /_/\_\@modeemi.fi                                  \/