System crash while scrubbing on 3.18.x

Hello,

Sorry for the long email...

I've found my system locks up when scrubbing with 3.18.x, but not with 3.17.8 across 2 systems.

I have the following BTRFS partitions on system 1:
/ (128GiB, 49GiB used on SSD)
/home (4.2TiB, 624GB used on HDD RAID volume)

I have the following BTRFS partitions on system 2:
/ (196GiB, 17GiB used on HDD RAID volume) /home (7.1TiB, 2.9TiB used on HDD RAID volume)

My OS is Netrunner 15 (which 98% Kubuntu) on system 1, and up-to-date debian testing on system 2.

I've never encountered a lock up while scrubbing /. Just with /home.

The systems never lock up immediately, but takes some time. VERY rarely I'll see the lockup when the scrub is at <100GiB completed. Typically it happens somewhere between 200-350GiB. A few times it's gone beyond 500GiB. This is probably why I've never encountered the issue with /, it's just not big enough on either system.

Both systems were otherwise idle while performing the scrubs that crashed the systems.

/home is on a partition on a RAID10 volume on a 3ware 9740-4i controller with 4x 3TB disks on system 1. On system 2, it's the same controller but with 4x 4TB disks (and / on system 2 is a partition on the same RAID volume rather than a separate disk). Both systems have 32GiB memory, and the otherwise the hardware is pretty different between the systems (AMD Vs. Intel, etc).

I suspect that the RAID controller probably isn't relevant. Both arrays and their drives are healthy.

I've also encountered the issue on a freshly formatted filesystem with my data copied from a backup on system 1.

I've tried tried scrubbing with btrfs-progs 3.17 (installed from the distribution repos on both systems), and btrfs-progs from git (using tag v3.18.x). Neither version made a difference.

In case this is helpful to anyone, here's how I've discovered the issue:
I decided to test btrfs with bcache on system 1 to see if the stability had improved since I'd tried bcache+btrfs about a year ago. I backed up /home on system 1 and then freshly formatted it and set it to use bcache. I was running Linux 3.18.8 and encountered the problem that I've described above. I assumed the bcache+btrfs combination was still broken so I formatted the system again (this time still using btrfs, but without bcache) and copied all my files back. I encountered the same issue without bcache. Realizing the issue wasn't bcache related, I did ANOTHER format, this time back to bcache+btrfs.

From here in my testing, I found that system 2 (which has no bcache) also crashed when scrubbing with Linux 3.18.8. I decided to try 3.17.8 on system 1 (since 3.18.8 seemed to be the common denominator between the 2 systems), found that fixed the issue, and then downgraded system 2 to use 3.17.8 as well, which also fixed the issue there.

(Note: At one point I also tried Linux 3.18.7 and 3.18.5, however, those kernels are affected by the scrub/crash issue as well.)

I found something else interesting when I tested against Linux 3.19.0. With 3.19.0, the bcache system always crashes fairly early in the scrub (<100GiB), but the non-bcache system has no issues. This suggests my problem with 3.19.0 is a bcache+btrfs issue (or simply an issue with bcache).

I'm not sure if bcache is relevant to the BTRFS devs at this point, but I thought I'd put that there for anyone who might find that information useful.

To summarize:
I've tested with 2 systems, and scrubbing caused crashes occurred on both with Linux 3.18.8, but not with 3.17.8 for both systems I've tested 1 system with and without bcache, and bcache made no difference between Linux 3.17.8 and 3.18.8.
I've tested with 3.19.0, and I crash when scrubbing on the bcache system, but not the non-bcache system.

Quoted 78 lines · Cameron Berkenpas

Hello,

Sorry for the long email...

I've found my system locks up when scrubbing with 3.18.x, but not with 3.17.8 across 2 systems.

I have the following BTRFS partitions on system 1:
/ (128GiB, 49GiB used on SSD)
/home (4.2TiB, 624GB used on HDD RAID volume)

I have the following BTRFS partitions on system 2:
/ (196GiB, 17GiB used on HDD RAID volume) /home (7.1TiB, 2.9TiB used on HDD RAID volume)

My OS is Netrunner 15 (which 98% Kubuntu) on system 1, and up-to-date debian testing on system 2.

I've never encountered a lock up while scrubbing /. Just with /home.

The systems never lock up immediately, but takes some time. VERY rarely I'll see the lockup when the scrub is at <100GiB completed. Typically it happens somewhere between 200-350GiB. A few times it's gone beyond 500GiB. This is probably why I've never encountered the issue with /, it's just not big enough on either system.

Both systems were otherwise idle while performing the scrubs that crashed the systems.

/home is on a partition on a RAID10 volume on a 3ware 9740-4i controller with 4x 3TB disks on system 1. On system 2, it's the same controller but with 4x 4TB disks (and / on system 2 is a partition on the same RAID volume rather than a separate disk). Both systems have 32GiB memory, and the otherwise the hardware is pretty different between the systems (AMD Vs. Intel, etc).

I suspect that the RAID controller probably isn't relevant. Both arrays and their drives are healthy.

I've also encountered the issue on a freshly formatted filesystem with my data copied from a backup on system 1.

I've tried tried scrubbing with btrfs-progs 3.17 (installed from the distribution repos on both systems), and btrfs-progs from git (using tag v3.18.x). Neither version made a difference.

In case this is helpful to anyone, here's how I've discovered the issue:
I decided to test btrfs with bcache on system 1 to see if the stability had improved since I'd tried bcache+btrfs about a year ago. I backed up /home on system 1 and then freshly formatted it and set it to use bcache. I was running Linux 3.18.8 and encountered the problem that I've described above. I assumed the bcache+btrfs combination was still broken so I formatted the system again (this time still using btrfs, but without bcache) and copied all my files back. I encountered the same issue without bcache. Realizing the issue wasn't bcache related, I did ANOTHER format, this time back to bcache+btrfs.

From here in my testing, I found that system 2 (which has no bcache)
also crashed when scrubbing with Linux 3.18.8. I decided to try 3.17.8 on system 1 (since 3.18.8 seemed to be the common denominator between the 2 systems), found that fixed the issue, and then downgraded system 2 to use 3.17.8 as well, which also fixed the issue there.

(Note: At one point I also tried Linux 3.18.7 and 3.18.5, however, those kernels are affected by the scrub/crash issue as well.)

I found something else interesting when I tested against Linux 3.19.0. With 3.19.0, the bcache system always crashes fairly early in the scrub (<100GiB), but the non-bcache system has no issues. This suggests my problem with 3.19.0 is a bcache+btrfs issue (or simply an issue with bcache).

I'm not sure if bcache is relevant to the BTRFS devs at this point, but I thought I'd put that there for anyone who might find that information useful.

Quoted 7 lines

To summarize:
I've tested with 2 systems, and scrubbing caused crashes occurred on both with Linux 3.18.8, but not with 3.17.8 for both systems I've tested 1 system with and without bcache, and bcache made no difference between Linux 3.17.8 and 3.18.8.
I've tested with 3.19.0, and I crash when scrubbing on the bcache system, but not the non-bcache system.

Better to have some stacks about the scrub crash.

Thanks,

-liubo
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello,

I should have mentioned this earlier. I get absolutely nothing in the logs. I can't look at dmesg either as the system is completely unresponsive.

Quoted 95 lines · Liu Bo

Quoted 78 lines · Cameron Berkenpas

Hello,

Sorry for the long email...

I've found my system locks up when scrubbing with 3.18.x, but not with 3.17.8 across 2 systems.

I have the following BTRFS partitions on system 1:
/ (128GiB, 49GiB used on SSD)
/home (4.2TiB, 624GB used on HDD RAID volume)

I have the following BTRFS partitions on system 2:
/ (196GiB, 17GiB used on HDD RAID volume) /home (7.1TiB, 2.9TiB used on HDD RAID volume)

My OS is Netrunner 15 (which 98% Kubuntu) on system 1, and up-to-date debian testing on system 2.

I've never encountered a lock up while scrubbing /. Just with /home.

The systems never lock up immediately, but takes some time. VERY rarely I'll see the lockup when the scrub is at <100GiB completed. Typically it happens somewhere between 200-350GiB. A few times it's gone beyond 500GiB. This is probably why I've never encountered the issue with /, it's just not big enough on either system.

Both systems were otherwise idle while performing the scrubs that crashed the systems.

/home is on a partition on a RAID10 volume on a 3ware 9740-4i controller with 4x 3TB disks on system 1. On system 2, it's the same controller but with 4x 4TB disks (and / on system 2 is a partition on the same RAID volume rather than a separate disk). Both systems have 32GiB memory, and the otherwise the hardware is pretty different between the systems (AMD Vs. Intel, etc).

I suspect that the RAID controller probably isn't relevant. Both arrays and their drives are healthy.

I've also encountered the issue on a freshly formatted filesystem with my data copied from a backup on system 1.

I've tried tried scrubbing with btrfs-progs 3.17 (installed from the distribution repos on both systems), and btrfs-progs from git (using tag v3.18.x). Neither version made a difference.

In case this is helpful to anyone, here's how I've discovered the issue:
I decided to test btrfs with bcache on system 1 to see if the stability had improved since I'd tried bcache+btrfs about a year ago. I backed up /home on system 1 and then freshly formatted it and set it to use bcache. I was running Linux 3.18.8 and encountered the problem that I've described above. I assumed the bcache+btrfs combination was still broken so I formatted the system again (this time still using btrfs, but without bcache) and copied all my files back. I encountered the same issue without bcache. Realizing the issue wasn't bcache related, I did ANOTHER format, this time back to bcache+btrfs.

From here in my testing, I found that system 2 (which has no bcache)
also crashed when scrubbing with Linux 3.18.8. I decided to try 3.17.8 on system 1 (since 3.18.8 seemed to be the common denominator between the 2 systems), found that fixed the issue, and then downgraded system 2 to use 3.17.8 as well, which also fixed the issue there.

(Note: At one point I also tried Linux 3.18.7 and 3.18.5, however, those kernels are affected by the scrub/crash issue as well.)

I found something else interesting when I tested against Linux 3.19.0. With 3.19.0, the bcache system always crashes fairly early in the scrub (<100GiB), but the non-bcache system has no issues. This suggests my problem with 3.19.0 is a bcache+btrfs issue (or simply an issue with bcache).

I'm not sure if bcache is relevant to the BTRFS devs at this point, but I thought I'd put that there for anyone who might find that information useful.

Quoted 7 lines

To summarize:
I've tested with 2 systems, and scrubbing caused crashes occurred on both with Linux 3.18.8, but not with 3.17.8 for both systems I've tested 1 system with and without bcache, and bcache made no difference between Linux 3.17.8 and 3.18.8.
I've tested with 3.19.0, and I crash when scrubbing on the bcache system, but not the non-bcache system.

Better to have some stacks about the scrub crash.

Thanks,

-liubo

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Minor update:

I've now tested 4.0-rc4 on systems. Both systems lockup during a scrub.

I've also tested a laptop with 4.0-rc4 and I have no issues. It could be that I "only" have 221GB on it, that it's an SSD, and/or the lack of a 3ware controller.

Within the next few days, I hope to have the time to test on a btrfs RAID1 system I have (btrfs RAID) as well.

-Cameron

Quoted 105 lines · Cameron Berkenpas

Hello,

I should have mentioned this earlier. I get absolutely nothing in the logs. I can't look at dmesg either as the system is completely unresponsive.

Quoted 95 lines · Liu Bo

Quoted 78 lines · Cameron Berkenpas

Hello,

Sorry for the long email...

I've found my system locks up when scrubbing with 3.18.x, but not with 3.17.8 across 2 systems.

I have the following BTRFS partitions on system 1:
/ (128GiB, 49GiB used on SSD)
/home (4.2TiB, 624GB used on HDD RAID volume)

I have the following BTRFS partitions on system 2:
/ (196GiB, 17GiB used on HDD RAID volume) /home (7.1TiB, 2.9TiB used on HDD RAID volume)

My OS is Netrunner 15 (which 98% Kubuntu) on system 1, and up-to-date debian testing on system 2.

I've never encountered a lock up while scrubbing /. Just with /home.

The systems never lock up immediately, but takes some time. VERY rarely I'll see the lockup when the scrub is at <100GiB completed. Typically it happens somewhere between 200-350GiB. A few times it's gone beyond 500GiB. This is probably why I've never encountered the issue with /, it's just not big enough on either system.

Both systems were otherwise idle while performing the scrubs that crashed the systems.

/home is on a partition on a RAID10 volume on a 3ware 9740-4i controller with 4x 3TB disks on system 1. On system 2, it's the same controller but with 4x 4TB disks (and / on system 2 is a partition on the same RAID volume rather than a separate disk). Both systems have 32GiB memory, and the otherwise the hardware is pretty different between the systems (AMD Vs. Intel, etc).

I suspect that the RAID controller probably isn't relevant. Both arrays and their drives are healthy.

I've also encountered the issue on a freshly formatted filesystem with my data copied from a backup on system 1.

I've tried tried scrubbing with btrfs-progs 3.17 (installed from the distribution repos on both systems), and btrfs-progs from git (using tag v3.18.x). Neither version made a difference.

In case this is helpful to anyone, here's how I've discovered the issue:
I decided to test btrfs with bcache on system 1 to see if the stability had improved since I'd tried bcache+btrfs about a year ago. I backed up /home on system 1 and then freshly formatted it and set it to use bcache. I was running Linux 3.18.8 and encountered the problem that I've described above. I assumed the bcache+btrfs combination was still broken so I formatted the system again (this time still using btrfs, but without bcache) and copied all my files back. I encountered the same issue without bcache. Realizing the issue wasn't bcache related, I did ANOTHER format, this time back to bcache+btrfs.

From here in my testing, I found that system 2 (which has no bcache)
also crashed when scrubbing with Linux 3.18.8. I decided to try 3.17.8 on system 1 (since 3.18.8 seemed to be the common denominator between the 2 systems), found that fixed the issue, and then downgraded system 2 to use 3.17.8 as well, which also fixed the issue there.

(Note: At one point I also tried Linux 3.18.7 and 3.18.5, however, those kernels are affected by the scrub/crash issue as well.)

I found something else interesting when I tested against Linux 3.19.0. With 3.19.0, the bcache system always crashes fairly early in the scrub (<100GiB), but the non-bcache system has no issues. This suggests my problem with 3.19.0 is a bcache+btrfs issue (or simply an issue with bcache).

I'm not sure if bcache is relevant to the BTRFS devs at this point, but I thought I'd put that there for anyone who might find that information useful.

Quoted 7 lines

To summarize:
I've tested with 2 systems, and scrubbing caused crashes occurred on both with Linux 3.18.8, but not with 3.17.8 for both systems I've tested 1 system with and without bcache, and bcache made no difference between Linux 3.17.8 and 3.18.8.
I've tested with 3.19.0, and I crash when scrubbing on the bcache system, but not the non-bcache system.

Better to have some stacks about the scrub crash.

Thanks,

-liubo

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

I've figured out at least part of the issue.

I had blk-mq enabled by default (SCSI_MQ_DEFAULT=y). I've since set SCSI_MQ_DEFAULT=n, and now I'm completely stable with 4.0-rc4, 3.18.9, and presumably all the others. This is on both the bcache and non-bcache systems.

All of my systems with btrfs and SCSI_MQ_DEFAULT=y are stable unless they're using a 3ware 9750 controller (which I believe is the 3w-sas driver from the top of my head). So either there's a bug with blk-mq+3w-sas+btrfs, or just blk-mq+3w-sas. I suspect the latter, but if I can find the time, I'll try to test with another filesystem.

I suspect other controllers/drivers could be affected as well.

Are any btrfs devs interested in me looking into this further?

As a side note, in spite of all the crashes I've had through out my testing etc, bcache+btrfs hasn't given me any issues like I've seen in the past. This is just a few weeks worth of testing, but it seems there's at least some sort of basic stability with bcache+btrfs now.