Issue (as expected) when upgrading from 3.12 to 3.13.7
Hello,
As I was expecting I'm getting new issues with bcache after upgrading my kernel:
[Mar30 12:24] INFO: task bcache_writebac:155 blocked for more than 120 seconds.
[ +0.000007] Not tainted 3.13.7-1-ARCH #1
[ +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0.000003] bcache_writebac D 0000000000000000 0 155 20x00000000
[ +0.000006] ffff8804049c5eb8 0000000000000046 ffff880405816c00 ffff8804049c5fd8
[ +0.000005] 0000000000014440 0000000000014440 ffff880405816c00 ffff8804049c5e38
[ +0.000004] ffffffff810925fa 24448b48e0d83524 fa83ffffee31e918 0000000000000046
[ +0.000004] Call Trace: [ +0.000010] [<ffffffff810925fa>] ? try_to_wake_up+0x1fa/0x2c0 [ +0.000006] [<ffffffff81092712>] ? default_wake_function+0x12/0x20 [ +0.000005] [<ffffffff810a2538>] ? __wake_up_common+0x58/0x90 [ +0.000023] [<ffffffffa03c7470>] ? read_dirty_endio+0x60/0x60 [bcache] [ +0.000005] [<ffffffff81516239>] schedule+0x29/0x70 [ +0.000006] [<ffffffff8108358d>] kthread+0xad/0xf0 [ +0.000005] [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180 [ +0.000005] [<ffffffff81521abc>] ret_from_fork+0x7c/0xb0 [ +0.000005] [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180
That's just sad to see how unstable was/is bcache sub system and I still don't understand why it was not marked as experimental.
3.12 was/is buggy when used with writeback mode, and I just had to live with cache-through. I tried to get some helps (and others too) but no answers.
I read there were issue with 3.13, so I delayed my kernel upgrade as far I could due to this. I was hoping that issues were fixed in 3.13.x stable tree but apparently not.
Hopefully this one will not be ignored.
Thanks
Hi,
Quoted 35 lines · Francis Moreau
As I was expecting I'm getting new issues with bcache after upgrading my kernel:
[Mar30 12:24] INFO: task bcache_writebac:155 blocked for more than 120 seconds.
[ +0.000007] Not tainted 3.13.7-1-ARCH #1
[ +0.000002] "echo 0 >/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0.000003]bcache_writebacD 0000000000000000 0 155 20x00000000
[ +0.000006]ffff8804049c5eb80000000000000046ffff880405816c00ffff8804049c5fd8
[ +0.000005] 0000000000014440 0000000000014440ffff880405816c00ffff8804049c5e38
[ +0.000004]ffffffff810925fa24448b48e0d83524fa83ffffee31e9180000000000000046[ +0.000004] Call Trace: [ +0.000010] [<ffffffff810925fa>] ? try_to_wake_up+0x1fa/0x2c0 [ +0.000006] [<ffffffff81092712>] ? default_wake_function+0x12/0x20 [ +0.000005] [<ffffffff810a2538>] ? __wake_up_common+0x58/0x90 [ +0.000023] [<ffffffffa03c7470>] ? read_dirty_endio+0x60/0x60 [bcache] [ +0.000005] [<ffffffff81516239>] schedule+0x29/0x70 [ +0.000006] [<ffffffff8108358d>] kthread+0xad/0xf0 [ +0.000005] [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180 [ +0.000005] [<ffffffff81521abc>] ret_from_fork+0x7c/0xb0 [ +0.000005] [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180That's just sad to see how unstable was/is bcache sub system and I still don't understand why it was not marked as experimental.
3.12 was/is buggy when used with writeback mode, and I just had to live with cache-through. I tried to get some helps (and others too) but no answers.
Quoted 5 lines
I read there were issue with 3.13, so I delayed my kernel upgrade as far I could due to this. I was hoping that issues were fixed in 3.13.x stable tree but apparently not.
Hopefully this one will not be ignored.
Francis - I've found it helps if you add all the relevant addresses mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this mail I'm going to explicitly CC Kent to help this get noticed. Additionally do you also see this problem with 3.14?
Signature · 1 line
Sitsofe | http://sucs.org/~sits/
Quoted 42 lines · Sitsofe Wheeler
Hi,
Quoted 39 lines · Francis Moreau
As I was expecting I'm getting new issues with bcache after upgrading my kernel:
[Mar30 12:24] INFO: task bcache_writebac:155 blocked for more than 120 seconds.
[ +0.000007] Not tainted 3.13.7-1-ARCH #1
[ +0.000002] "echo 0 >/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0.000003]bcache_writebacD 0000000000000000 0 155 20x00000000
[ +0.000006]ffff8804049c5eb80000000000000046ffff880405816c00ffff8804049c5fd8
[ +0.000005] 0000000000014440 0000000000014440ffff880405816c00ffff8804049c5e38
[ +0.000004]ffffffff810925fa24448b48e0d83524fa83ffffee31e9180000000000000046[ +0.000004] Call Trace: [ +0.000010] [<ffffffff810925fa>] ? try_to_wake_up+0x1fa/0x2c0 [ +0.000006] [<ffffffff81092712>] ? default_wake_function+0x12/0x20 [ +0.000005] [<ffffffff810a2538>] ? __wake_up_common+0x58/0x90 [ +0.000023] [<ffffffffa03c7470>] ? read_dirty_endio+0x60/0x60 [bcache] [ +0.000005] [<ffffffff81516239>] schedule+0x29/0x70 [ +0.000006] [<ffffffff8108358d>] kthread+0xad/0xf0 [ +0.000005] [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180 [ +0.000005] [<ffffffff81521abc>] ret_from_fork+0x7c/0xb0 [ +0.000005] [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180That's just sad to see how unstable was/is bcache sub system and I still don't understand why it was not marked as experimental.
3.12 was/is buggy when used with writeback mode, and I just had to live with cache-through. I tried to get some helps (and others too) but no answers.
I read there were issue with 3.13, so I delayed my kernel upgrade as far I could due to this. I was hoping that issues were fixed in 3.13.x stable tree but apparently not.
Quoted 7 lines
Quoted 1 line
Hopefully this one will not be ignored.
Francis - I've found it helps if you add all the relevant addresses mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this mail I'm going to explicitly CC Kent to help this get noticed. Additionally do you also see this problem with 3.14?
Sorry 3.14 is not available yet on my distro.
Thanks
Quoted 8 lines · Francis Moreau
Quoted 5 lines · Sitsofe Wheeler
Francis - I've found it helps if you add all the relevant addresses mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this mail I'm going to explicitly CC Kent to help this get noticed. Additionally do you also see this problem with 3.14?
Sorry 3.14 is not available yet on my distro.
But 3.14.rc7 is via linux-git right?
Signature · 1 line
Sitsofe | http://sucs.org/~sits/
Quoted 12 lines · Sitsofe Wheeler
Quoted 8 lines · Francis Moreau
Quoted 5 lines · Sitsofe Wheeler
Francis - I've found it helps if you add all the relevant addresses mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this mail I'm going to explicitly CC Kent to help this get noticed. Additionally do you also see this problem with 3.14?
Sorry 3.14 is not available yet on my distro.
But 3.14.rc7 is via linux-git right?
I was initialy reporting for 3.13.7.
still having the same trace with 3.13.8.
Quoted 49 lines · Sitsofe Wheeler
Hi,
Quoted 35 lines · Francis Moreau
As I was expecting I'm getting new issues with bcache after upgrading my kernel:
[Mar30 12:24] INFO: task bcache_writebac:155 blocked for more than 120 seconds.
[ +0.000007] Not tainted 3.13.7-1-ARCH #1
[ +0.000002] "echo 0 >/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0.000003]bcache_writebacD 0000000000000000 0 155 20x00000000
[ +0.000006]ffff8804049c5eb80000000000000046ffff880405816c00ffff8804049c5fd8
[ +0.000005] 0000000000014440 0000000000014440ffff880405816c00ffff8804049c5e38
[ +0.000004]ffffffff810925fa24448b48e0d83524fa83ffffee31e9180000000000000046[ +0.000004] Call Trace: [ +0.000010] [<ffffffff810925fa>] ? try_to_wake_up+0x1fa/0x2c0 [ +0.000006] [<ffffffff81092712>] ? default_wake_function+0x12/0x20 [ +0.000005] [<ffffffff810a2538>] ? __wake_up_common+0x58/0x90 [ +0.000023] [<ffffffffa03c7470>] ? read_dirty_endio+0x60/0x60 [bcache] [ +0.000005] [<ffffffff81516239>] schedule+0x29/0x70 [ +0.000006] [<ffffffff8108358d>] kthread+0xad/0xf0 [ +0.000005] [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180 [ +0.000005] [<ffffffff81521abc>] ret_from_fork+0x7c/0xb0 [ +0.000005] [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180That's just sad to see how unstable was/is bcache sub system and I still don't understand why it was not marked as experimental.
3.12 was/is buggy when used with writeback mode, and I just had to live with cache-through. I tried to get some helps (and others too) but no answers.
Quoted 5 lines
I read there were issue with 3.13, so I delayed my kernel upgrade as far I could due to this. I was hoping that issues were fixed in 3.13.x stable tree but apparently not.
Hopefully this one will not be ignored.
Francis - I've found it helps if you add all the relevant addresses mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this mail I'm going to explicitly CC Kent to help this get noticed. Additionally do you also see this problem with 3.14?
Quoted 2 lines · Francis Moreau
That's just sad to see how unstable was/is bcache sub system and I still don't understand why it was not marked as experimental.
IMO, It should still be marked as experimental. Possibly, even removed from mainline as the maintainer is not particularly responsive to major bugs. Certainly not production ready and it's been through many releases of the kernel now.
-Peter
Quoted 9 lines · Peter Kieser
Quoted 2 lines · Francis Moreau
That's just sad to see how unstable was/is bcache sub system and I still don't understand why it was not marked as experimental.
IMO, It should still be marked as experimental. Possibly, even removed from mainline as the maintainer is not particularly responsive to major bugs. Certainly not production ready and it's been through many releases of the kernel now.
Well, removing from mainline is quite extreme but moving it to the staging stuff would be a possible alternative.
And it should definitely be marked experimental in stable trees such as 3.12 (LTS kernel) IMHO. Specially since those trees don't get any fixes.
On Sun, 30 Mar 2014, Francis Moreau wrote:
Hi Francis: A quick question for you if you have the time:
How is the bcache constructed? ie which devices are in it?
I have had similar lockup problems when using partitions as backing, but not whole devices. That is to say:
{ ram0 : sdb1 } often deadlocks for me (depending on the offset
of the partition)
{ ram0 : sdb } always seems to work fine
Although in neither case am I able to tear down the device and flush it all to backing after heavy use: That always locks up for me. (Fortunately I'm making throwaway filesystems for a build daemon so that part I can live with).
Quoted 5 lines · Vivek Dasmohapatra
On Sun, 30 Mar 2014, Francis Moreau wrote:
Hi Francis: A quick question for you if you have the time:
How is the bcache constructed? ie which devices are in it?
sure: I'm using bcache on a very basic setup: no MD or LVM involved. /dev/sda4 (900Mo) is the backing device while /dev/sdb (120G) is the cache device. On top of bcache0 I'm using ext4 and I'm using it as my root device.
Quoted 6 lines
I have had similar lockup problems when using partitions as backing, but not whole devices. That is to say:
{ ram0 : sdb1 } often deadlocks for me (depending on the offset
of the partition)
Quoted 7 lines
{ ram0 : sdb } always seems to work fine
Although in neither case am I able to tear down the device and flush it all to backing after heavy use: That always locks up for me. (Fortunately I'm making throwaway filesystems for a build daemon so that part I can live with).
For now it doesn't endup to lock up (cross fingers). It only fills the kernel logs, well I hope.
Thanks.
Quoted 11 lines · Francis Moreau
Quoted 5 lines · Vivek Dasmohapatra
On Sun, 30 Mar 2014, Francis Moreau wrote:
Hi Francis: A quick question for you if you have the time:
How is the bcache constructed? ie which devices are in it?sure: I'm using bcache on a very basic setup: no MD or LVM involved.
/dev/sda4(900Mo) is the backing device while/dev/sdb(120G) is the cache device. On top of bcache0 I'm using ext4 and I'm using it as my root device.
Ok, interesting. I've had some luck tweaking the offset of the backing device partition: Some offsets trigger the bug consistently and others never seem to, so it seems to be some sort of weird alignment bug.
Quoted 10 lines · Vivek Dasmohapatra <vivek <at> collabora.co.uk> writes:
Quoted 7 lines · Francis Moreau
Quoted 5 lines · Vivek Dasmohapatra
On Sun, 30 Mar 2014, Francis Moreau wrote:
Hi Francis: A quick question for you if you have the time:
How is the bcache constructed? ie which devices are in it?
Quoted 10 lines
Quoted 4 lines
sure: I'm using bcache on a very basic setup: no MD or LVM involved.
/dev/sda4(900Mo) is the backing device while/dev/sdb(120G) is the cache device. On top of bcache0 I'm using ext4 and I'm using it as my root device.Ok, interesting. I've had some luck tweaking the offset of the backing device partition: Some offsets trigger the bug consistently and others never seem to, so it seems to be some sort of weird alignment bug.
I'd started a new thread before I realized how to reply to this.
The problem appears to be that bcache_writeback is trying to do
something it shouldn't when in writethrough mode
(bcache_writeback is on by default even for writethrough).
echo 0 > /sys/block/bcache0/bcache/writeback_running
...appears to fix it.