bcachefs: keep writing to device when there is no high-level I/O activity.
Hi,
I was using two partitions on same nvme device to compare filesystem performance, and I consistantly observed a strange behavior:
After 10 minutes fio test with bcachefs on one partition, performance degrade significantly for other filesystems on other partition (same device).
ext4 150M/s --> 143M/s xfs 150M/s --> 134M/s btrfs 127M/s --> 108M/s
Several round tests show the same pattern that bcachefs seems occupy some device resource even when there is no high-level I/O.
I monitor /proc/diskstats, and it confirmed that bcachefs do keep writing the device. Following is the time serial samples for "writes_completed" on my bcachefs partition:
writes_completed @timestamp
0 @1724748233.712
4 @1724748248.712 <--- mkfs
4 @1724748263.712
65 @1724748278.712
25350 @1724748293.712
63839 @1724748308.712 <--- fio started
352228 @1724748323.712
621350 @1724748338.712
903487 @1724748353.712
...
12790311 @1724748863.712
13100041 @1724748878.712
13419642 @1724748893.712
13701685 @1724748908.712 <--- fio done (10minutes)
13701769 @1724748923.712 <--- from here, average 5~7writes/second for 2000 seconds
13701852 @1724748938.712
13701953 @1724748953.712
13702032 @1724748968.712
13702133 @1724748983.712
13702213 @1724748998.712
13702265 @1724749013.712
13702357 @1724749028.712
...
13712984 @1724750858.712
13713076 @1724750873.712
13713196 @1724750888.712
13713299 @1724750903.712
13713386 @1724750918.712
13713463 @1724750933.712
13713501 @1724750948.712 <--- writes stopped here
13713501 @1724750963.712
13713501 @1724750978.712
...Is this behavior expected?
My test script:
set -e
for fsa in "btrfs" "ext4" "bcachefs" "xfs"
do
if [ $fsa == 'ext4' ]; then
mkfs -t ext4 -F /dev/nvme0n1p1
else
mkfs -t $fsa -f /dev/nvme0n1p1
fi
mount -t $fsa /dev/nvme0n1p1 /disk02/dir1
for fsb in "ext4" "bcachefs" "xfs" "btrfs"
do
if [ $fsb == 'ext4' ]; then
mkfs -t ext4 -F /dev/nvme0n1p2
else
mkfs -t $fsb -f /dev/nvme0n1p2
fi
mount -t $fsb /dev/nvme0n1p2 /disk02/dir2
cd /disk02/dir1 && fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --runtime=600 --numjobs=8 --time_based=1 --output=/disk02/fio.${fsa}.${fsb}.0
sleep 30
cd /disk02/dir2 && fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --runtime=600 --numjobs=8 --time_based=1 --output=/disk02/fio.${fsa}.${fsb}.1
sleep 30
cd /disk02
umount /disk02/dir2
done
umount /disk02/dir1
doneAnd here is a report for one round of test matrix:
+----------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+ | R|W | ext4 | bcachefs | xfs | btrfs | +----------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+ | ext4 | [ext4]147MB/s|147MB/s | [ext4]146MB/s|146MB/s | [ext4]150MB/s|150MB/s | [ext4]149MB/s|149MB/s | | | [ext4]146MB/s|146MB/s | [bcachefs]72.2MB/s|72.2MB/s | [xfs]149MB/s|149MB/s | [btrfs]132MB/s|132MB/s | | bcachefs | [bcachefs]71.9MB/s|71.9MB/s | [bcachefs]65.1MB/s|65.1MB/s | [bcachefs]69.6MB/s|69.6MB/s | [bcachefs]65.8MB/s|65.8MB/s | | | [ext4]143MB/s|143MB/s | [bcachefs]71.5MB/s|71.5MB/s | [xfs]134MB/s|133MB/s | [btrfs]108MB/s|108MB/s | | xfs | [xfs]148MB/s|148MB/s | [xfs]147MB/s|147MB/s | [xfs]152MB/s|152MB/s | [xfs]151MB/s|151MB/s | | | [ext4]147MB/s|147MB/s | [bcachefs]71.3MB/s|71.3MB/s | [xfs]148MB/s|148MB/s | [btrfs]127MB/s|127MB/s | | btrfs | [btrfs]132MB/s|132MB/s | [btrfs]112MB/s|111MB/s | [btrfs]110MB/s|110MB/s | [btrfs]110MB/s|110MB/s | | | [ext4]147MB/s|146MB/s | [bcachefs]69.7MB/s|69.7MB/s | [xfs]146MB/s|146MB/s | [btrfs]125MB/s|125MB/s | +----------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
(The rows are for the FS on the first partition, and the cols are on the second partition)
The version of bcachefs-tools on my system is 1.9.1.
(The impact is worse, ext4 dropped to 80M/s, when I was using bcachefs-tools from debian repos which is too *old*, and known to cause bcachefs problems. And that is the reason that I do this kind of test.)
Thanks
David
Quoted 8 lines · David Wang
Hi,
I was using two partitions on same nvme device to compare filesystem performance, and I consistantly observed a strange behavior:
After 10 minutes fio test with bcachefs on one partition, performance degrade significantly for other filesystems on other partition (same device).
Quoted 6 lines
ext4 150M/s --> 143M/s xfs 150M/s --> 134M/s btrfs 127M/s --> 108M/sSeveral round tests show the same pattern that bcachefs seems occupy some device resource even when there is no high-level I/O.
This is is a known issue, it should be either journal reclaim or rebalance.
(We could use some better stats to see exactly which it is)
The algorithm for how we do background work needs to change; I've written up a new one but I'm a ways off from having time to implement it
Could be a fun one for someone new to take on.
Quoted 93 lines
I monitor
/proc/diskstats, and it confirmed that bcachefs do keep writing the device. Following is the time serial samples for "writes_completed" on my bcachefs partition:
writes_completed@timestamp0 @1724748233.712 4 @1724748248.712 <--- mkfs 4 @1724748263.712 65 @1724748278.712 25350 @1724748293.712 63839 @1724748308.712 <--- fio started 352228 @1724748323.712 621350 @1724748338.712 903487 @1724748353.712 ... 12790311 @1724748863.712 13100041 @1724748878.712 13419642 @1724748893.712 13701685 @1724748908.712 <--- fio done (10minutes) 13701769 @1724748923.712 <--- from here, average 5~7writes/second for 2000 seconds 13701852 @1724748938.712 13701953 @1724748953.712 13702032 @1724748968.712 13702133 @1724748983.712 13702213 @1724748998.712 13702265 @1724749013.712 13702357 @1724749028.712 ... 13712984 @1724750858.712 13713076 @1724750873.712 13713196 @1724750888.712 13713299 @1724750903.712 13713386 @1724750918.712 13713463 @1724750933.712 13713501 @1724750948.712 <--- writes stopped here 13713501 @1724750963.712 13713501 @1724750978.712 ...Is this behavior expected?
My test script:
set -e for fsa in "btrfs" "ext4" "bcachefs" "xfs" do if [ $fsa == 'ext4' ]; then mkfs -t ext4 -F /dev/nvme0n1p1 else mkfs -t $fsa -f /dev/nvme0n1p1 fi mount -t $fsa /dev/nvme0n1p1 /disk02/dir1 for fsb in "ext4" "bcachefs" "xfs" "btrfs" do if [ $fsb == 'ext4' ]; then mkfs -t ext4 -F /dev/nvme0n1p2 else mkfs -t $fsb -f /dev/nvme0n1p2 fi mount -t $fsb /dev/nvme0n1p2 /disk02/dir2 cd /disk02/dir1 && fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --runtime=600 --numjobs=8 --time_based=1 --output=/disk02/fio.${fsa}.${fsb}.0 sleep 30 cd /disk02/dir2 && fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --runtime=600 --numjobs=8 --time_based=1 --output=/disk02/fio.${fsa}.${fsb}.1 sleep 30 cd /disk02 umount /disk02/dir2 done umount /disk02/dir1 doneAnd here is a report for one round of test matrix:
+----------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+ | R|W | ext4 | bcachefs | xfs | btrfs | +----------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+ | ext4 | [ext4]147MB/s|147MB/s | [ext4]146MB/s|146MB/s | [ext4]150MB/s|150MB/s | [ext4]149MB/s|149MB/s | | | [ext4]146MB/s|146MB/s | [bcachefs]72.2MB/s|72.2MB/s | [xfs]149MB/s|149MB/s | [btrfs]132MB/s|132MB/s | | bcachefs | [bcachefs]71.9MB/s|71.9MB/s | [bcachefs]65.1MB/s|65.1MB/s | [bcachefs]69.6MB/s|69.6MB/s | [bcachefs]65.8MB/s|65.8MB/s | | | [ext4]143MB/s|143MB/s | [bcachefs]71.5MB/s|71.5MB/s | [xfs]134MB/s|133MB/s | [btrfs]108MB/s|108MB/s | | xfs | [xfs]148MB/s|148MB/s | [xfs]147MB/s|147MB/s | [xfs]152MB/s|152MB/s | [xfs]151MB/s|151MB/s | | | [ext4]147MB/s|147MB/s | [bcachefs]71.3MB/s|71.3MB/s | [xfs]148MB/s|148MB/s | [btrfs]127MB/s|127MB/s | | btrfs | [btrfs]132MB/s|132MB/s | [btrfs]112MB/s|111MB/s | [btrfs]110MB/s|110MB/s | [btrfs]110MB/s|110MB/s | | | [ext4]147MB/s|146MB/s | [bcachefs]69.7MB/s|69.7MB/s | [xfs]146MB/s|146MB/s | [btrfs]125MB/s|125MB/s | +----------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+(The rows are for the FS on the first partition, and the cols are on the second partition)
The version of bcachefs-tools on my system is 1.9.1.
(The impact is worse, ext4 dropped to 80M/s, when I was using bcachefs-tools from debian repos which is too *old*, and known to cause bcachefs problems. And that is the reason that I do this kind of test.)Thanks
David
Hi,
Quoted 13 lines · Kent Overstreet
Quoted 12 lines · David Wang
Hi,
I was using two partitions on same nvme device to compare filesystem performance, and I consistantly observed a strange behavior:
After 10 minutes fio test with bcachefs on one partition, performance degrade significantly for other filesystems on other partition (same device).
ext4 150M/s --> 143M/s xfs 150M/s --> 134M/s btrfs 127M/s --> 108M/s
Quoted 8 lines
Quoted 2 lines
Several round tests show the same pattern that bcachefs seems occupy some device resource even when there is no high-level I/O.
This is is a known issue, it should be either journal reclaim or rebalance.
(We could use some better stats to see exactly which it is)
I kprobe bch2_submit_wbio_replicas and then bch2_btree_node_write, confirmed that the background writes were from bch2_journal_reclaim_thread.
(And then, by skimming the code in __bch2_journal_reclaim, I noticed those trace_and_count stats)
Quoted 8 lines
The algorithm for how we do background work needs to change; I've written up a new one but I'm a ways off from having time to implement it
Could be a fun one for someone new to take on.
Quoted 1 line
A Fun and scary one....
For the issue in this thread,
I think *idle* should be defined to be device wide:
when bcachefs is idle while other FS on the same block device is busy, those background threads should be throttled to some degree.
Thanks
David
Quoted 53 lines · David Wang
Hi,
I was using two partitions on same nvme device to compare filesystem performance, and I consistantly observed a strange behavior:
After 10 minutes fio test with bcachefs on one partition, performance degrade significantly for other filesystems on other partition (same device).
ext4 150M/s --> 143M/s xfs 150M/s --> 134M/s btrfs 127M/s --> 108M/sSeveral round tests show the same pattern that bcachefs seems occupy some device resource even when there is no high-level I/O.
I monitor
/proc/diskstats, and it confirmed that bcachefs do keep writing the device. Following is the time serial samples for "writes_completed" on my bcachefs partition:
writes_completed@timestamp0 @1724748233.712 4 @1724748248.712 <--- mkfs 4 @1724748263.712 65 @1724748278.712 25350 @1724748293.712 63839 @1724748308.712 <--- fio started 352228 @1724748323.712 621350 @1724748338.712 903487 @1724748353.712 ... 12790311 @1724748863.712 13100041 @1724748878.712 13419642 @1724748893.712 13701685 @1724748908.712 <--- fio done (10minutes) 13701769 @1724748923.712 <--- from here, average 5~7writes/second for 2000 seconds 13701852 @1724748938.712 13701953 @1724748953.712 13702032 @1724748968.712 13702133 @1724748983.712 13702213 @1724748998.712 13702265 @1724749013.712 13702357 @1724749028.712 ... 13712984 @1724750858.712 13713076 @1724750873.712 13713196 @1724750888.712 13713299 @1724750903.712 13713386 @1724750918.712 13713463 @1724750933.712 13713501 @1724750948.712 <--- writes stopped here 13713501 @1724750963.712 13713501 @1724750978.712 ...
Quoted 1 line
Is this behavior expected?
We've got tracepoints and counters (in sysfs) for checking what's doing the writing.
If it's a moderate amount of IO, it's probably journal reclaim; it trickles out writes if the journal has room (it's trying to spread out bursty work). If it's a lot of IO, it might be a bug.
Changing journal reclaim behaviour (and other background tasks) is on the todo list, the current behaviour is not great for machines where we want to "race to idle" to save power.